Joint load estimation program, computer-readable recording medium having joint load estimation program recorded thereon, joint load estimation device, and joint load estimation method

Abstract

[PROBLEM] Provided is a joint load estimation program or the like for estimating a load of force applied to a joint of a subject on the basis of an image in which the exercise state of the subject is captured. [SOLUTION] The present invention is a joint load estimation program, which causes a computer to execute: a base exercise state acquisition step for acquiring an exercise state of a subject on the basis of an image in which the exercise state of the subject is captured; a rigid link model acquisition step for acquiring a three-dimensional rigid link model on the basis of a predetermined feature amount related to the human body of the subject; a rigid link system exercise state acquisition step for estimating the exercise state of the three-dimensional rigid link model in accordance with the exercise state of a subject 2 acquired by the base exercise state acquisition step, and acquiring exercise information about the three-dimensional rigid link model; and a joint load calculation step for calculating the joint reaction force applied to the joint tissue by inverse dynamics calculation based on the exercise information about the three-dimensional rigid link model.

Description

Joint load estimation program, computer-readable recording medium having the joint load estimation program recorded thereon, joint load estimation device, and joint load estimation method

[0001] The present invention relates to a joint load estimation program, a computer-readable recording medium on which the joint load estimation program is recorded, a joint load estimation device, and a joint load estimation method.

[0002] As disclosed in Patent Document 1, techniques such as optical motion capture are known as methods for analyzing three-dimensional human motion. However, optical motion capture has the drawback of requiring the attachment of optical markers to the subject. Furthermore, the optical markers must be captured by multiple cameras, and if they are hidden from the cameras, it becomes difficult to obtain accurate data. Therefore, research has been conducted into determining the position information of each joint of the subject from image information and calculating a multi-joint link model in which the subject's body is modeled as a combination of hinge joints, without imposing the burden of attaching optical markers or the like to the human body.

[0003] Japanese Patent Application Laid-Open No. 2016-6415

[0004] However, in such research, there was a problem that errors in the joint position information obtained from images could cause changes in the inter-joint distance. If such a model were used to perform inverse dynamics calculations to specifically calculate the forces acting on joint torque and joint reaction force, there was a risk of the accuracy of the calculation results decreasing. Furthermore, since inverse dynamics calculations use, for example, joint angular velocity and joint angular acceleration due to time differences, errors in estimating the motion state before starting the inverse dynamics calculation could have a relatively large effect on the inverse dynamics calculation. Therefore, it was difficult to calculate the joint reaction force acting on joint tissues using inverse dynamics calculations based on captured images.

[0005] The present invention has been made to solve such problems, and aims to provide a joint load estimation program that can calculate the joint reaction force acting on joint tissues based on images of the subject's movement state, a computer-readable recording medium on which the joint load estimation program is recorded, a joint load estimation device, and a joint load estimation method.

[0006] To achieve the above object, according to one embodiment of the present invention, there is provided a joint load estimation program for estimating a load of a force acting on a body joint based on a captured image, the program causing a computer to execute the following steps: a base motion state acquisition step for acquiring a motion state of a subject based on captured images of the subject's motion state; a rigid link model acquisition step for acquiring a three-dimensional rigid link model based on predetermined feature quantities related to the subject's body; a rigid link system motion state acquisition step for estimating a motion state of the three-dimensional rigid link model in accordance with the motion state of the subject acquired in the base motion state acquisition step and acquiring motion information of the three-dimensional rigid link model; and a joint load calculation step for calculating a joint reaction force acting on joint tissue by inverse dynamics calculation based on the motion information of the three-dimensional rigid link model. According to one embodiment of the present invention configured in this manner, the motion state of the three-dimensional rigid link model is estimated in accordance with the motion state of the subject acquired in the base motion state acquisition step based on the captured image, and the joint load calculation step can calculate the joint reaction force acting on the joint tissue by inverse dynamics calculation based on the motion information of the three-dimensional rigid link model. As a result, even if the subject's motion state is acquired based on a photographed image, the link lengths and connection relationships in the motion information of the three-dimensional rigid link model can be maintained at predetermined lengths and relationships, and the joint reaction forces acting on the joint tissues can be calculated by inverse dynamics calculation based on the motion information, allowing the load on the body's joints to be analyzed.In addition, advice and services based on the specific numerical values ​​of the joint reaction forces can be provided to the subject.

[0007] According to one embodiment of the present invention, preferably, the joint load calculation step calculates a compressive force as the joint reaction force. According to one embodiment of the present invention configured in this manner, the joint load calculation step calculates a compressive force as the joint reaction force. As a result, even if the motion state of the subject is acquired based on photographed images of the motion state of the subject, the link lengths and connection relationships in the motion information of the three-dimensional rigid link model can be maintained at predetermined lengths and relationships, the compressive force acting on joint tissues can be calculated by inverse dynamics calculation based on the motion information, and the compressive load on the joints of the body can be analyzed.

[0008] According to one embodiment of the present invention, preferably, the joint load calculation step calculates a shear force as the joint reaction force. According to one embodiment of the present invention configured in this manner, the joint load calculation step calculates a shear force as the joint reaction force. As a result, even when the motion state of the subject is acquired based on images capturing the motion state of the subject, the link lengths and connection relationships in the motion information of the three-dimensional rigid link model can be maintained at predetermined lengths and relationships, the shear force acting on the joint tissue can be calculated by inverse dynamics calculation based on the human motion information, and the shear force load acting on the joints of the body can be analyzed.

[0009] According to one embodiment of the present invention, preferably, the joint load calculation step calculates a torsional moment as the joint reaction force. According to one embodiment of the present invention configured in this manner, the joint load calculation step calculates a torsional moment as the joint reaction force. As a result, even when the motion state of the subject is acquired based on photographed images of the motion state of the subject, the link lengths and connection relationships in the motion information of the three-dimensional rigid link model can be maintained at predetermined lengths and relationships, the torsional moment applied to the joint tissue can be calculated by inverse dynamics calculation based on the human motion information, and the load of the torsional moment applied to the joints of the body can be analyzed.

[0010] According to one embodiment of the present invention, the joint load calculation step preferably comprises a floor reaction force estimating step of estimating a floor reaction force based on parameters of position and velocity in a generalized coordinate system of the three-dimensional rigid link model, without relying on measurement of the floor reaction force by a floor reaction force meter. According to one embodiment of the present invention configured in this manner, the joint load calculation step preferably comprises a floor reaction force estimating step of estimating a floor reaction force based on parameters such as position and velocity in a generalized coordinate system of the three-dimensional rigid link model, without relying on measurement of the floor reaction force by a floor reaction force meter. This makes it possible to estimate the floor reaction force of the subject based on images capturing the subject's motion state.

[0011] According to one embodiment of the present invention, the joint load calculation step preferably includes an inverse dynamics calculation step of estimating joint forces by inverse dynamics calculation using the acceleration in the generalized coordinate system of the three-dimensional rigid link model and the estimated floor reaction force. According to one embodiment of the present invention configured in this manner, the joint load calculation step preferably includes an inverse dynamics calculation estimation step of estimating joint forces that are the basis for joint reaction forces by inverse dynamics calculation using the acceleration in the generalized coordinate system of the three-dimensional rigid link model and the estimated floor reaction force. This makes it possible to estimate joint forces acting on joint portions between links of a three-dimensional rigid link model based on images capturing the motion state of a subject.

[0012] According to one embodiment of the present invention, the joint load calculation step preferably includes a muscle tension estimation step of estimating a tension in accordance with a muscle force model based on a muscle force model corresponding to a predetermined link of the three-dimensional rigid link model. According to one embodiment of the present invention configured in this manner, the joint load calculation step preferably includes a muscle tension estimation step of estimating a tension in accordance with the muscle force model based on a muscle force model corresponding to a predetermined link of the three-dimensional rigid link model. This makes it possible to correct the joint force obtained in the inverse dynamics calculation estimation step by taking into account muscle tension elements based on the muscle force model, thereby enabling more accurate estimation of the joint load.

[0013] According to one embodiment of the present invention, the method preferably further comprises a rigid link model adjustment step of adjusting the dimensions of the acquired three-dimensional rigid link model by estimating the lengths of at least some of the links based on an image of the subject. According to one embodiment of the present invention configured in this manner, the rigid link model adjustment step allows the three-dimensional rigid link model to be adjusted based on an image of the subject. This makes it easier to match the dimensions of the three-dimensional rigid link model to the physique of the subject being photographed, and the accuracy of the motion information of the three-dimensional rigid link model is likely to be improved. Therefore, it is easier to obtain motion state data with an accuracy that is easy to use for inverse dynamics calculations, and it is possible to calculate the joint reaction force acting on the joint tissues by inverse dynamics calculations that analyze the load on the forces acting on the joints of the human body.

[0014] According to one embodiment of the present invention, preferably, the method further comprises a notification step of notifying the user that the joint reaction force is outside of a predetermined range when the joint reaction force calculated in the joint load calculation step is outside of a predetermined range. According to one embodiment of the present invention configured as described above, the method further comprises a notification step of notifying the user that the joint reaction force is outside of the predetermined range when the joint reaction force calculated in the joint load calculation step is outside of the predetermined range. This allows the subject to easily recognize that the joint reaction force is outside of the predetermined range from the image of the subject, making it easier to recognize points to note about the subject's walking style, etc.

[0015] According to one embodiment of the present invention, preferably, the rigid link system motion state acquisition step estimates the motion information of the three-dimensional rigid link model by performing inverse kinematics calculation of the three-dimensional rigid link model. According to one embodiment of the present invention configured in this manner, the rigid link system motion state acquisition step estimates the motion information of the three-dimensional rigid link model by performing inverse kinematics calculation of the three-dimensional rigid link model. This makes it possible to estimate the motion state of the three-dimensional rigid link model from an image obtained by a camera, and the motion information obtained by the inverse kinematics calculation can be used for inverse dynamics calculation.

[0016] According to one embodiment of the present invention, preferably a computer-readable recording medium having any of the above-mentioned joint load estimation programs recorded thereon. According to one embodiment of the present invention configured in this manner, the computer-readable recording medium having the above-mentioned joint load estimation program recorded thereon can cause a computer to execute predetermined steps, thereby making it possible to estimate the load of forces acting on joints of the body.

[0017] According to one embodiment of the present invention, there is preferably provided a joint load estimation device for estimating a load of a force acting on a joint of a body based on a captured image, the joint load estimation device comprising: a base motion state acquisition functional unit for acquiring a motion state of a subject based on an image captured by a camera; a rigid link model acquisition functional unit for acquiring a three-dimensional rigid link model based on predetermined feature quantities related to the subject's body; a rigid link system motion state acquisition functional unit for estimating the motion state of the three-dimensional rigid link model acquired by the three-dimensional rigid link model acquisition functional unit in accordance with the motion state of the subject and acquiring motion information of the three-dimensional rigid link model; and a joint load calculation functional unit for calculating a joint reaction force acting on joint tissue by inverse dynamics calculation based on the motion information of the three-dimensional rigid link model. According to one embodiment of the present invention configured in this manner, the motion state of the three-dimensional rigid link model is estimated in accordance with the motion state of the subject acquired by the base motion state acquisition functional unit based on the captured image, and the joint load calculation step can calculate the joint reaction force acting on the joint tissue by inverse dynamics calculation based on the motion information of the three-dimensional rigid link model. As a result, even if the subject's motion state is acquired based on a photographed image, the link lengths and connection relationships in the motion information of the three-dimensional rigid link model can be maintained at predetermined lengths and relationships, and the joint reaction forces acting on the joint tissues can be calculated by inverse dynamics calculation based on the motion information, allowing the load on the body's joints to be analyzed.In addition, advice and services based on the specific numerical values ​​of the joint reaction forces can be provided to the subject.

[0018] According to one embodiment of the present invention, there is preferably provided a joint load estimation method for estimating a load of a force acting on a joint of a body based on a captured image, the method comprising: a base motion state acquisition step of acquiring a motion state of the subject based on a captured image of the motion state of the subject; a rigid link model acquisition step of acquiring a three-dimensional rigid link model based on predetermined feature quantities related to the subject's body; a rigid link system motion state acquisition step of estimating a motion state of the three-dimensional rigid link model in accordance with the motion state of the subject acquired in the base motion state acquisition step and acquiring motion information of the three-dimensional rigid link model; and a joint load calculation step of calculating a joint reaction force acting on joint tissue by inverse dynamics calculation based on the motion information of the three-dimensional rigid link model. According to one embodiment of the present invention configured in this manner, the motion state of the three-dimensional rigid link model is estimated in accordance with the motion state of the subject acquired in the base motion state acquisition step based on the captured image, and the joint load calculation step can calculate the joint reaction force acting on the joint tissue by inverse dynamics calculation based on the motion information of the three-dimensional rigid link model. As a result, even if the subject's motion state is acquired based on a photographed image, the link lengths and connection relationships in the motion information of the three-dimensional rigid link model can be maintained at predetermined lengths and relationships, and the joint reaction forces acting on the joint tissues can be calculated by inverse dynamics calculation based on the motion information, allowing the load on the body's joints to be analyzed.In addition, advice and services based on the specific numerical values ​​of the joint reaction forces can be provided to the subject.

[0019] According to the joint load estimation program, computer-readable recording medium having the joint load estimation program recorded thereon, joint load estimation device, and joint load estimation method of the present invention, the joint reaction force acting on joint tissues can be calculated based on images of the subject's motion state.

[0020] 1. is a schematic diagram of a joint load estimation system equipped with a joint load estimation program according to one embodiment of the present invention. 1 is a schematic diagram of a joint load estimation system according to one embodiment of the present invention. 2 is a block diagram showing the configuration of a joint load estimation device in the joint load estimation system according to one embodiment of the present invention. 3 is a diagram showing how the joint load estimation device of FIG. 3 is connected to another server via the Internet. 4 is a block diagram showing the configuration of the joint load estimation device in the joint load estimation system according to one embodiment of the present invention. 5 is a block diagram showing the configuration of the joint load calculation function unit of FIG. 6. 7 is a diagram showing how body landmarks in a color image of a subject are estimated by a joint load estimation device according to one embodiment of the present invention. 8 is a diagram illustrating how body landmarks in a color image of a subject are estimated by a joint load estimation device according to one embodiment of the present invention. 9 is a diagram showing a color image and a depth image acquired by a camera of the joint load estimation system according to one embodiment of the present invention. 10 is a diagram showing an example of a three-dimensional rigid link model acquired by the joint load estimation device according to one embodiment of the present invention. 11 is a diagram showing an outline of a control flow executed by the joint load estimation device of the joint load estimation system according to one embodiment of the present invention. 12 is a diagram explaining in more detail the rigid link model acquisition step of FIG. 11. 13 is a diagram showing how the position of a generalized coordinate system of a rigid link system is acquired over time in the joint load estimation system according to one embodiment of the present invention. Figure 1 is a diagram showing how joint loads are estimated over time in a joint load estimation system according to one embodiment of the present invention. Figure 2 is a diagram showing how the proportions of major links in a subject are estimated by a rigid link model adjustment function unit of a joint load estimation device according to one embodiment of the present invention. Figure 3 is a diagram showing how the lengths of links in a three-dimensional rigid link model are estimated by a rigid link model adjustment function unit of a joint load estimation device according to one embodiment of the present invention. Figure 4 is a diagram showing an outline of a control flow for joint load estimation in a joint load estimation system according to one embodiment of the present invention. Figure 5 is a diagram showing an explanation of parameters of inverse dynamics calculation in joint load estimation executed by a joint load estimation device according to one embodiment of the present invention.1 is a diagram illustrating the relationship between a three-dimensional rigid link model and a muscle tension model in joint load estimation performed by a joint load estimation device according to one embodiment of the present invention. FIG. 2 is a diagram showing elements of knee joint load along the X and Z axes in joint load estimation performed by a joint load estimation device according to one embodiment of the present invention. FIG. 3 is a diagram showing elements of knee joint load along the Y and Z axes in joint load estimation performed by a joint load estimation device according to one embodiment of the present invention. FIG. 4 is a diagram showing elements of knee joint load along the Y and Z axes in joint load estimation performed by a joint load estimation device according to one embodiment of the present invention. FIG. 5 is a diagram showing schematically a control flow executed by a joint load estimation device of a joint load estimation system according to a modified embodiment of the present invention. FIG. 6 is a diagram showing schematically a control flow executed by a joint load estimation device of a joint load estimation system according to a modified embodiment of the present invention.

[0021] A joint load estimation system 1 according to an embodiment of the present invention, including a computer that executes a joint load estimation program 12 (see FIG. 2), will be described below with reference to the accompanying drawings. The embodiments of the present disclosure have been described as examples, and it will be apparent to those skilled in the art that many modifications, changes, and substitutions are possible within the spirit and scope of the present invention. Therefore, the present invention is not limited to the disclosed embodiments, and various modifications, changes, etc. are possible in form and details without departing from the scope of the claims. Furthermore, the components disclosed in the specification can be freely combined.

[0022] As shown in FIG. 1 , a joint load estimation system 1 according to one embodiment of the present invention estimates the load of forces acting on the joints of a subject's body. The joint load estimation system 1 performs inverse kinematics calculations of a rigid link system motion model based on images of the motion state of a subject 2, and then performs inverse dynamics calculations based on the results to analyze the load of forces acting on the joints of the subject's body. The joint load estimation system 1 also functions as a motion state estimation system that performs inverse kinematics calculations of a rigid link system motion model. Thus, the joint load estimation system 1 estimates the motion state so as to obtain data used in dynamics calculations, such as inverse dynamics calculations, for analyzing the load of forces acting on the joints of the subject's body. The joint load estimation system 1 also supports the acquisition of motion state data of the subject that can be used in inverse dynamics calculations. In this way, the joint load estimation system 1 is both a motion data acquisition support system for inverse dynamics calculations and, as described below, a rigid link system motion model construction system. Similarly, the joint load estimation program is also a motion state data acquisition support program for inverse dynamics calculations, and the joint load estimation method described below is also a motion state data acquisition support method for inverse dynamics calculations. In this embodiment, "exercise" refers to an object changing its spatial position over time. Therefore, "exercise" includes active physical movement for sports, but is not limited to active physical movement for sports. The subject's "exercise" is expressed by the subject's "motion."

[0023] As shown in FIG. 2 , the joint load estimation system 1 includes a camera 4 that captures images of the motion of a subject 2 and a joint load estimation device 10 that estimates the load of forces acting on the joints of the human body. The joint load estimation system 1 is a non-contact joint load estimation system that acquires motion state data of the subject 2 simply by capturing images of the subject's motion using the camera 4, and estimates joint loads. The joint load estimation system 1 and the joint load estimation device 10 can estimate joint loads from images of the subject's motion in a non-contact manner, without attaching devices such as sensors to the human body or directly measuring forces using a force plate (floor reaction force meter). Inverse dynamics calculation is a calculation that calculates joint torque and joint reaction force from all or part of position, velocity, acceleration, joint angle, angular velocity, angular acceleration, and floor reaction force. As described below, inverse dynamics calculation is distinguished from inverse kinematics calculation, which calculates joint angles from body landmark positions. In particular, the accuracy of data (motion information data) such as position and velocity used in the inverse dynamics calculation is required to be high, because if an error occurs in this data, it will easily become a larger error in the inverse dynamics calculation. Therefore, the motion state data for the inverse dynamics calculation cannot be easily obtained by simple motion capture, and it is necessary to consciously obtain and construct motion state data that is unlikely to cause errors (and is usable) in the inverse dynamics calculation, as in this embodiment.

[0024] The camera 4 is electrically connected to the joint load estimation device 10. The camera 4 is an RGBD camera capable of capturing a color image C and a depth image D, as shown in FIGS. 7 and 9 . The RGBD camera is configured to capture not only color images (including grayscale and black-and-white images) from an RGB camera but also the distance to the object. Therefore, the camera 4 can capture a depth image D (see FIG. 9 ) containing depth information of the captured image. For example, the camera 4 is an Intel RealSense D455f. The camera 4 is equipped with a distance sensor (depth sensor) capable of capturing depth information. While a single camera can capture an image of the object relatively easily, multiple cameras may also be used. The camera 4 may be an optical motion capture device. While a camera capable of capturing depth information is preferred, the camera 4 is not limited to this. The camera 4 may also be a camera attached to a smartphone, a digital camera, or the like. Note that in FIG. 9 , for ease of explanation, a color image C and a depth image D of the same location are shown side by side. In this embodiment, a color image is combined with a depth image measured by a distance sensor, but a grayscale image (black and white image) may also be combined with a depth image. This may result in a certain decrease in estimation accuracy, but has the advantage of speeding up processing.

[0025] The joint load estimation device 10 is a joint load estimation device that estimates force loads acting on joints of the human body. The joint load estimation device 10 also functions as a motion state estimation device (motion state estimation device for acquiring data for inverse dynamics calculations) that supports the acquisition of motion state data of a subject to be used in inverse dynamics calculations for analyzing force loads acting on joints of the human body. The joint load estimation device 10 is configured, for example, by a computer that executes a predetermined joint load estimation program 12 (see FIG. 2 ). The predetermined joint load estimation program 12 estimates force loads acting on joints of the human body, as will be described later.

[0026] The joint load estimation device 10 includes a storage device 33 and a processor 34. The joint load estimation device 10 also functions as a control unit for a series of control processes. The storage device 33 is, for example, a volatile memory capable of high-speed reading and writing of information, and is used as a storage area and a working area when the processor 34 processes information. The storage device 33 may be a non-volatile storage or non-volatile memory, for example, a flash memory such as an eMMC, UFS, or SSD. The storage device 33 stores programs and various data for executing the joint load estimation program of this embodiment. The storage device 33 constitutes a computer-readable tangible recording medium and stores the joint load estimation program 12. The processor 34 controls the operation of the device as a control unit. The processor 34 is, for example, a CPU. Note that the processor 34 may also be an electronic circuit such as an MPU. The processor 34 performs various processes by reading and executing programs and data stored in the storage device. If the program of this embodiment is stored in another computer-readable recording medium, the processor 34 may execute instructions from the program in that recording medium. The computer-readable recording medium may be any other recording medium capable of storing data, such as a USB memory, an SD card, or a DVD. Any of the programs described in the specification can be recorded on a computer-readable recording medium and executed by a computer to perform the predetermined functions described in the specification. Any of the programs can be executed by a computer, either alone or in any combination, to perform the predetermined functions described in the specification.

[0027] The joint load estimation device 10 may be a tablet computer, a smartphone, other personal digital assistant devices, a smart watch, a wearable device, or other electronic devices. The joint load estimation device 10 may include a communication device 36 for transmitting and receiving data to other devices, a display device 38 such as a display for displaying output results to the user, and an input device 40 for receiving input commands to the joint load estimation device 10. The communication device 36 may be a device or module for wireless communication or a device or module for wired communication. The display device 38 may be, for example, a monitor display, a display screen of a personal digital assistant, or a display screen of a smartphone. The joint load estimation device 10 can output the analysis results obtained by the joint load estimation device 10 to the display screen of the display device 38. The input device 40 may be, for example, a keyboard or a mouse for receiving keyboard input. These components are connected by a bus 31. Note that an interface is provided between the bus 31 and each component device as needed.

[0028] The joint load estimation device 10 includes a functional unit capable of controlling connected devices, such as the camera 4, and a program capable of controlling the camera 4.

[0029] As shown in FIG. 4 , some or all of the functional units of the joint load estimation device 10 may be provided on a server on the cloud. For example, the functional units of the joint load estimation device 10 may be executed by a program stored on the cloud. Furthermore, for example, the functional units of the joint load estimation device 10 may not be provided on the user's computer A or the server that acquires images of the subject's motion state, but may be provided on another server B that can connect to the computer A or the server via the Internet 41. For example, the functional units of the joint load estimation device 10 may be partially provided on the user's computer A or the computer A that acquires images of the subject's motion state, and partially provided on another server B that can connect to the server via the Internet 41. In this way, the functional units of the joint load estimation device 10 may be distributed and provided or executed on multiple servers. The server is, for example, an electronic device that functions as a computer, and may be, for example, a smartphone, mobile phone, smart watch, tablet PC, notebook PC, desktop PC, etc. Similarly, the server B is, for example, an electronic device that functions as a computer, and may be, for example, a smartphone, mobile phone, smart watch, tablet PC, notebook PC, desktop PC, etc.

[0030] The joint load estimation device 10 is a joint load estimation program that can be executed by a computer to estimate the load of forces acting on joints of a human body, and can execute the following steps: a base motion state acquisition step that recognizes a subject based on an image of the subject's motion state and acquires the motion state of the subject; a rigid link model acquisition step that acquires a three-dimensional rigid link model based on predetermined feature quantities related to the human body; a rigid link system motion state acquisition step that estimates the motion of the three-dimensional rigid link model acquired in the rigid link model acquisition step in accordance with the motion state of the subject acquired in the base motion state acquisition step and acquires motion information of the three-dimensional rigid link model; and a joint load calculation step that calculates joint reaction forces (e.g., compressive force, shear force) acting on joint tissues by inverse dynamics calculation based on the motion information of the three-dimensional rigid link model. The rigid link system motion state acquisition step can acquire the motion information of the three-dimensional rigid link model by performing inverse kinematics calculation of the three-dimensional rigid link model. The joint load calculation step further comprises: a floor reaction force estimation step of estimating a floor reaction force based on parameters such as the position, velocity, and acceleration in a generalized coordinate system of the three-dimensional rigid link model without contact measurement of the floor reaction force of the subject with a floor reaction force meter; an inverse dynamics calculation estimation step of estimating a joint force by inverse dynamics calculation using the acceleration in the generalized coordinate system of the three-dimensional rigid link model and the estimated floor reaction force; a muscle tension estimation step of estimating a tension in accordance with a muscle force model corresponding to a predetermined link of the three-dimensional rigid link model; and a notification step of notifying the user that the joint reaction force is outside a predetermined range when the joint reaction force calculated in the joint load calculation step is outside the predetermined range. The joint load calculation step calculates, for example, a compressive force and / or a shear force and / or a moment as the joint reaction force. The joint load calculation step further comprises a notification step of notifying the user that the joint reaction force is outside the predetermined range when the joint reaction force calculated in the joint load calculation step is outside the predetermined range.As shown below, the joint load estimation device 10 is configured with the functions executed by this joint load estimation program 12, including a base motion state acquisition function unit 14, a rigid link model acquisition function unit 16, a rigid link system motion state acquisition function unit 18, a rigid link model adjustment function unit 28, a rigid link model application function unit 19, and a joint load calculation function unit 60. The base motion state acquisition function unit 14, the rigid link model acquisition function unit 16, the rigid link system motion state acquisition function unit 18, the rigid link model adjustment function unit 28, the rigid link model application function unit 19, and the joint load calculation function unit 60 of the joint load estimation device 10 are configured from all or part of the joint load estimation program 12.

[0031] The joint load estimation device 10 comprises a base motion state acquisition function unit 14 that acquires the motion state of the subject 2 being photographed based on an image taken by the camera 4, a rigid link model acquisition function unit 16 that acquires a three-dimensional rigid link model 17 based on predetermined features related to the body of the subject 2, a rigid link system motion state acquisition function unit 18 that estimates the motion of the three-dimensional rigid link model 17 acquired by the rigid link model acquisition function unit 16 in accordance with the motion state of the subject 2 being photographed acquired by the base motion state acquisition function unit 14 and acquires motion information of the three-dimensional rigid link model 17, and a joint load calculation function unit 60 that calculates the joint load.

[0032] The base motion state acquisition function unit 14 executes a base motion state acquisition step for acquiring motion information of the subject, such as posture information of the subject (e.g., body landmarks 50C), based on an image capturing the motion state of the subject. The posture information of the subject is, for example, information about the body landmarks 50C of the subject. In this embodiment, the body landmarks 50C are characteristic local structures of the human body that are relatively important in analyzing the posture of the subject. As the body landmarks 50C, for example, all or some of the joints (shoulder joints, elbow joints, wrists, hip joints, knee joints, ankles, etc.), eyes, nose, parietal region, acromion, anterior superior iliac spine, greater trochanter, heel, and toes are used. By obtaining the position information of the body landmarks, posture information of the subject is obtained, and the posture of the subject is recognized by the base motion state acquisition function unit 14. In this way, the base motion state acquisition step acquires the motion state of the subject, including the coordinates of the body landmarks 50C, by estimating the body landmarks 50C of the human body. As a method for acquiring the motion state of the subject, instead of the body landmarks 50C, vectors indicating the directions of body parts connecting joints may be acquired and used.

[0033] The base motion state acquisition function unit 14 further includes a body landmark estimation function unit 20 that recognizes and estimates body landmarks 50 in a two-dimensional coordinate system based on an image capturing the subject's motion state; a position estimation function unit 22 that estimates the positions of the estimated body landmarks 50 as position information in the two-dimensional coordinate system; and a three-dimensional coordinate estimation unit 24 that converts the position information of the body landmarks in the two-dimensional coordinate system into position information in a three-dimensional coordinate system using depth information in the depth direction of the captured depth image.

[0034] The body landmark estimation function unit 20 estimates the positions of the body landmarks 50 using an AI program stored in the base motion state acquisition function unit 14. This AI program is acquired in advance as a trained model using machine learning or the like so that it can estimate the positions of body landmarks in the human body. Note that the body landmark estimation function unit 20 may also estimate the positions of the body landmarks 50 using an image analysis method (e.g., image comparison) that does not rely on an AI program. By combining the body landmark estimation function unit 20 and the position estimation function unit 22, the base motion state acquisition function unit 14 can estimate the positions of the body landmarks 50 as position information based on relatively simple equipment. Furthermore, by combining the body landmark estimation function unit 20, position estimation function unit 22, and three-dimensional coordinate estimation unit 24 of the base motion state acquisition function unit 14, the positions of the body landmarks 50 can be estimated as position information in a three-dimensional coordinate system based on relatively simple equipment (for example, one RGBD camera or one RGB camera), and the accuracy of fitting when the three-dimensional rigid link model 17 is fitted to the estimated body landmarks 50 can be further improved.

[0035] The rigid link model acquisition function unit 16 executes a rigid link model acquisition step of acquiring a three-dimensional rigid link model 17 (see FIG. 10 ) based on predetermined features related to the human body of the subject 2, such as height and / or weight. The rigid link model acquisition function unit 16 stores typical three-dimensional rigid link models corresponding to height. For example, heights and corresponding three-dimensional rigid link models are stored as pairs in a table. For example, the three-dimensional rigid link model corresponding to height is a three-dimensional rigid link model corresponding to an average physique estimated from the height. Therefore, the rigid link model acquisition function unit 16 can easily determine a three-dimensional rigid link model based on input of height. Furthermore, for example, the rigid link model acquisition function unit 16 may set a three-dimensional rigid link model based on input of height and weight. In this case, the rigid link model acquisition function unit 16 further executes a feature input step of receiving input of predetermined features related to the human body to be used in the rigid link model acquisition step. By executing the feature input step, it becomes possible to input feature values ​​from an input window of the user interface, etc. The rigid link model acquisition function unit 16 also determines, for example, the mass, moment of inertia, center of gravity, etc. of the links of the three-dimensional rigid link model 17.

[0036] As shown in FIG. 10 , in this technology, the subject's body is approximated by a three-dimensional rigid link model 17, which is a rigid link system. The rigid link system model is a model in which the subject's body is segmented at arbitrary joints and each joint is approximated as a single rigid body. Each segmented rigid body is a link 30, and a joint connecting two links 30 is a joint 32. The link 30 is assumed to be a rigid body and a rigid joint whose volume and shape do not change. The body dynamics structure is represented as a model in which rigid links 30 are connected, and the three-dimensional rigid link model 17 can simulate a human physique. Therefore, the three-dimensional rigid link model 17 includes links 30 and joints 32. However, information on muscles, tendons, etc. is omitted from the three-dimensional rigid link model 17. The three-dimensional rigid link model 17 may include information on the mass, moment of inertia, center of gravity, etc. of the link. The acquired three-dimensional rigid link model 17 is acquired and recorded as data that allows the posture to be freely changed to reproduce various motion states of the subject by the rigid link model acquisition function unit 16, the rigid link system motion state acquisition function unit 18, etc. Furthermore, it may be possible to input and set the lengths actually measured by a user, etc., to some or all of the links 30, etc. of the three-dimensional rigid link model 17.

[0037] Regarding the segmentation that determines the structure of the three-dimensional rigid link model 17, the joints at which the model is segmented can be changed depending on the purpose of the analysis. For example, if the torso of the body is segmented by dividing it into lumbar joints, the torso alone will be divided into 24 links. However, if one is interested only in the load on the lower back or the load on the neck, as shown in FIG. 15B, the torso area from the neck to the feet can be segmented into about three parts in the height direction, allowing analysis of the load on these parts. Of course, the model may be segmented at parts other than the joints.

[0038] Here, the position of the generalized coordinate system of the three-dimensional rigid link model 17 will be described. As shown in FIG. 10 , the joints 32 of the three-dimensional rigid link model 17, which is a rigid link system model, have a movable axis ranging from one to six depending on their location. The more movable axes a joint 32 has, the more freely the connected link 30 can move. The number of movable axes can be called the degrees of freedom. For example, a hip joint may have three movable axes and be modeled as a three-degree-of-freedom joint. In contrast, knee joints and elbow joints often move around a single movable axis, so they may be modeled as one-degree-of-freedom joints. Each joint can have a parameter that determines the degree of rotation around each movable axis. If there are N joints, each joint is identified by a number such as 0, 1, 2, ..., j ..., N-1. The parameter of joint j is denoted as qj. qj is expressed as a vector with the same number of elements as the movable axes of the corresponding joint. For example, the parameters of a hip joint with three degrees of freedom are expressed as a three-dimensional vector. The parameters qj of all joints included in the rigid link system are collectively denoted as q. Although this varies depending on the method for modeling the rigid link system, for example, q is a vector with approximately 30 dimensions. q = [q0 q1 q2 q3 ... qj ... qN-2 qN-1] This q is referred to as the position in the generalized coordinate system. By determining the specific value of q, the pose of any three-dimensional rigid link model 17 can be defined. In this way, once all joint angles are determined, the pose of the entire body can be defined. However, the position and direction of the entire body itself must be defined using a method other than joint angles. To define the position and direction of the entire body, the coordinate system origin and the representative link q0 are considered to be connected by a virtual joint with six degrees of freedom (translation and rotation). The rigid link model acquisition function unit 16 has the function of obtaining the motion information of the above three-dimensional rigid link model by performing inverse kinematic calculations on the three-dimensional rigid link model.

[0039] The rigid link model acquisition function unit 16 further recognizes and acquires body landmarks 50 in the three-dimensional rigid link model 17. The body landmarks 50 are similar to the body landmarks 50C recognized in the color image C described above, and are characteristic local structures of the human body that are relatively important in analyzing a person's posture. In the three-dimensional rigid link model 17, the body landmarks 50 can be provided at any position of each link 30 or joint 32 on the three-dimensional rigid link model 17. The body landmarks 50 are provided, for example, at positions corresponding to all or part of the joints between the links 30 (shoulder joints, elbow joints, wrists, hip joints, knee joints, ankles, etc.), eyes, nose, parietal region, acromion, anterior superior iliac spine, greater trochanter, heel, and toes.

[0040] Given the position q in the generalized coordinate system of the rigid link model and determining the pose of the rigid link model, the three-dimensional position of this body landmark 50 can be calculated using forward kinematics. Assuming there are M body landmarks 50, each landmark 50 is identified by a number: 0, 1, 2, ..., m, ..., M-1. In this case, the three-dimensional position of body landmark m can be expressed as Pm(q) as a function of q. Pm(q) is a three-dimensional vector whose elements are the x, y, and z coordinates of body landmark m. Changing the position q in the generalized coordinate system changes the position, orientation, and pose of the entire body, and the positions of the body landmarks also change. The first-order time derivative of the position q in the generalized coordinate system can be expressed as follows: This corresponds to the velocity in the generalized coordinate system. Similarly, the second derivative can be expressed as follows, and the expression of the second derivative corresponds to acceleration. Therefore, the dynamics of the whole body can be described by the following equation of motion for a rigid link system. The left side shows the force required to move the body, and the right side shows the force exerted by the person and the external force, and the left and right sides are in balance. The term H(q) denotes the inertia matrix in the generalized coordinate system, The term indicates the terms including gravity, centrifugal force, and Coriolis force. indicates the joint torque, fc indicates the external force (ground reaction force), and G indicates the transformation matrix that converts fc into the generalized coordinate system.

[0041] In this technology, we distinguish between forward kinematics, inverse kinematics, forward dynamics, and inverse dynamics. Forward kinematics calculates the position and orientation of each rigid link from the position q in a generalized coordinate system. For example, it calculates the position of the hand from the joint angle. Inverse kinematics calculates the position q in a generalized coordinate system from the position and orientation of each rigid link. For example, it calculates the joint angle from the position of the hand. Forward dynamics is The acceleration is calculated from the joint torque and the external force (floor reaction force) of For example, the acceleration of the hand is calculated from the joint torque. Inverse dynamics is calculated from the motion and external force fc. The joint torque is calculated from the floor reaction force, for example. It is used to calculate the joint torque and reaction force using a force plate, for example. Note that the motion in inverse dynamics is, for example, the position, velocity, and acceleration in a generalized coordinate system. In this way, inverse dynamics calculation is a calculation that calculates the joint torque and joint reaction force from all or part of the position, velocity, acceleration, joint angle, angular velocity, angular acceleration, and floor reaction force. Due to the nature of the calculation, errors in the motion state data used in inverse dynamics calculation tend to produce relatively large errors in the calculation of joint torque and reaction force. Therefore, it is preferable to reduce the errors in the motion state data used in inverse dynamics calculation as much as possible.

[0042] The joint load estimation device 10 may further include a rigid link model adjustment function unit 28 that causes the computer to execute a rigid link model adjustment step of adjusting the dimensions of the three-dimensional rigid link model acquired by the rigid link model acquisition step by estimating the length of a specified link based on an image of the subject.

[0043] The rigid link system motion state acquisition function unit 18 estimates the motion state of the three-dimensional rigid link model 17 acquired in the rigid link model acquisition step in accordance with the posture information of the human body of the subject 2 acquired in the base motion state acquisition step, and executes the rigid link system motion state acquisition step of acquiring posture information, which is motion information of the three-dimensional rigid link model 17. The rigid link system motion state acquisition function unit 18 is constituted by a part of the joint load estimation program.

[0044] The fitting of such a three-dimensional rigid link model 17 (rigid link system fitting) will now be described. A process is performed to fit the position Pm(q) of the body landmark 50 of the rigid link system model to the three-dimensional position Pm* of the body landmark 50C estimated by the base motion state acquisition function unit 14. This fitting process is realized by solving an optimization problem that minimizes the difference between the position Pm* of the body landmark 50C and the position Pm(q) of the body landmark 50 set in the rigid link system. As explained above regarding the rigid link system, the position Pm(q) of the body landmark 50 is determined by determining the position q in the generalized coordinate system. The optimization problem is to find q such that Pm* and Pm(q) coincide. Note that the rigid link system motion state acquisition function unit 18 may have a function that makes it easier to find a solution by providing a regularization term in the optimization problem when the number of body landmarks is small.

[0045] The body landmarks 50C and body landmarks 50 indicate feature points that are important in analyzing the motion state of a human body, such as the joints, eyes, ears, top of the head, toes, fingertips, etc. The joint load estimation device 10 further includes a rigid link model application function unit 19 that executes a rigid link model application function that maintains constant link lengths between joints of the three-dimensional rigid link model even during motion, based on the motion information of the three-dimensional rigid link model 17. The rigid link model application function unit 19 can estimate the motion of the links while maintaining constant lengths of each link as a rigid body.

[0046] As shown in Figure 6, the joint load calculation function unit 60 further includes a floor reaction force estimation function unit 62 that estimates floor reaction force, an inverse dynamics calculation function unit 64 that performs inverse dynamics calculation, a muscle tension estimation function unit 66 that estimates muscle tension, a joint load estimation function unit 68, and an alarm function unit 61 that alarms when the joint reaction force falls outside a specified range.

[0047] The floor reaction force estimating function unit 62 can estimate the floor reaction force. The floor reaction force is represented by a vector fc that combines the floor reaction forces (for example, acting upward from the soles of the feet) that act on the lower parts of the left and right feet as forces received from the floor. For example, fc,0 and fc,1 represent the floor reaction force vectors acting on the left and right feet. The vector fc summarizing the ground reaction forces may include external forces other than the ground reaction force vector acting on the feet, such as external forces acting on a cane held by the subject, or external forces acting on the hand.

[0048] Here, there are two ways to define each floor reaction force vector fc,i, depending on whether the point of application of the external force is known or not. 1. When the point of application of the external force is unknown When the point of application of the external force is unknown, each floor reaction force fc,i is expressed as a six-dimensional vector as follows. Here, indicate the translational forces in the X-axis, Y-axis, and Z-axis directions in three-dimensional space, respectively. indicate the rotational forces (moments) around the X-axis, Y-axis, and Z-axis in three-dimensional space, respectively. 2. When the point of application of the external force is clear When the point of application of the external force is clear, the individual ground reaction forces can be expressed as a four-dimensional vector as follows, where indicate the translational forces in the X-axis, Y-axis, and Z-axis directions in three-dimensional space, respectively. indicates the rotational force (moment) around the Z axis in three-dimensional space.

[0049] The floor reaction force estimation function unit 62 estimates fc, which is a collection of floor reaction force vectors, by solving an optimization problem to calculate the floor reaction force. Let q be the position in the generalized coordinate system, and fc be the velocity. , joint torque Using these variables, the calculated acceleration is given by can be obtained. where: is a forward dynamics calculation of a rigid link system, and can be calculated using the method shown in [Roy Featherstone, 2008], i.e., R. Featherstone, Rigid body dynamics algorithms. New York, NY: Springer, 2008. This calculated acceleration and the measured acceleration, which is the output of the generalized coordinate system position q of the rigid link system at time t. fc is found by solving the following optimization problem that minimizes the difference between The objective function of this optimization problem is teeth, This term acts as a regularization term to uniquely determine the solution to the optimization problem. teeth, is a diagonal matrix with positive weights corresponding to each element.

[0050] In this technology In addition to this, one more regularization term is added: This term is the weighted sum of squares of the joint torque. teeth is a diagonal matrix with positive weights corresponding to each element. The guideline is to minimize the joint torque (the force exerted by the person) as much as possible, and to This term is expected to prevent specific joint torques from becoming excessively large and to estimate ground reaction forces that are closer to reality.

[0051] Next, we consider the calculation of the ground reaction force (constraints). A constraint is set for the following. This constraint assumes that the part in contact with the floor does not slip, and restricts each ground reaction force vector to be contained within the friction cone. The optimization problem including this constraint is as follows. However, is a three-dimensional vector corresponding to the translational force component of the ground reaction force, μ is the friction coefficient, and u is a unit vector with a norm of 1 indicating the normal direction of the contact surface. It is calculated under the following constraints: For any i, In addition, "i" in fc,i is used as a subscript to identify the external force (floor reaction force), "j" is used as a subscript to identify the joint for f, and similarly "k" is used as a subscript to identify the muscle tension for f. Here, If the friction is within the friction cone, the component of the force in the direction parallel to the contact surface But the friction force Since the friction cone is smaller than the ground reaction force vector, the contacting part will not slip. On the other hand, if the ground reaction force vector goes outside the friction cone, the contacting part will slip. In other examples of constraint conditions, the friction cone that restricts the range of the ground reaction force vector may be approximated by a polygonal cone. Also, when the range of the contact surface is clear, such as the sole of the foot touching the floor, the center of pressure of the contact surface can be calculated by The point of action may be set to the ground reaction force vector, and constraints may be placed on the translational force component and rotational force (moment) component so that the center of pressure falls within the contact surface.

[0052] Next, the inverse dynamics calculation function unit 64 executes the inverse dynamics calculation. The inverse dynamics calculation function unit 64 calculates the measured acceleration q, which is the output of the generalized coordinate system position q of the rigid link system at time t. and the ground reaction force vectors are summarized as follows: and joint torques are calculated by inverse dynamics calculation. and joint forces Here, the subscript j identifies the Nj joints of the rigid link system, where j=0, 1, 2, ..., N-1. Figure 17 illustrates the relationship between the joint torque and the joint force acting on the joint j (for example, the knee joint) of the three-dimensional rigid link model 17. Joint force is the force acting between rigid links via joints, and is a six-dimensional vector as follows: are translational forces in the X-axis, Y-axis, and Z-axis directions in three-dimensional space, respectively. are rotational forces (moments) around the X-axis, Y-axis, and Z-axis in three-dimensional space, respectively. Therefore, the joint forces are It is shown as follows.

[0053] Here, we will explain the joint force, focusing on how it differs from the joint reaction force. The joint force includes all force components acting between rigid links. This is necessary for the calculation algorithm described above. The joint force can be analyzed by decomposing it into a joint torque, such as a driving force, and a joint reaction force. Therefore, each element of the joint force includes the joint reaction force and the driving force. The driving force here corresponds to the force exerted by, for example, human muscles and is called the joint torque. It is a force acting in the direction of joint movement. In contrast, the joint reaction force is, for example, a component acting in a direction other than the direction of joint movement. The joint reaction force can also be considered a constraint force of the joint. For example, if a force is applied to the knee in a direction other than bending and straightening, it will not move in that direction, and a resistance force will act. This is the joint reaction force. If this force is too large, it can cause joint destruction, such as fractures or dislocations. Note that in this embodiment, muscle tension is not taken into account in the joint reaction force. By adding the influence of muscle tension to the joint reaction force, more accurate joint load calculations are possible.

[0054] Next, the muscle tension estimation function unit 66 estimates muscle tension. As shown in Figure 18, a muscle model that generates unidirectional tension in a rigid link system that models the human body is considered. This muscle model has a starting point and an end point fixed at a specific location on any rigid link. It is possible to generate tension in the direction of the line segment connecting the starting point and the end point. Assuming there are K muscle models, and the tension of each is An example of the configuration of such a muscle model (muscle force model) is shown in FIG. 18. FIG. 18 illustrates the relationship between the links corresponding to the lower limbs in the three-dimensional rigid link model 17 and the muscle models. In this example, only the main muscles of the lower limbs are modeled. For example, as shown in FIG. 18, the muscle model corresponding to the links of the lower limbs models all or some of the following muscles: f1: rectus femoris, f2: iliopsoas, f3: gluteus maximus, f4: medial hamstrings, f5: vastus medialis, f6: short head of biceps femoris, f7: gastrocnemius, f8: soleus, and f9: tibialis anterior. The level of detail of the muscle model configuration can be changed depending on the purpose. For muscle tension estimation, the tension of all muscle models is calculated. The vector that summarizes It can be shown as follows: Using the joint torque in the generalized coordinate system calculated by inverse dynamics calculation, muscle tension is calculated by solving the following optimization problem. where X is is a transformation matrix that transforms the joint torque in the generalized coordinate system. teeth, is a diagonal matrix with positive weights corresponding to each element. As a prerequisite, for any k, When calculating joint loads, the step of estimating muscle tension can be omitted, in which case joint loads are calculated without taking muscle tension into account.

[0055] Next, the joint load estimation function unit 68 has a function of estimating the joint load. The joint load is a joint reaction force, that is, a force component such as a compressive force or a shear force acting on the soft tissue of the joint, and is also called joint mechanical stress. The joint load estimation function unit 68 has a compressive force calculation function 67 that calculates a compressive force as the joint reaction force. The joint load estimation function unit 68 also has a shear force calculation function 69 that calculates a shear force as the joint reaction force. The joint load is calculated using the following formula: This can be generalized as: where, are the translational forces along the X, Y, and Z axes in the coordinate system of a particular link, respectively. are the rotational forces (moments) around the X-axis, Y-axis, and Z-axis, respectively. Joint load is the joint force calculated by inverse dynamics calculation and the muscle tension calculated by muscle tension calculation Calculate using the following formula: where: is muscle tension is a transformation matrix that converts the joint force into the rigid link system. This formula allows you to calculate the joint load taking into account the dynamics and tension of the rigid link system. If you want to omit the inverse dynamics calculation, you can use the following: is expressed as follows, which is the joint load excluding muscle tension. The joint force acting between links (bones) and the joint reaction force acting on the joint tissue (soft tissue) are in an action-reaction relationship, so they can basically be calculated using the same method.

[0056] Next, specific calculation of compressive force and shear force among joint loads will be described with reference to Figures 19 to 21 using an example of element extraction of hip joint load. Figure 19 is a diagram explaining elements of knee joint load in joint load estimation performed by a joint load estimation device in one embodiment of the present invention. Figure 20 is a diagram explaining elements of knee joint load in joint load estimation performed by a joint load estimation device in one embodiment of the present invention. Figure 21 is a diagram explaining elements of knee joint load in joint load estimation performed by a joint load estimation device in one embodiment of the present invention. Calculated joint load contains all the force components transmitted between the rigid links through the joint. For intuitive understanding, we can also convert the joint loads into a form that is easier to interpret anatomically, for example. In FIG. 19, for the sake of explanation, the X-axis and Z-axis are used to convert the Similarly, in FIG. 20, for the sake of explanation, the y-axis and Z-axis are used. In FIG. 21, the components of the y-axis and the z-axis are shown. or In this way, for example, in the case of a knee joint, the coordinate system O on the lower leg side of the knee joint is Convert the converted is an element of corresponds to the compressive force acting on the joint tissue. is the shear force acting on the joint tissue, corresponds to the moment in the varus-valgus direction. corresponds to the joint torque, which is the joint driving force, and is excluded from the joint load because it is a component of the force exerted by the person, not the joint load. corresponds to the moment in the twisting direction. into the coordinate system of the part of interest and extract elements as appropriate, it is possible to obtain indices related to joint diseases, such as compressive force and shear force, as specific numerical values. In this embodiment, a method for obtaining indices of joint load, such as compressive force, shear force, and moment, has been described using the knee joint as an example, but it is also possible to apply a similar theory to obtain compressive force, shear force, etc. for other joints, such as the hip joint and waist joint.

[0057] The notification function unit 61 notifies the subject when the joint reaction force falls outside a predetermined range, e.g., when the joint reaction force falls outside the normal load range and a large load is applied that may cause pain to the subject. For example, the notification function unit 61 notifies the subject that the joint reaction force falls outside the predetermined range by displaying text or pictures on the display device 38 or by audio. The joint load estimation system 1 can calculate joint loads such as joint reaction forces as specific numerical values ​​in real time. For example, the joint load estimation system 1 can display (visualize) joint loads such as joint reaction forces as specific numerical values ​​or images based on numerical values ​​in approximately real time from the measurement by the camera 4 (e.g., by continuously displaying the joint load with a delay of about one second from the measurement by the camera 4). Therefore, the notification function unit 61 can notify the subject that the joint reaction force falls outside the predetermined range based on specific calculation grounds in approximately real time, making it easier for the subject to compare their own joint sensations with the joint load estimation results.

[0058] Next, a series of operations related to the joint load estimation system and the joint load estimation method of this embodiment will be described with reference to Figures 11 to 16. Referring to Figure 11, when the joint load estimation device 10 starts control at a certain time t1, in S1, the base motion state acquisition function unit 14 executes a base motion state acquisition step of acquiring information about the motion state of the subject 2, for example, posture information of the human body, based on an image (color image C and / or depth image D) captured by the camera 4.

[0059] The base motion state acquisition step S1 can include, for example, the following steps S1a to S1c. Specifically, as shown in S1a in Fig. 11, the joint load estimation device 10 uses the camera 4 to capture and acquire a color image C (see Fig. 7) of the subject and a depth image D (see Fig. 9) corresponding to the color image. The color image C and the depth image D are acquired by the camera 4 almost simultaneously for the moving subject 2. When S1a ends, the joint load estimation device 10 proceeds to S1b.

[0060] Next, as shown in S1b, the joint load estimation device 10 uses the body landmark estimation function unit 20 of the joint load estimation device 10 to estimate body landmarks 50C in the image of the subject in color image C (see FIG. 7 ). The body landmark estimation function unit 20 estimates, recognizes, and stores body landmarks 50C in the image of the subject, such as joints (hip joints, knee joints, ankles, etc.), eyes, parietal region, acromion, anterior superior iliac spine, greater trochanter, and toes. Furthermore, the position estimation function unit 22 of the body landmark estimation function unit 20 recognizes and stores the body landmarks 50C as position information in a two-dimensional coordinate system, such as a pixel coordinate system. The body landmark estimation function unit 20 estimates the body landmarks 50C using an AI program such as machine learning. The position information of the body landmarks 50C is stored in the joint load estimation device 10.

[0061] As shown in S1c, the joint load estimation device 10 uses the three-dimensional coordinate estimation unit 24 to combine the position information of the body landmark 50C estimated by the position estimation function unit 22 with the information of the depth image D to convert the body landmark 50C into three-dimensional coordinates. Thus, the position of the body landmark 50C is acquired by the joint load estimation device 10 as position data in a three-dimensional coordinate system that also combines depth information. When S1c is completed, the base motion state acquisition step S1 ends and the process proceeds to S3. Note that the base motion state acquisition step S1 does not need to include all of steps S1a to S1c, and may be composed of some or a combination of parts thereof.

[0062] Next, S2 will be described with reference to Figures 11 and 12. After the start of a series of control processes, the joint load estimation device 10 causes the rigid link model acquisition function unit 16 to execute a rigid link model acquisition step in S2, in which the rigid link model acquisition function unit 16 acquires a three-dimensional rigid link model 17 based on predetermined feature quantities related to the human body. Note that the rigid link model acquisition step S2 does not necessarily have to be executed after the start of a series of control processes, and may be executed separately in advance to acquire the three-dimensional rigid link model 17. Conversely, the rigid link model acquisition step S2 may be executed following S1.

[0063] 12, the rigid link model acquisition step S2 may include, for example, steps S2a to S2d. The rigid link model acquisition step S2 does not need to include all of steps S2a to S2d, and may be configured with some or a combination of parts thereof.

[0064] 11, the joint load estimation device 10 starts the rigid link model acquisition step S2 and then proceeds to S2a. In S2a, the joint load estimation device 10 uses the rigid link model acquisition function unit 16 to acquire a three-dimensional rigid link model 17 based on predetermined features related to the human body, such as height. For example, the rigid link model acquisition function unit 16 reads out and sets a typical three-dimensional rigid link model corresponding to height from a stored table. When S2a ends, the joint load estimation device 10 proceeds to S2b.

[0065] Next, in S2b, the joint load estimation device 10 further adjusts the three-dimensional rigid link model 17 acquired in S2a based on the captured images. For example, the rigid link model adjustment function unit 28 captures color images C (and / or depth images D, if necessary) of the subject standing still in an upright position within a range of approximately 1 to approximately 100 frames, for example, from the front, and acquires them. Any direction other than the front may be used as long as the subject is visible. The rigid link model adjustment function unit 28 may also capture images of postures other than the standing position and use these images as reference for analyzing the length of the links 52. For example, if the position of the joints is difficult to determine due to the state of clothing, etc., including images of postures other than the standing position can make it easier to bring the three-dimensional rigid link model 17 closer to the subject. The three-dimensional rigid link model 17 is defined so that the length of each link does not change and remains constant because the length of each link is rigid.

[0066] Next, in S2b, as shown in Fig. 15A, the rigid link model adjustment function unit 28 determines the ratio of the main links 52C of the photographed subject 2 based on the photographed image of the subject 2 standing still. For example, the rigid link model adjustment function unit 28 can estimate the body landmarks 50C in the photographed image and estimate the ratio of the length of the links 52C between the body landmarks 50C to the total height. For example, the total length from the top of the head to the toes can be calculated as k ｈ When the length k1 of the link 52C in the torso portion, the length k2 of the link 52C from the femur to the knee, and the length k3 of the link 52C from the knee to the toe can be calculated as the ratio to the total length k. In this case, by repeatedly capturing and analyzing still images of the standing position within a range of 1 to 100 frames, it is possible to reduce errors in length measurement. If such ratios can be estimated, the actual length of a given link can be calculated from the actual height L of the subject input by the user. For example, as shown in FIG. 15B, the actual length of the link 52 in the torso portion is L1 ＊ = L * k1 / k ｈ , the actual length L2 of the link 52 from the femur to the knee ＊ = L * k2 / k ｈ , the actual length L3 of the link 52 from the knee to the toe ＊ = L * k3 / kｈ , the actual length of each link 52 can be estimated and calculated as follows. The rigid body link model adjustment function unit 28 estimates at least the major predetermined links that occupy a relatively large portion of the human body. When S2b ends, the joint load estimation device proceeds to S2c. The major predetermined links are, for example, a trunk link, a pelvis link, a right thigh link, a left thigh link, a right lower leg link, a left lower leg link, a right foot link, a left foot link, etc. The major predetermined links may be some of these links, or may be any links including these links.

[0067] As shown in S2c, the rigid link model adjustment function unit 28 adjusts the provisional three-dimensional rigid link model 17 acquired in S2a based on the actual length of each link calculated based on the captured image. For example, the lengths of links corresponding to major parts of the provisional three-dimensional rigid link model 17 acquired in S2a, such as the torso link 52, the femur-to-knee link 52, and the knee-to-toe link 52, are adjusted to correspond to their actual lengths. For example, the links 52 corresponding to major skeletons are links corresponding to skeletons between the acquired body landmarks 50. As a result, the three-dimensional rigid link model 17 is formed to be relatively close to the physique of the photographed subject 2. The lengths of the major links of the three-dimensional rigid link model 17 are within a range of −1.5% to +1.5% of the lengths of the corresponding parts of the actual subject 2. After S2c is completed, the joint load estimation device proceeds to S2d.

[0068] As a modified example, instead of performing steps S2b and S2c described above, the rigid link model adjustment function unit 28 can also perform a function to adjust the length of a predetermined link (e.g., a relatively large primary link) in frames capturing different poses using the average link length of each image. First, instead of performing step S2b described above, the rigid link model adjustment function unit 28 acquires color images and depth images of the subject 2 in different poses (postures). For example, the rigid link model adjustment function unit 28 repeatedly captures N frames, e.g., a number of frames ranging from 4 to 100. It is preferable that the subject 2 assume a different pose (posture) in each frame. Based on the images of the multiple poses captured in this manner, the length of the primary link 52C of the captured subject 2 is estimated. For example, the subject may be photographed walking at regular intervals, with each captured frame representing a different pose.

[0069] Next, to perform the scale adjustment S2c of the 3D rigid link model, the rigid link model adjustment function unit 28 estimates the positional relationship of the body landmarks 50C from the color image or the color image and the depth image, and calculates the length of each link of the rigid link model from the positional relationship of the body landmarks 50C. Alternatively, the ratio of each link length of the rigid link model to the total height may be estimated. Because errors may occur in measuring the position of the body landmarks 50C, the link length error can be reduced by calculating the average link length of multiple poses. For example, each link is identified by a subscript r = 0, 1, 2, ..., R-1. The link length of link r in the nth frame is denoted as kn,r. In the following equation, kh is the distance between the body landmarks at the top of the head and the toes when standing, which corresponds to the height. L is the actual measured height of the subject input by the user. Kr corresponds to the average value for each frame. In this way, the actual length of each link is calculated based on the average of the link lengths corresponding to the various poses. In addition, if the actual measured value L of the subject's height cannot be obtained, In this way, the rigid link model adjustment function unit 28 may have a function that can estimate the actual length of each link. In this case, by repeatedly photographing and analyzing up to N frames of the subject, it is possible to reduce errors in length measurement. Even in this modified example, the rigid link model adjustment function unit 28 proceeds to S2d after completing the scale adjustment of the three-dimensional rigid link model corresponding to S2c.

[0070] In this embodiment, as shown in S2d, the joint load estimation device 10 acquires the adjusted three-dimensional rigid link model 17 using the rigid link model adjustment function unit 28. When S2d is completed, the rigid link model acquisition step S2 ends and the process proceeds to S3. Note that the joint load estimation device 10 can arbitrarily calculate the posture, etc. of the acquired three-dimensional rigid link model 17 and execute the rigid link system motion state acquisition step S3, which will be described later.

[0071] 10 , in S3, the joint load estimation device 10 executes a rigid link system motion state acquisition step S3 in which the rigid link system motion state acquisition function unit 18 estimates the motion of the three-dimensional rigid link model 17 in accordance with the motion state of the photographed subject acquired in the base motion state acquisition step, and acquires motion information of the three-dimensional rigid link model 17. For example, by applying the body landmarks 50 of the adjusted three-dimensional rigid link model 17 to the body landmarks 50C in the photographed image, the three-dimensional rigid link model 17 is used to suit the subject to be analyzed, thereby improving the accuracy of the finally calculated position data, etc. Note that the rigid link system motion state acquisition step S3 is not limited to estimating the motion of the entire three-dimensional rigid link model, and may also estimate the motion of a portion of it. For example, the rigid link system motion state acquisition step S3 may estimate only the motion of the lower body side of the three-dimensional rigid link model by matching the body landmarks 50C relating to the hip joints, knee joints, ankles, and toes of the subject with the body landmarks 50 relating to the hip joints, knee joints, ankles, and toes of the three-dimensional rigid link model 17.

[0072] The joint load estimation device 10, for example, associates the three-dimensional coordinate positions of the subject's body landmarks 50C (see FIG. 8 ) obtained in S1c with the three-dimensional coordinate positions of the body landmarks 50 of the three-dimensional rigid link model 17 (see FIG. 9 ) obtained in S2d. Therefore, a process is performed to fit the three-dimensional rigid link model to the three-dimensional positions (Pm*) of the subject's body landmarks as shown in FIG. 8 . This fitting process is realized, for example, by solving an optimization problem that minimizes the difference between the three-dimensional position (Pm*) and the three-dimensional position (Pm(q)) of the body landmark 50 of the three-dimensional rigid link model. At this time, the three-dimensional position (Pm(q)) of the body landmark 50 is determined by determining the position q in the generalized coordinate system. Therefore, the optimization problem is to find q such that Pm* and Pm(q) coincide. The method for formulating the optimization problem can be changed depending on the number of body landmarks and the degree of freedom of the rigid link system.

[0073] In this way, in S3, the joint load estimation device 10 uses the rigid link system motion state acquisition function unit 18 to acquire motion information of the three-dimensional rigid link model 17 at a level where errors are suppressed to a level that makes it applicable to inverse dynamics calculations. At this time, in S3, the joint load estimation device 10 uses the rigid link system motion state acquisition function unit 18 to acquire the three-dimensional positions and generalized coordinate system positions of the body landmarks 50 of the three-dimensional rigid link model at a certain time t (e.g., time t1). When S3 ends, the joint load estimation device 10 proceeds to END. The acquired motion state data is used for inverse dynamics calculations by other functional units of the joint load estimation device 10, allowing the torque, velocity, joint angle, etc. associated with each joint to be calculated with a predetermined accuracy. For example, in the acquired motion state data, the error in the lengths of links corresponding to the same part is suppressed to within approximately ±5% of the initial length, more preferably ±3%.

[0074] Next, with reference to FIG. 13 etc., a process for estimating the motion of the three-dimensional rigid link model 17 over time and acquiring the position in the generalized coordinate system of the rigid link system will be described. FIG. 13 shows how the position in the generalized coordinate system of the rigid link system is acquired over time. As shown in FIG. 13 , the joint load estimation device 10 acquires three-dimensional coordinates of body landmarks based on a color image or the like at a certain time t1 (indicated by S1(t) in FIG. 12 ), acquires an adjusted three-dimensional rigid link model 17 at time t1 (indicated by S2(t) in FIG. 13 ), and further acquires the position in the generalized coordinate system of the rigid link system (indicated by S3(t) in FIG. 13 ). As shown in S11(t), the joint load estimation device 10 saves the position q of the generalized coordinate system of the rigid link system at time t. After executing S11(t), the joint load estimation device 10 proceeds to S1(t).

[0075] As shown in Figure 13, the joint load estimation device 10 similarly executes steps S1, S2, and S3 at time t2, which is t1 + Δt, to obtain the position of the generalized coordinate system of the rigid link system. At time t2, the joint load estimation device 10 also acquires three-dimensional coordinates of body landmarks based on color images, etc. (shown by S1(t) in Figure 13). At time t2, the three-dimensional rigid link model 17 shown by S2(t) is almost unchanged from that at time t1, so the joint load estimation device 10 can reuse the three-dimensional rigid link model 17 obtained at time t1 (shown by S2(t) in Figure 13). Based on S1(t) and S2(t), the joint load estimation device 10 obtains the position of the generalized coordinate system of the rigid link system at time t2 (shown by S3(t) in Figure 13). After executing S3(t), the joint load estimation device 10 saves the position q of the generalized coordinate system of the rigid link system at time t, as shown in S11(t). After S11(t) is executed, the joint load estimation device 10 proceeds to S1(t) again. In this way, the joint load estimation device 10 can acquire the position of the generalized coordinate system of the rigid link system for each time t, and can estimate the motion state over time. The joint load estimation device 10 can also acquire the position of the generalized coordinate system of the rigid link system within the time range to be analyzed.

[0076] In this way, the joint load estimation device 10 can acquire the position of the rigid link system in the generalized coordinate system for each time t, and estimate the motion state over time. In the motion state estimated by the joint load estimation device 10, the length of each link as a part (the length of the link between joints) remains approximately constant, even if the posture and position change over time in accordance with the motion state. Based on the motion state data obtained by the joint load estimation device 10, the torque, velocity, joint angle, etc. related to each joint can be calculated with a predetermined accuracy by inverse dynamics calculation. In this way, the joint load estimation device makes it easy to obtain motion state data that is easy to use in inverse dynamics calculation. Note that when estimating the motion of the three-dimensional rigid link model 17 over time and only acquiring the position of the rigid link system in the generalized coordinate system, estimation of the motion of the three-dimensional rigid link model 17 is repeated over time, as shown in FIG. 13 .

[0077] Next, with reference to FIG. 14 and other figures, a process for estimating joint loads after the position of the generalized coordinate system of the rigid link system has been acquired over time will be described. FIG. 14 shows how joint loads are estimated after the position of the generalized coordinate system of the rigid link system has been acquired over time. As shown in FIG. 14, the joint load estimation device 10 acquires three-dimensional coordinates of body landmarks based on a color image or the like at a certain time t1 (indicated by S1(t) in FIG. 14), acquires an adjusted three-dimensional rigid link model 17 at time t1 (indicated by S2(t) in FIG. 14), and further acquires the position of the generalized coordinate system of the rigid link system (indicated by S3(t) in FIG. 14). As shown in S4(t), the joint load estimation device 10 estimates the floor reaction force fc, joint torque τj, joint force fj, and joint load fs at time t, and estimates the load of forces acting on the joints. After executing S4(t), the joint load estimation device 10 proceeds to S1(t).

[0078] As shown in Figure 14, the joint load estimation device 10 similarly executes steps S1, S2, S3, and S4 at time t2, which is t1 + Δt, to estimate the force load on the joint. In this way, the joint load estimation device 10 executes S3(t) and then estimates the joint reaction force on the joint tissue at time t, as shown in S4(t). Details of step S4 will be given below. After executing S4(t), the joint load estimation device 10 again proceeds to S1(t). In this way, the joint load estimation device 10 can estimate the motion state over time for each time t, and can estimate the joint reaction force on the joint tissue. The joint load estimation device 10 can estimate the joint reaction force on the joint tissue within the time range to be analyzed.

[0079] In the motion state estimated by the joint load estimation device 10, the length of each link as a part (the length of the link between joints) remains approximately constant even if the posture and position change over time to match the motion state. Based on the motion state data obtained by the joint load estimation device 10, the torque, speed, joint angle, etc. related to each joint can be calculated with a predetermined accuracy by inverse dynamics calculation, and the joint reaction force related to the joint tissue can be estimated.

[0080] After executing S3 in FIG. 11 (S3 in FIG. 14), the joint load estimation device 10 proceeds to step S4 as shown in FIG. 14 to estimate the joint load. The joint load calculation function unit 60 can execute the joint load estimation process after executing the motion state estimation process as shown in FIGS. 11 and 14. As shown in FIG. 16, after starting the joint load estimation process, the joint load calculation function unit 60 acquires the position, velocity, acceleration, etc. in a generalized coordinate system by inverse kinematics calculation of a rigid link system as a preparatory step in S4a. Note that detailed explanation of the formulas and formulas will be omitted as they overlap with the explanation above. After acquiring the position, velocity, acceleration, etc. in the generalized coordinate system, the joint load calculation function unit 60 proceeds to S4b.

[0081] In S4b, the joint load calculation function unit 60 estimates the floor reaction force using the floor reaction force estimation function unit 62. As described in the explanation of the floor reaction force estimation function unit, the floor reaction force is estimated based on the calculated acceleration and the measured acceleration, which is the output of the generalized coordinate system position q of the rigid link system at time t. fc is found by solving the following optimization problem that minimizes the difference between After S4b is executed, the joint load calculation function unit 60 proceeds to S4c.

[0082] In S4c, the joint load calculation function unit 60 executes inverse dynamics calculation using the inverse dynamics calculation function unit 64. As described above, the inverse dynamics calculation function unit 64 calculates the measured acceleration q, which is the output of the generalized coordinate system position q of the rigid link system at time t. and the ground reaction force vectors are summarized as follows: Using this, joint torque is calculated by inverse dynamics calculation. and joint forces After executing S4c, the inverse dynamics calculation function unit 64 proceeds to S4d.

[0083] In S4d, the joint load calculation function unit 60 estimates the tension of the muscle model by the muscle tension estimation function unit 66. The muscle tension is calculated by solving a predetermined optimization problem using the joint torque calculated by the inverse dynamics calculation. After executing S4d, the joint load calculation function unit 60 proceeds to S4e.

[0084] In S4e, the joint load calculation function unit 60 calculates the joint load by the joint load estimation function unit 68. The joint load estimation function unit 68 calculates the joint load by the joint load The joint force calculated by inverse dynamics calculation and the muscle tension calculated by muscle tension calculation The calculated joint load is calculated using a predetermined formula. By converting the coordinate system into an arbitrary coordinate system, the compressive and shear forces acting on the joint tissues can be calculated as specific parameters. Therefore, the joint load calculation function unit 60 calculates the joint reaction forces, such as compressive and shear forces, acting on the joint tissues by inverse dynamics calculation based on the motion information of the above-mentioned three-dimensional rigid link model. This technology calculates the joint reaction forces, such as compressive and shear forces, acting on the joint tissues based on images measured by the camera 4. This allows for relatively easy visualization of joint loads without the need for diagnostic tests using X-rays or other methods at a hospital. Therefore, this technology can visualize the joint loads, which are the root cause of deformation and inflammation, before the joints become significantly deformed or inflamed, and can be used for early detection and prevention before treatment or diagnosis. For example, this technology can be used for preventive purposes, preventing symptoms from worsening until surgery is required.

[0085] According to one embodiment of the present invention configured as described above, the motion state of the three-dimensional rigid link model 17 is estimated in accordance with the motion state of the subject acquired in the base motion state acquisition step based on the captured image, and the joint load calculation step calculates the joint reaction force acting on the joint tissue by inverse dynamics calculation based on the motion information of the three-dimensional rigid link model 17. As a result, even if the motion state of the subject is acquired based on a captured image, the link lengths and connection relationships in the motion information of the three-dimensional rigid link model 17 can be maintained at predetermined lengths and relationships, and the joint reaction force acting on the joint tissue can be calculated by inverse dynamics calculation based on the motion information, allowing the load on the force acting on the joints of the body to be analyzed. Furthermore, advice and services can be provided to the subject based on specific numerical values ​​of the joint reaction force.

[0086] According to one embodiment of the present invention configured as described above, the joint load calculation step calculates a compressive force as the joint reaction force. As a result, even if the motion state of the subject is acquired based on photographed images of the motion state of the subject, the link lengths and connection relationships in the motion information of the three-dimensional rigid link model 17 can be maintained at predetermined lengths and relationships, the compressive force acting on the joint tissue can be calculated by inverse dynamics calculation based on the motion information, and the compressive load acting on the joints of the body can be analyzed.

[0087] According to one embodiment of the present invention configured as described above, the joint load calculation step calculates shear force as the joint reaction force. As a result, even if the motion state of the subject is acquired based on photographed images of the motion state of the subject, the link lengths and connection relationships in the motion information of the three-dimensional rigid link model 17 can be maintained at predetermined lengths and relationships, the shear force acting on the joint tissue can be calculated by inverse dynamics calculation based on the human motion information, and the shear force load acting on the joints of the body can be analyzed.

[0088] According to one embodiment of the present invention configured as described above, the joint load calculation step calculates a torsional moment as the joint reaction force. As a result, even if the subject's motion state is acquired based on photographed images of the subject's motion state, the link lengths and connection relationships in the motion information of the three-dimensional rigid link model 17 can be maintained at predetermined lengths and relationships, the torsional moment acting on the joint tissue can be calculated by inverse dynamics calculation based on the human motion information, and the load of the torsional moment acting on the joints of the body can be analyzed.

[0089] According to one embodiment of the present invention configured as described above, the joint load calculation step includes a floor reaction force estimation step of estimating the floor reaction force based on parameters such as the position and velocity in the generalized coordinate system of the three-dimensional rigid link model 17, without relying on measurements of the floor reaction force by a floor reaction force meter. This makes it possible to estimate the floor reaction force of the subject based on images capturing the subject's motion state.

[0090] According to one embodiment of the present invention configured as described above, the joint load calculation step includes an inverse dynamics calculation estimation step of estimating joint forces that are the basis for joint reaction forces by inverse dynamics calculation using the acceleration in the generalized coordinate system of the three-dimensional rigid link model and the estimated ground reaction force. This makes it possible to estimate joint forces acting on joints between links of the three-dimensional rigid link model 17 based on images capturing the motion state of the subject.

[0091] According to one embodiment of the present invention configured as described above, the joint load calculation step includes a muscle tension estimation step of estimating tension in accordance with the muscle force model based on a muscle force model corresponding to a predetermined link of the three-dimensional rigid link model 17. This makes it possible to correct the joint force obtained in the inverse dynamics calculation estimation step by taking into account the muscle tension element based on the muscle force model, thereby enabling more accurate estimation of the joint load.

[0092] According to one embodiment of the present invention configured as described above, the rigid link model adjustment step allows the three-dimensional rigid link model 17 to be adjusted based on images of the subject. This makes it easier to match the dimensions of the three-dimensional rigid link model 17 to the physique of the subject being photographed, and the accuracy of the motion information of the three-dimensional rigid link model is likely to be improved. Therefore, motion state data with an accuracy that is easy to use in inverse dynamics calculations can be more easily obtained, and the joint reaction force acting on the joint tissues can be calculated by inverse dynamics calculations that analyze the load of forces acting on the joints of the human body.

[0093] According to one embodiment of the present invention configured as described above, the method further includes a notification step of notifying the subject that the joint reaction force is outside the predetermined range when the joint reaction force calculated in the joint load calculation step is outside the predetermined range. This allows the subject to easily recognize that the joint reaction force is outside the predetermined range from the image of the subject, making it easier to recognize points to note about the subject's walking style, etc.

[0094] According to one embodiment of the present invention configured as described above, the rigid link system motion state acquisition step estimates the motion information of the three-dimensional rigid link model by performing inverse kinematics calculation of the three-dimensional rigid link model 17. This makes it possible to estimate the motion state of the three-dimensional rigid link model 17 from an image obtained by a camera, and the motion information obtained by the forward dynamics calculation can be used for the inverse dynamics calculation.

[0095] According to one embodiment of the present invention configured in this manner, a computer-readable recording medium having the above-described joint load estimation program recorded thereon can be used to cause a computer to execute predetermined steps, thereby estimating the force load acting on the joints of the human body.

[0096] According to one embodiment of the present invention configured as described above, the motion state of the three-dimensional rigid link model 17 is estimated in accordance with the motion state of the subject acquired by the base motion state acquisition function unit based on captured images, and the joint load calculation step calculates the joint reaction force acting on the joint tissues by inverse dynamics calculation based on the motion information of the three-dimensional rigid link model 17. As a result, even if the motion state of the subject is acquired based on captured images, the link lengths and connection relationships in the motion information of the three-dimensional rigid link model 17 can be maintained at predetermined lengths and relationships, and the joint reaction force acting on the joint tissues can be calculated by inverse dynamics calculation based on the motion information, allowing the load on the force acting on the joints of the body to be analyzed. Furthermore, advice and services can be provided to the subject based on specific numerical values ​​of the joint reaction force.

[0097] According to one embodiment of the present invention configured as described above, the motion state of the three-dimensional rigid link model 17 is estimated in accordance with the motion state of the subject acquired in the base motion state acquisition step based on the captured image, and the joint load calculation step calculates the joint reaction force acting on the joint tissue by inverse dynamics calculation based on the motion information of the three-dimensional rigid link model 17. As a result, even if the motion state of the subject is acquired based on a captured image, the link lengths and connection relationships in the motion information of the three-dimensional rigid link model 17 can be maintained at predetermined lengths and relationships, and the joint reaction force acting on the joint tissue can be calculated by inverse dynamics calculation based on the motion information, allowing the load on the force acting on the joints of the body to be analyzed. Furthermore, advice and services can be provided to the subject based on specific numerical values ​​of the joint reaction force.

[0098] The embodiments for implementing the present invention are not limited to those described above, and other modifications may be applied. Various alternative embodiments and examples will be apparent to those skilled in the art based on the disclosed technology. According to one modification, the camera 4 is not limited to an RGBD camera, but may be an RGB camera. In this case, as shown in FIG. 22 , even if the camera 4 cannot capture a depth image, depth information in the image may be estimated from an RGB color image using AI or the like. This modification differs from the previous embodiment in that depth information is estimated. Since the other aspects are basically the same as the previous embodiment, a description of the similar parts will be omitted.

[0099] As shown in FIG. 22 , the joint load estimation device 10 includes a depth estimation function unit that executes a depth estimation step in S21, after acquiring a color image in S1a in the base motion state acquisition step S1. The depth estimation function unit executes a depth estimation step in S21, estimating depth information in the depth direction in the image based on the color image. For example, the joint load estimation device 10 includes a depth information estimation program that estimates depth information in the depth direction in the image. For example, the joint load estimation device 10 can estimate a depth image (or depth information) from an RGB image. This depth information estimation program (AI program) is generated in advance as a trained model using machine learning or the like so as to be able to estimate depth information in the depth direction in the RGB image. After executing S21, the process proceeds to S1c. In S1c, as described above, the joint load estimation device 10 uses the three-dimensional coordinate estimation unit 24 to combine the position information of the body landmark 50 estimated by the position estimation function unit 22 with the depth information obtained in S21 to generate three-dimensional coordinates of the body landmark 50.

[0100] As another modification, in the base motion state acquisition step S1, the joint load estimation device 10 may omit S21 as shown in FIG. 22 after acquiring a color image in S1a. That is, S1a, S1b, and S1c may be executed in order, and in S1c, the joint load estimation device 10 may convert the body landmarks 50 into two-dimensional coordinates based on the position information of the body landmarks 50. Thus, by estimating and applying a three-dimensional rigid link model 17 to the two-dimensional coordinate conversion of the body landmarks 50 in S1c, it is possible to obtain three-dimensional motion state data of the subject with accuracy that is easy to use in inverse dynamics calculations. In this modification, the three-dimensional motion state of the subject obtained by applying the three-dimensional rigid link model 17 to the two-dimensional coordinate conversion of the body landmarks 50 in S1c may reflect the actual motion state with lower accuracy than the three-dimensional motion state of the subject obtained by applying the three-dimensional rigid link model 17 to the three-dimensional coordinate conversion of the body landmarks 50 in S1c as in the first embodiment. However, the configuration of this modified example is also effective in that it is possible to obtain three-dimensional motion state data of a subject with a precision that is easy to use in inverse dynamics calculations using a simpler method.

[0101] As another variation, as shown in FIG. 23 , after a color image is acquired, the three-dimensional coordinate positions of the body landmarks 50C may be estimated directly from the acquired image using AI or the like. This variation differs from the previously described embodiment in that the three-dimensional coordinate positions of the body landmarks 50C are directly estimated from the acquired image without estimating two-dimensional coordinates or depth information. Since the other aspects are basically the same as the previously described embodiment, a description of the similar parts will be omitted. The motion state estimation device 10 in this variation includes a three-dimensional coordinate estimation unit 24 that directly estimates position information of the body landmarks 50 in a three-dimensional coordinate system using AI or the like, using the captured image. The three-dimensional coordinate estimation unit 24 includes a program that causes a computer to execute the step of directly estimating this position information. As shown in FIG. 23 , in the base motion state acquisition step S1, after acquiring a color image in S1a, the three-dimensional coordinate estimation unit 24 directly estimates position information of the body landmarks 50C in the three-dimensional coordinate system using AI or the like, using the acquired image, in S1c. For example, the three-dimensional coordinate estimation unit 24 can estimate position information of the physical landmark 50C in a three-dimensional coordinate system from, for example, an RGB image. The program (AI program) of this three-dimensional coordinate estimation unit 24 is a machine learning model that has been trained in advance so as to be able to estimate position information of the physical landmark 50C in the three-dimensional coordinate system from an image. Therefore, in S1c, the three-dimensional coordinate estimation unit 24 converts the physical landmark 50 into three-dimensional coordinates based on the captured image of the subject 2.

[0102] As another variation, as shown in FIG. 24 , after acquiring a color image in S1a, the three-dimensional coordinate positions of the body landmarks 50C may be estimated using AI or the like based on the two-dimensional coordinate positions estimated from the acquired image. This variation differs from the previously described embodiment in that the three-dimensional coordinate positions of the body landmarks 50C are estimated based on the two-dimensional coordinate positions estimated from the acquired image. Since the remaining aspects are essentially the same as the previously described embodiment, a description of the similar parts will be omitted. In the base motion state acquisition step S1, after acquiring a color image in S1a, the motion state estimation device 10 may omit S21 as shown in FIG. 22 and estimate the three-dimensional coordinate positions of the body landmarks 50C from the estimation of the body landmarks 50C as shown in S1b. For example, S1a, S1b, and S1c are executed in order. In S1b, the body landmark estimation function unit 20 equipped with an AI program estimates the two-dimensional coordinate positions of the body landmarks 50C in the subject's image in color image C (see FIG. 7 ). Next, in S1c, the three-dimensional coordinate position of the physical landmark 50C may be estimated from the two-dimensional coordinate position of the physical landmark 50C by the three-dimensional coordinate estimation unit 24 equipped with an AI program. The motion state estimation device 10 includes such a physical landmark estimation function unit 20 and a three-dimensional coordinate estimation unit 24. The physical landmark estimation function unit 20 and the three-dimensional coordinate estimation unit 24 each include a program that causes a computer to execute a step of estimating a two-dimensional coordinate position and a step of estimating a three-dimensional coordinate position. The program (AI program) of the physical landmark estimation function unit 20 is a machine learning model trained in advance to be able to estimate the two-dimensional coordinate position of the physical landmark 50C in an image of the subject from an image. The program (AI program) of the three-dimensional coordinate estimation unit 24 is a machine learning model trained in advance to be able to estimate position information of the physical landmark 50 in a three-dimensional coordinate system from the two-dimensional coordinate position of the physical landmark 50C. Therefore, in S1c, the three-dimensional coordinate estimation unit 24 performs three-dimensional coordinate conversion of the physical landmark 50.Therefore, by estimating and applying the three-dimensional rigid link model 17 to the three-dimensional coordinate positions of the body landmarks 50 obtained in S1c, it is possible to obtain three-dimensional motion state data of the subject with accuracy that is easy to use in inverse dynamics calculations. In this modified example, the three-dimensional motion state of the subject obtained by applying the three-dimensional rigid link model 17 to the three-dimensional coordinate positions based on the two-dimensional coordinate positions of the body landmarks 50C in S1c may reflect the actual motion state with lower accuracy than the three-dimensional motion state of the subject obtained by applying the three-dimensional rigid link model 17 to the three-dimensional coordinate conversion of the body landmarks 50C in S1c as in the first embodiment. However, the configuration of this modified example is also effective in that it allows for a simpler method to obtain three-dimensional motion state data of the subject with accuracy that is easy to use in inverse dynamics calculations.

Claims

1. A joint load estimation program for estimating the load of forces acting on joints of the body based on captured images, the program causing a computer to execute the following steps: a base motion state acquisition step for acquiring the motion state of a subject based on captured images of the subject's motion state; a rigid body link model acquisition step for acquiring a three-dimensional rigid body link model based on predetermined features related to the subject's body; a rigid body link system motion state acquisition step for estimating the motion state of the three-dimensional rigid body link model in accordance with the motion state of the subject acquired in the base motion state acquisition step and acquiring motion information of the three-dimensional rigid body link model; and a joint load calculation step for calculating the joint reaction force acting on joint tissues by inverse dynamics calculation based on the motion information of the three-dimensional rigid body link model.

2. The joint load estimation program according to claim 1, wherein the joint load calculation step calculates a compressive force as the joint reaction force.

3. The joint load estimation program according to claim 1, wherein the joint load calculation step calculates a shear force as the joint reaction force.

4. The joint load estimation program according to claim 1, wherein said joint load calculation step calculates a torsional moment as said joint reaction force.

5. A joint load estimation program as described in claim 1, wherein the joint load calculation step includes a floor reaction force estimation step that estimates the floor reaction force based on the position and velocity parameters in a generalized coordinate system of the three-dimensional rigid link model, without measuring the floor reaction force using a floor reaction force meter.

6. A joint load estimation program as described in claim 5, wherein the joint load calculation step includes an inverse dynamics calculation step of estimating joint forces by inverse dynamics calculation using the acceleration in a generalized coordinate system of the three-dimensional rigid link model and the estimated floor reaction force.

7. A joint load estimation program as described in claim 6, wherein the joint load calculation step includes a muscle tension estimation step that estimates tension in accordance with the muscle force model based on a muscle force model corresponding to a specified link of the three-dimensional rigid link model.

8. The joint load estimation program according to claim 1, further comprising a rigid link model adjustment step for adjusting the dimensions of the acquired three-dimensional rigid link model by estimating the lengths of at least some of the links based on an image of the subject.

9. The joint load estimation program according to claim 1, further comprising a notification step of notifying the user that the joint reaction force is outside a predetermined range when the joint reaction force calculated in the joint load calculation step is outside the predetermined range.

10. A joint load estimation program as described in claim 1, wherein the rigid link system motion state acquisition step estimates the motion information of the three-dimensional rigid link model by performing inverse kinematics calculations of the three-dimensional rigid link model.

11. A computer-readable recording medium on which the joint load estimation program according to any one of claims 1 to 10 is recorded.

12. A joint load estimation device that estimates the load of forces acting on body joints based on captured images, comprising: a base motion state acquisition function unit that acquires the motion state of a subject based on images captured by a camera; a rigid link model acquisition function unit that acquires a three-dimensional rigid link model based on predetermined features related to the subject's body; a rigid link system motion state acquisition function unit that estimates the motion state of the three-dimensional rigid link model acquired by the rigid link model acquisition function unit in accordance with the motion state of the subject and acquires motion information of the three-dimensional rigid link model; and a joint load calculation function unit that calculates the joint reaction force acting on joint tissues by inverse dynamics calculation based on the motion information of the three-dimensional rigid link model.

13. A joint load estimation method for estimating the load of forces acting on joints of a body based on captured images, comprising: a base motion state acquisition step for acquiring the motion state of the subject based on captured images of the motion state of the subject; a rigid link model acquisition step for acquiring a three-dimensional rigid link model based on predetermined features related to the subject's body; a rigid link system motion state acquisition step for estimating the motion state of the three-dimensional rigid link model in accordance with the motion state of the subject acquired in the base motion state acquisition step and acquiring motion information of the three-dimensional rigid link model; and a joint load calculation step for calculating the joint reaction force acting on joint tissues by inverse dynamics calculation based on the motion information of the three-dimensional rigid link model.

Images