Information processing device, information processing method, and information processing program

Abstract

Provided are: an information processing device that has a simple device configuration, achieves installation space saving, achieves a reduction in effort required for preparation, and is usable for estimation of at least one of joint torque and muscle tension; an information processing method; and an information processing program. A motion analysis device 13 includes a two-dimensional information estimation unit 22, a three-dimensional information estimation unit 23, a derivation unit 26, a subject person muscle skeleton model generation unit 27, and an external force estimation unit 28. The two-dimensional information estimation unit 22 estimates a two-dimensional joint position from frame images of a moving image obtained by capturing a subject person under analysis as an imaging subject. The three-dimensional information estimation unit 23 estimates motion information including first data in the dimension of position and second data in the dimension of velocity, and estimates the three-dimensional position and the angle of a floor surface. The derivation unit derives third data in the dimension of acceleration, and the subject person muscle skeleton model generation unit 27 generates a subject person muscle skeleton model. The external force estimation unit estimates an external force from an object surface, which includes the floor surface, in contact with the subject person under analysis on the basis of the first to third data, and the three-dimensional position and the angle of the object surface.

Description

Information processing device, information processing method, and information processing program

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program.

[0002] There are technologies that estimate joint torque, muscle tension, and other parameters from human motion measured by motion capture. Motion measurement can be achieved using an inertial measurement unit (IMU) and multiple video cameras. For example, Patent Document 1 (Patent Document 1) describes an information processing device including an acquisition unit and a conversion unit. The acquisition unit acquires position-dimensional data related to an object expressed in a generalized coordinate system. The conversion unit acquires at least acceleration-dimensional data from multiple inertial sensors attached to the object and converts the acquired acceleration data into acceleration-dimensional data expressed in the generalized coordinate system based on the position-dimensional data related to the object expressed in the generalized coordinate system. The object includes multiple segments and joints connecting two or more of the segments, and the generalized coordinate system includes variables for the rotation angle around at least one axis for each joint. This information processing device further includes an estimation unit. The estimation unit further includes an estimation unit that estimates at least one of an external force acting on the object and a torque generated at a joint, based on position dimension data expressed in the generalized coordinate system, velocity dimension data related to the object obtained by differentiating the position dimension data, and acceleration dimension data converted by the conversion unit. The estimation unit estimates at least one of an external force acting on the object and a torque generated at a joint by performing one or both of a forward dynamics calculation and an inverse dynamics calculation.

[0003] Other methods for measuring movement include optical motion capture and using multiple video cameras.

[0004] Japanese Patent Application Laid-Open No. 2020-201138

[0005] However, when an inertial sensor is used as in the information processing device described in Patent Document 1, various problems arise, such as the large number of parts required for the device, the need to secure a large space for installation of the device when in use, and the need for preparatory steps such as calibration before measurement. These problems still remain even with optical motion capture and methods that measure movement using multiple video cameras.

[0006] Therefore, an object of the present invention is to provide an information processing device, an information processing method, and an information processing program that have a simple device configuration, require less space for installation, and require less preparation effort, and can be used to estimate at least one of joint torque and muscle tension.

[0007] The information processing device of the present invention includes a two-dimensional information estimation unit, a three-dimensional information estimation unit, a derivation unit, a subject musculoskeletal model generation unit, and an external force estimation unit. The two-dimensional information estimation unit estimates two-dimensional time series data of the subject's two-dimensional joint positions from a plurality of frame images constituting a moving image of the subject. The three-dimensional information estimation unit estimates motion information based on the estimated two-dimensional time series data, which smoothly displaces along the time series and includes first data in the position dimension and second data in the velocity dimension expressed in a generalized coordinate system, and estimates the three-dimensional position and angle of the floor surface based on the motion information. The derivation unit derives third data in the acceleration dimension expressed in the generalized coordinate system by first differentiating the second data. The subject musculoskeletal model generation unit generates the subject musculoskeletal model by applying the height and weight of the subject to the general musculoskeletal model. The external force estimation unit estimates the external force from the surface of the object that is in contact with the subject of analysis through dynamic analysis based on the first data and the second data, the three-dimensional position and angle of the object surface including the floor surface, the third data, and the subject's musculoskeletal model.

[0008] Preferably, the information processing device further includes a dynamics analysis unit and a muscle information estimation unit. The dynamics analysis unit performs dynamics analysis based on the external force and the first to third data, and calculates each joint torque of a plurality of joints connecting each of the plurality of segments. The muscle information estimation unit estimates at least one of muscle activity and muscle tension based on the first and second data, the subject's musculoskeletal model, and each joint torque.

[0009] Preferably, the external force estimation unit determines whether each contact point, set on a part of the body of the subject of analysis that is in contact with the object surface, is in contact with the object surface based on the relative speed and distance of the contact point from the object surface, and if it is determined that each contact point is in contact, estimates the external force at the contact point. The object surface may be a floor surface. If the object surface is a floor surface, the external force estimation unit determines whether each contact point is in contact with the floor based on the speed and height from the floor of multiple contact points set on the soles of the feet, and if it is determined that each contact point is in contact, estimates the external force at the contact point.

[0010] The information processing method of the present invention includes a two-dimensional information estimation step, a three-dimensional information estimation step, a derivation step, a subject musculoskeletal model generation step, and an external force estimation step. The two-dimensional information estimation step estimates two-dimensional time series data of two-dimensional joint positions of the subject from a plurality of frame images constituting a moving image of the subject of analysis. The three-dimensional information estimation step estimates motion information based on the estimated two-dimensional time series data, which smoothly displaces along the time series and includes first data in the position dimension and second data in the velocity dimension expressed in a generalized coordinate system, and estimates the three-dimensional position and angle of the floor surface based on this motion information. The derivation step derives third data in the acceleration dimension expressed in a generalized coordinate system by first differentiating the second data. The subject musculoskeletal model generation step generates the subject musculoskeletal model by applying the height and weight of the subject of analysis to the generalized musculoskeletal model. The external force estimation step estimates the external force from the surface of the object that is in contact with the subject of analysis by dynamic analysis based on the first data, the second data, the three-dimensional position and angle of the object surface including the floor surface, the third data, and the subject's musculoskeletal model.

[0011] The information processing program of the present invention causes a computer to execute the above steps.

[0012] According to the present invention, the device has a simple configuration, requires less space for installation, and requires less preparation effort, and can be used to estimate at least one of joint torque and muscle tension.

[0013] 1 is an explanatory diagram of a motion analysis system including a motion analysis device according to an embodiment;

[0014] The motion analysis system 10 shown in FIG. 1 is an example of an information processing system that analyzes the motion of a subject, specifically, the motion of at least a portion of the subject's body. The motion analysis system 10 includes a user terminal 11 used by the subject, a motion analysis device 13, which is an example of an information processing device, and a client terminal 15 that acquires analysis results and displays images showing the analysis results. The user terminal 11, the motion analysis device 13, and the client terminal 15 communicate with each other, i.e., send and receive various data, via a communication network CN. For example, the motion analysis system 10 uses an imaging device (camera) 17 (see FIG. 2) installed in the user terminal 11 to capture video images of the subject, and the motion analysis device 13 analyzes the subject's motion based on the captured video data (hereinafter referred to as video image data) and transmits the analysis results to the client terminal 15.

[0015] The user terminal 11 includes an imaging device 17 that captures moving images of the subject of analysis during exercise, and a transmission unit (not shown) that transmits the captured moving image data to the motion analysis device 13. In this example, a smartphone is used as the user terminal 11, but any device equipped with the imaging device and transmission unit is acceptable. One imaging device 17 is sufficient for one user (the subject of analysis in this example). While FIG. 1 depicts only one user terminal 11, multiple user terminals 11 may be used by multiple users. The user terminal 11 is installed so that the imaging device 17 is mounted in a fixed position and orientation. As a result, moving images are obtained as fixed-point captured moving images captured from a fixed point. The imaging device 17 is positioned so that the entire body of the subject of analysis moving from the start to the end of the exercise to be analyzed is captured.

[0016] The client terminal 15 is, for example, a terminal used by a trainer who provides physical training guidance to the subject of analysis, but the user of the client terminal 15 is not limited to this example. The motion analysis system 10 may not include a client terminal 15. For example, if the user terminal 11 includes a display unit for displaying images and a receiving unit for acquiring (receiving) image data, and the subject of analysis views the analysis results related to his or her own motion, the client terminal 15 may not be included. While only one client terminal 15 is depicted in FIG. 1 , there may be multiple client terminals 15, each used by multiple clients, such as trainers. The client terminal 15 may also be used as a management terminal for managing various settings and / or data of the motion analysis device 13. The management terminal may also be provided separately from the client terminal 15.

[0017] The motion analysis device 13 is for analyzing the motion of the subject based on the acquired video image data. Details of the motion analysis device 13 will be described later with reference to another drawing.

[0018] The user terminal 11, the client terminal 15, and the motion analysis device 13 are each composed of a computer. The user terminal 11 and the client terminal 15 may operate on a browser by receiving a program that runs on a browser from the motion analysis device 13, or may be equipped with predetermined application software and operate by executing the program of this application software. The motion analysis device 13 is equipped with a predetermined program, and by executing this program, it functions as each unit described below and performs predetermined processing. The motion analysis device 13 performs predetermined processing in response to input of each piece of information from the user terminal 11 and the client terminal 15, respectively.

[0019] The program causes a computer to execute a two-dimensional information estimation step, a three-dimensional information estimation step, a derivation step, a subject musculoskeletal model generation step, and an external force estimation step. The two-dimensional information estimation step estimates two-dimensional time series data of two-dimensional joint positions of the subject from multiple frame images constituting a moving image of the subject of analysis. The three-dimensional information estimation step estimates motion information that smoothly displaces along the time series and includes first data in the position dimension and second data in the velocity dimension expressed in a generalized coordinate system based on the estimated two-dimensional time series data, and estimates the three-dimensional position and angle of the floor surface based on this motion information. The derivation step derives third data in the acceleration dimension expressed in a generalized coordinate system by first differentiating the second data. The subject musculoskeletal model generation step generates the subject musculoskeletal model by applying the height and weight of the subject of analysis to the generalized musculoskeletal model. The external force estimation step estimates the external force from the surface of the object that is in contact with the subject of analysis by dynamic analysis based on the first data, the second data, the three-dimensional position and angle of the object surface including the floor surface, the third data, and the subject's musculoskeletal model.

[0020] 2 , the motion analysis device 13 includes an acquisition unit 21, a two-dimensional information estimation unit 22, a three-dimensional information estimation unit 23, a derivation unit 26, a subject musculoskeletal model generation unit 27, and an external force estimation unit 28. The motion analysis device 13 preferably further includes a dynamics analysis unit 31 and a muscle information estimation unit 32, as is done in this example. The derivation unit 26, the subject musculoskeletal model generation unit 27, the external force estimation unit 28, the dynamics analysis unit 31, the muscle information estimation unit 32, etc. constitute a musculoskeletal analysis unit 35 that analyzes and determines muscle information including at least one of muscle activity and muscle tension based on three-dimensional information about the subject to be analyzed.

[0021] The acquisition unit 21 acquires moving image data sent from the imaging device 17. The moving image data is composed of a plurality of frame images arranged in chronological order. The acquisition unit 21 outputs the plurality of frame images arranged in chronological order to the 2D information estimation unit 22 as frame time-series data.

[0022] The two-dimensional information estimation unit 22 estimates time-series data (hereinafter referred to as two-dimensional time-series data) of the subject's two-dimensional joint positions (hereinafter referred to as two-dimensional joint positions) as two-dimensional information based on the frame time-series data. That is, the two-dimensional information estimation unit 22 estimates the subject's two-dimensional joint positions in each of the multiple frame images that make up the frame time-series data, and generates two-dimensional time-series data that is time-series data of the estimated two-dimensional joint positions. This two-dimensional time-series data is the two-dimensional information.

[0023] The two-dimensional joint positions are the two-dimensional positions of multiple joints, such as elbows, wrists, finger joints, neck, and knees, in each frame image of the time series data of multiple frame images input from the acquisition unit 21.

[0024] The two-dimensional joint positions are expressed in an image coordinate system. In the image coordinate system, the vertex located at the leftmost and uppermost corner of the image area defined on the xy plane is set as the origin (0,0), and the direction toward the right is treated as the positive direction of the x-axis (positive values ​​gradually increase), and the direction toward the bottom is treated as the negative direction of the y-axis (negative values ​​gradually decrease). The two-dimensional information estimation unit 21 outputs the estimated two-dimensional information to the three-dimensional information estimation unit 23.

[0025] The method for estimating the two-dimensional information is not particularly limited, and any known method can be used. For example, the two-dimensional joint positions can be estimated by regression, as in DeepPose (described in the following Non-Patent Document 1: A. Toshev and C. Szegedy, "DeepPose: Human Pose Estimation via Deep Neural Networks," 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, Ohio, USA, 2014, pp. 1653-1660.), in which time-series data of frame images constituting previously acquired video data is input into a convolutional neural network, and each joint point on the image coordinate system is estimated by regression. Two-dimensional joint positions may be estimated using an improved method that improves estimation accuracy by representing the probability distribution of joint positions as a heat map, such as OpenPose (described in the following non-patent document 2: Z. Cao, G. Hidalgo, T Simon, S. Wei, and Y. Sheikh. 2021. “OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields,” IEEE Trans. Pattern Anal. Mach. Intell. 43, 1 (Jan. 2021), 172-186.), or by grouping candidate joint points.

[0026] Because the two-dimensional information estimation unit 22 obtains two-dimensional information from moving images captured by only one imaging device 17, the only equipment installed under the subject of analysis is a single smartphone serving as the user terminal 11. This saves space and simplifies the device configuration. Furthermore, because it is only necessary to capture images of the subject's movements using the imaging device 17 of the user terminal 11, no preparations such as equipment calibration are required, and therefore preparations for starting the analysis are also simplified.

[0027] When two-dimensional information is input from the two-dimensional information estimation unit 22, the three-dimensional information estimation unit 23 responds to this input and estimates movement information and the three-dimensional position and angle of the floor surface based on the two-dimensional information. The movement information includes first data of a position dimension and second data of a velocity dimension for the person being analyzed, which smoothly displace over time and are expressed in a generalized coordinate system. For example, when time-series data of two-dimensional joint positions, i.e., two-dimensional time-series data, is received as two-dimensional information, the three-dimensional information estimation unit 23 generates first data of a position dimension and second data of a velocity dimension, which are expressed in a generalized coordinate system and in which the two-dimensional joint positions smoothly displace over time. The three-dimensional information estimation unit 23 estimates the three-dimensional position and angle (angle with respect to the horizontal plane) of the floor surface based on the movement information. The first data and second data are time-series data, and the first data is calculated using the q function described below. ｍ The second data is q ｍ The three-dimensional information estimation unit 23 outputs the estimated first data and second data to the derivation unit 26, and outputs the three-dimensional position and angle of the floor surface to the external force estimation unit 28.

[0028] Here, a generalized coordinate system will be described. A generalized coordinate system is a coordinate system capable of expressing the posture of a subject of analysis using variables corresponding to the degrees of freedom of the model when the subject of analysis is modeled. In this embodiment, the generalized coordinate system is similar to the generalized coordinate system described in the above-mentioned Patent Document 1, in which, for example, the head, chest, abdomen, pelvis, left and right thighs, left and right shins, and left and right feet are defined as segments SG, and joints JT(i) connecting the segments are defined (where i = 1 to N, where N is a natural number equal to or greater than 2). Of the segments SG, the segment SG(B) corresponding to the pelvis is defined as the base segment SG(B). The six degrees of freedom displacement of segments SG other than the base segment SG(B) relative to the origin of the absolute coordinate system are not variables of the generalized coordinate system, and only the base segment SG(B) is treated as being displaceable with six degrees of freedom. The six degrees of freedom include three translational directions (XYZ) and three rotational directions (yaw, roll, and pitch).

[0029] The posture of the subject of analysis expressed in the generalized coordinate system (hereinafter referred to as the "generalized position") q ｍis a set of variables with a dimension of position, and is expressed, for example, by the following formula (1). In the following explanation, ' denotes a first derivative, and '' denotes a second derivative. In formula (1), q ０ is a vector with seven components in total, consisting of four elements: the X coordinate, Y coordinate, and Z coordinate of the base segment SG(B), and a quaternion representing the orientation. ｉ is a vector of rotation angles with a number of dimensions corresponding to the degrees of freedom of each joint JT(i) (where i = 1 to N). For example, a joint corresponding to a knee is defined as having two degrees of freedom, the bending and extension direction of the knee and the twisting direction of the shin, and is therefore expressed as a two-dimensional vector. A joint corresponding to a hip joint is defined as having three degrees of freedom, and is therefore expressed as a three-dimensional vector.

[0030]

[0031] The three-dimensional information estimation unit 23 is a machine learning model that generates, for example, positional dimension data of the analysis subject expressed in a three-dimensional generalized coordinate system based on the two-dimensional joint positions that constitute the two-dimensional information input from the two-dimensional information estimation unit 22. The three-dimensional information estimation unit 23 uses a data generation method such as CVAE (conditional variation autoencoder) to generate generalized positions q ｍ and q ｍ The CVAE illustrated here generates three-dimensional motion information of the subject of analysis smoothly in time series and in consistency with the floor surface. Here, the motion information q ｍ and q ｍ Let ' be the state variable x.

[0032]

[0033] CVAE is a machine learning model that generates new data based on information about a certain condition. In this example, x, which is a state variable x at time t, ｔ In order to generate x, which is the state variable x at time t-1, ｔ－１ is the condition. That is, CVAE is x ｔ－１When input, the next state variable x with the highest probability is ｔ is generated as a motion state. This probability can be obtained from a probability distribution obtained by learning a dataset containing time-series data of multiple motions (time-series motion data). This probability distribution is a priori probability distribution learned in the latent space of machine learning, and is expressed as the latent variable z ｔ Therefore, the probability distribution for the learning parameter θ has its mean value μ θ and standard deviation σ θ Using the above, it can be expressed by the Gaussian distribution formula of the following formula (3): In formula (3) and formulas (11) and (12) described later, N (italic letters in the formulas) represents the Gaussian distribution.

[0034]

[0035] Furthermore x ｔ and Z ｔ The relationship between x and ｔ－１ From x ｔ Using the amount of change Δθ from

[0036]

[0037] Therefore, x ｔ－１ Since Δθ is derived from the information of x ｔ The motion information at t = 0, x ０ Since x cannot be derived from the above equation (4), it may be generated by creating a probability distribution of x at time t=0 using a Gaussian mixture model.

[0038] There are many x in learning CVAE ｔ and x ｔ－１ The pair of q and q is used as a data set of motion data (motion data set). For example, the motion data is a human q expressed in a generalized coordinate system calculated using inverse kinematics calculation based on marker information acquired by optical motion capture. ｍ and q ｍ Alternatively, the image may be acquired by a method using an IMU (Inertial Measurement Unit) or a video camera.

[0039] In CVAE learning, a learning parameter θ is searched for that minimizes the loss function L. The loss function L can be expressed by the following equation (5).

[0040]

[0041] L ｒｅｃ is the training data x ｔ and x ｔ－１ and x reconstructed based on ｔ The L2 norm of the hat is expressed by equation (6).

[0042]

[0043] L ｊｏｉｎｔ is the training data x ｔ The joint position J of the rigid link model obtained from ｔ and the reconstructed x ｔ The joint position J of the rigid link model obtained from the hat ｔ The L2 norm of the hat is expressed by Equation 7. ｔ is t time q ｍ q ｍｔ are the X, Y, and Z coordinate components of the

[0044]

[0045] L ｃｏｎｔａｃｔ is a loss function that evaluates the contact point of the foot with the floor surface and the ground contact state with the floor surface. ｔ At the same time, a variable c indicates the probability that the contact point is in contact with the floor surface. ｔ The training data c of the ground state, which is binary data, can be reconstructed by estimating the hat. ｔ [0,1] is x ｔ It is possible to determine the contact point position and speed data obtained from the above equation. It is preferable to set a first threshold value TH1 for the height of the contact point from the floor surface and a second threshold value TH2 for the speed of the contact point in advance, and determine that the contact point has touched the ground when the height of the contact point from the floor surface is equal to or less than TH1 and the speed of the contact point is equal to or less than TH2. There are no particular limitations on TH1 and TH2, but in this example, TH1 is set to, for example, 0.05 m and the second threshold value is set to 0.8 m / s to determine that the contact point has touched the ground. Lｃｏｎｔａｃｔ Is L ＢＣＥ , L ｖｅｌ and can be expressed by equation (8). ＢＣＥ is calculated using binary cross-entropy from training data c ｔ and reconstructed c ｔ This improves the accuracy of CVAE estimation of the ground contact state. ｖｅｌ is expressed by equation (9), and the velocity of the contact point in contact state is v ｔ It acts to suppress the increase of hat. Note that the contact point may be set to a part of the body (contact part) that comes into contact with an object surface, such as the seat surface of a bicycle saddle or the surface of a wall, and the floor surface may be replaced with the object surface. In other words, the floor surface is an example of an object surface that the body comes into contact with. "Coming into contact" means that contact is planned and / or expected.

[0046]

[0047]

[0048] The Gaussian mixture model can be trained using the same motion data set as in the CVAE. Here, x is calculated based on the two-dimensional joint positions acquired from the two-dimensional information estimation unit 22. ｔ A method for estimating hat will now be described.

[0049] The trained CVAE is a time series of three-dimensional motion information x from the observed two-dimensional joint positions to the final observation time T. ０：Ｔ First, we use the Gaussian mixture model to generate x ０ Generate x in CVAE. ｔ－１ When this input is input, z ｔ is sampled and x ｔ Therefore, x is generated in CVAE. ０ When input is made, z １ is sampled and x １ By repeating this process until time T, the latent variable z and the state variable x from t = 1 to t = T are generated. １：Ｔ is sampled and x １：ＴAt this point, x ０：Ｔ does not match the information of the observed two-dimensional joint position, and an error occurs. Therefore, we need to find the x ０ and z １：Ｔ The final estimated motion information x is obtained by performing an optimization calculation to find the optimal solution of (x). Furthermore, since the observed two-dimensional joint positions do not include information about the floor position, the floor position dimension data g is also optimized in the same way. The entire objective function can be expressed by the following equation (10).

[0050]

[0051]

[0052] ε ｍｏｔ is expressed by equation (11), and ε ＣＶＡＥ and ε ｉｎｉｔ As expressed by the following equation (12), the purpose is to maximize the log likelihood for the probability distribution of the CVAE and the Gaussian mixture model, and the closer to the center of the probability distribution, the smaller the optimization cost. Hereinafter, λ is the weight for each term.

[0053]

[0054] ε in Equation (10) ｄａｔａ is an estimate of the loss of observed data. ｄａｔａ is expressed by the following equation when the two-dimensional joint positions are input to the three-dimensional information estimation unit 23:

[0055]

[0056] In equation (13), when "j" is the joint position, P ｔ ｊ is the three-dimensional position of the joint j, and Π represents the pinhole projection, which is a function that projects the joint j onto a two-dimensional image coordinate system using the position and focal length information of the imaging device 17 used for the image capture. ｔ ｊis the two-dimensional joint position on the observed image coordinate system. Therefore, the joint position information calculated from the motion information generated by the optimization calculation is projected two-dimensionally, and equation (13) works to minimize the error with the two-dimensional joint position calculated by the two-dimensional information estimation unit 22. σ ｔ ｊ is the reliability of the estimated 2D joint position, and more reliable observation data is weighted more heavily. ρ is the Geman-McClure function, which is used to improve robustness against outliers.

[0057] ε in equation (10) ｒｅｇ is composed of the following elements:

[0058]

[0059] ε ｓｋｅｌ and ε ｅｎｖ is a term set so that the motion information generated from the CVAE is smooth in time series and is consistent with the floor surface.

[0060]

[0061] l ｔ ｉ and ｔ－ｌ ｉ is the length of the segment calculated from the generated motion information, and acts to reduce fluctuations from time t to time t-1.

[0062]

[0063] c ｔ is the contact point p generated by the CVAE ｔ ｊ The first term of equation (16) is p ｔ ｊ The second term of equation (16) is the contact point p ｔ ｊ When is in the ground state, p ｔ ｊ The height z from the floor of the object is set to be lower than the threshold value δ.

[0064] ε ｇｎｄ is the initial value g in the optimization of the floor position data ｉｎｉｔThe change from the initial value g is suppressed by the following equation (17). ｉｎｉｔ In this example, the horizontal plane is at a height of 0 m.

[0065]

[0066] By solving the optimization problem using the objective function described above, three-dimensional time-series motion information x expressed in a generalized coordinate system is obtained. ０：Ｔ Estimate.

[0067] When the derivation unit 26 receives the second data from the three-dimensional information estimation unit 23, the derivation unit 26 first differentiates the second data in response to this input, thereby deriving third data of the dimension of acceleration expressed in the generalized coordinate system. This third data is time-series data. The derivation unit 26 in this example calculates the generalized velocity [q ｍ ]' is first differentiated, and the generalized acceleration [q ｍ The derivation unit 26 associates the first data and second data input from the three-dimensional information estimation unit 23 with the derived third data, and outputs the associated data to the external force estimation unit 28 and the dynamics analysis unit 31. The derivation unit 26 also outputs the first data and second data input from the three-dimensional information estimation unit 23 to the muscle information estimation unit 32.

[0068] The subject musculoskeletal model generation unit 27 generates a subject musculoskeletal model by applying the height and weight of the subject to a general musculoskeletal model. The subject musculoskeletal model generation unit 27 incorporates a general musculoskeletal model M, in which weight allocations and center of gravity positions of segments (muscles and bones and markers) for a certain height and weight are set, and when the height HT and weight BW of the subject to be analyzed are input, scaling conversion is performed based on them to generate a subject musculoskeletal model that matches the skeleton of the subject to be analyzed.

[0069] The height HT and weight BW are data measured during exercise of the subject. The height HT and weight BW may be stored in advance in an exercise data storage unit (not shown) when the measurements are taken, or may be input to the subject musculoskeletal model generation unit 27 by an input operation from either the user terminal 11 or the client terminal 15.

[0070] For example, when the external force estimation unit 28 receives first to third data, which are time-series data, from the derivation unit 26 and the three-dimensional position and angle of the floor surface from the three-dimensional information estimation unit 23, the external force estimation unit 28 estimates the external force from the floor surface that is in contact with the subject of analysis by dynamic analysis based on the first to third data, the three-dimensional position and angle of the floor surface, and the subject's musculoskeletal model in response to these inputs. That is, the external force estimation unit 28 inputs motion information into the subject's musculoskeletal model and determines whether each contact point is in contact with the floor surface based on the velocity and height from the floor of multiple contact points set and arranged on the soles of the feet. Specifically, the contact point is determined to be in contact if its velocity is equal to or less than the second threshold TH2 and its height is equal to or less than the first threshold, and is determined not to be in contact otherwise. A floor reaction force is generated at a contact point determined to be in contact with the floor surface. Therefore, the external force estimation unit 28 performs dynamic analysis based on the motion information and estimates the external force at the contact point by optimization calculation so that the equation of motion for the whole body is established. In this example, the generalized position q generated as above ｍ , generalized speed q ｍ ', generalized acceleration q ｍ '',The external force is estimated as time series data based on the three-dimensional position and angle of the floor surface and the subject's musculoskeletal model.

[0071] The objective function can be expressed by the following equation (18), and an inverse dynamics calculation is performed.

[0072]

[0073] In formula (18), F ｉ N on each foot ｆ The ground reaction force (vector) at each contact point i. ｍ ) is the mass property vector, and C(q ｍ , q ｍ ') is the centrifugal force vector and the Coriolis force vector, G(q ｍ ) is the gravity vector. E is the external force vector, and F ｉ This constraint ensures that no dynamic error occurs between the pelvis, which is the root joint, and the global space. F ｉ is the maximum value F ｉｍａｘ The maximum value F ｉ ｍａｘ can be set to any value; in this example, body weight BW is multiplied by gravitational acceleration, and this multiplication is further multiplied by 0.4. The value by which the product of body weight BW and gravitational acceleration is further multiplied is not limited to 0.4 as in this example, but is preferably within the range of 0.28 to 0.52. By setting it within this range, error variation is kept extremely small. It is particularly preferable that the value by which the product of body weight BW and gravitational acceleration is further multiplied by 0.4 as in this example. Sensitivity analysis confirmed that even when the value is changed by 30% up or down, the change in error is reliably kept to less than 1%. Similarly, for object surfaces other than floors, the three-dimensional position and angle of a predefined object surface that comes into contact with the body, such as a seat or wall, can be input, and the contact state can be determined using thresholds based on the distance and relative velocity between a contact point provided on a body part that can come into contact with the object surface and the target object surface. The external force input from the target object surface to the contact point can be calculated using Equation (18) in the same way as the floor reaction force. In other words, the height of the contact point set on the sole of the foot from the floor is an example of the distance, and the velocity of the contact point set on the sole of the foot is an example of the relative velocity with respect to the object surface. When the object surface is the aforementioned seat surface, the number of contact points set may be multiple or just one. Furthermore, when the subject of analysis is holding a cane, the cane can be considered to be an integral part of the subject's body, and the contact point can be set on the bottom end of the cane that comes into contact with the floor. In this way, the subject's body can be conceived as including other objects (in this example, the cane) that displace along with the body during exercise.

[0074] When the external force is input from the external force estimator 28 and the first to third data are input from the derivator 26, the dynamics analyzer 31 performs a dynamics analysis in response to these inputs based on the external force, the first to third data, and the subject's musculoskeletal model. The dynamics analyzer 31 thereby calculates each joint torque of a plurality of joints connecting each of a plurality of segments.

[0075] In the dynamics analysis unit 31 of this example, the generalized force vector τ is calculated by inverse dynamics calculation using the following equation (19): The generalized force vector τ includes translational and rotational forces expressed in a generalized coordinate system, and the rotational force is the joint torque.

[0076]

[0077] When the first data and second data are input from the derivation unit 26 and the joint torque is input from the dynamics analysis unit 31, the muscle information estimation unit 32 responds to these inputs and estimates at least one of muscle activity and muscle tension based on the first data and second data, the subject's musculoskeletal model, and each joint torque. Both muscle activity and muscle tension are time-series data.

[0078] Joint torque is generated by the muscle tension of multiple corresponding muscle models, and the muscle information estimation unit 32 distributes the joint torque to the muscle tension of each muscle model through optimization calculations. The optimization calculations are performed to minimize the sum of squares or cubes of whole-body muscle activity, subject to the constraints of the Hill-type muscle model. This results in more optimal muscle activity and muscle tension being output. The Hill-type muscle model determines muscle tension based on the increase or decrease in muscle activity, i.e., the magnitude of muscle activity that increases or decreases over time, while depending on the muscle length and contraction velocity (the rate of change in muscle length) during exercise. The muscle length and contraction velocity are determined from the first data and the second data.

[0079] Here, in equation (20), "j" is the joint degree of freedom. In this example, the generalized force vector τ at the joint degree of freedom j is ｊ and muscle tension F ｍ The relationship is expressed by the following equations (20) and (21): ｍ is the muscle activity of muscle m, and F ｍ ０ is the maximum muscle tension of muscle m, and l ｍ is the muscle length, and v ｍ is the contraction velocity of muscle m, and r ｍ，ｊ is the moment arm of muscle m at joint degree of freedom j.

[0080]

[0081]

[0082]

[0083]

[0084]

[0085] The muscle information estimation unit 32 solves an optimization problem using objective functions such as the above equations (22) to (24) to estimate the generalized force vector τ ｊ is distributed to muscle tension. J represents the objective function, and minJ represents minimization of J. p is an arbitrary exponent that is set in advance. It is preferable to set p within the range of 2 to 4. A value of 2 or greater ensures a more accurate estimation of muscle activity than when p is less than 2, and a value of 4 or less ensures a more accurate estimation of muscle activity than when p is greater than 4. At least one of the muscle activity and muscle tension estimated in this manner is transmitted to the client terminal 15 or the like, as described above, and displayed on a display that is the display unit of the client terminal 15 in this example. The display may be provided as part of the motion analysis device 13.

[0086] As described above, according to this example, joint torque and muscle tension can be estimated despite using only one imaging device 17. Furthermore, according to the above configuration, three-dimensional positions such as joint positions can be generated (estimated) with high accuracy despite using only one imaging device 17. Furthermore, because the external force from the floor is estimated after estimating the three-dimensional position and angle of the floor surface, joint torque and muscle tension can be estimated with high accuracy. Furthermore, according to the above configuration, the three-dimensional positional relationship between the floor surface and the leg is also estimated, so joint torque, muscle tension, muscle activity, etc. can be determined with extremely high accuracy.

[0087] 13 Motion analysis device 17 Imaging device 22 Two-dimensional information estimation unit 23 Three-dimensional information estimation unit 26 Derivation unit 27 Subject musculoskeletal model generation unit 28 External force estimation unit 31 Dynamics analysis unit 32 Muscle information estimation unit

Claims

1. An information processing device comprising: a two-dimensional information estimation unit that estimates two-dimensional time series data of two-dimensional joint positions of a subject to be analyzed from a plurality of frame images that constitute a moving image having the subject as the subject; a three-dimensional information estimation unit that estimates motion information based on the estimated two-dimensional time series data, the motion information including first data of a position dimension and second data of a velocity dimension that smoothly displaces along a time series and is expressed in a generalized coordinate system, and estimates a three-dimensional position and angle of a floor surface based on the motion information; a derivation unit that derives third data of an acceleration dimension expressed in a generalized coordinate system by first differentiating the second data; a subject musculoskeletal model generation unit that generates a subject musculoskeletal model by applying the height and weight of the subject to be analyzed to a general musculoskeletal model; and an external force estimation unit that estimates an external force from the object surface that comes into contact with the subject to be analyzed by dynamics analysis based on the first data, the second data, the three-dimensional position and angle of the object surface including the floor surface, the third data, and the subject musculoskeletal model.

2. The information processing device according to claim 1, further comprising: a dynamics analysis unit that performs dynamics analysis based on the external force and the first data to the third data, and calculates each joint torque of a plurality of joints connecting each of a plurality of segments; and a muscle information estimation unit that estimates at least one of muscle activity and muscle tension based on the first data and the second data, the subject's musculoskeletal model, and each joint torque.

3. An information processing device as described in claim 1 or 2, wherein the external force estimation unit determines whether each contact point is in contact with the object surface based on the relative speed and distance between the contact point, which is set on a part of the body of the subject of analysis that is in contact with the object surface, and estimates the external force at the contact point if it is determined that the contact point is in contact.

4. The information processing device according to claim 1 or 2, wherein the object surface is a floor surface.

5. The information processing device of claim 4, wherein the external force estimation unit determines whether each of a plurality of contact points set on the sole of the foot is in contact with the floor surface based on the speed of the contact points and their height from the floor surface, and estimates the external force at the contact points when it is determined that the contact points are in contact.

6. An information processing method comprising: a two-dimensional information estimation step of estimating two-dimensional time series data of two-dimensional joint positions of a subject from a plurality of frame images constituting a moving image having the subject as a subject; a three-dimensional information estimation step of estimating motion information including first data of a position dimension and second data of a velocity dimension that smoothly displaces along a time series and is expressed in a generalized coordinate system based on the estimated two-dimensional time series data, and estimating a three-dimensional position and angle of a floor surface based on the motion information; a derivation step of deriving third data of an acceleration dimension expressed in a generalized coordinate system by first differentiating the second data; a subject musculoskeletal model generation step of generating a subject musculoskeletal model by applying the height and weight of the subject to be analyzed to a general musculoskeletal model; and an external force estimation step of estimating an external force from the object surface in contact with the subject to be analyzed by dynamics analysis based on the first data, the second data, the three-dimensional position and angle of the object surface including the floor surface, the third data, and the subject musculoskeletal model.

7. A motion evaluation program that causes a computer to execute the following steps: a two-dimensional information estimation step of estimating two-dimensional time series data of two-dimensional joint positions of a subject from a plurality of frame images that constitute a moving image having the subject as a subject; a three-dimensional information estimation step of estimating motion information including first data of a position dimension and second data of a velocity dimension that smoothly displaces along a time series and is expressed in a generalized coordinate system based on the estimated two-dimensional time series data, and estimating a three-dimensional position and angle of a floor surface based on the motion information; a derivation step of deriving third data of an acceleration dimension expressed in a generalized coordinate system by first differentiating the second data; a subject musculoskeletal model generation step of generating a subject musculoskeletal model by applying the height and weight of the subject to be analyzed to a general musculoskeletal model; and an external force estimation step of estimating an external force from the object surface that comes into contact with the subject to be analyzed by dynamics analysis based on the first data, the second data, the three-dimensional position and angle of the object surface including the floor surface, the third data, and the subject musculoskeletal model.

Images