Motion analysis device, motion analysis method, and motion analysis program

Abstract

Provided are a motion analysis device, a motion analysis method, and a motion analysis program that facilitate system configuration and can achieve a decrease in installation space. A preprocessing unit (23) of a motion analysis device (13) reduces the frame rate of moving image data obtained by imaging a subject from a fixed point, and a posture estimation unit (29) estimates a three-dimensional posture. A conversion unit (32) converts first joint position coordinates, a first joint rotation vector, and first skin vertex coordinates by using a predetermined conversion matrix. A position correction unit (33) corrects second joint position coordinates and second skin vertex coordinates so that the average value of the heights of the lowest vertexes among the second skin vertex coordinates corresponding to the ground contact timing is 0. A height reference correction unit (36) corrects third joint position coordinates and third skin vertex coordinates on the basis of correction information for correcting a calculated height, which is calculated from the third skin vertex coordinates, to the height of the subject, and a walking variable calculation unit (41) calculates a walking variable for the subject.

Description

Motion analysis device, motion analysis method, and motion analysis program

[0001] The present invention relates to a motion analysis device, a motion analysis method, and a motion analysis program.

[0002] There is a technology for estimating and analyzing human body movements using motion capture. For example, Patent Document 1 (JP-A-2005-102666) describes an information processing device including an acquisition unit and a conversion unit. The acquisition unit acquires position dimension data of an object expressed in a generalized coordinate system. The conversion unit acquires at least acceleration dimension data from multiple inertial sensors attached to the object and converts the acquired acceleration data into acceleration dimension data expressed in the generalized coordinate system based on the position dimension data of the object expressed in the generalized coordinate system. The object includes multiple segments and joints connecting two or more of the segments, and the generalized coordinate system includes variables for at least a rotation angle around one or more axes for each joint. This information processing device further includes an estimation unit. The estimation unit further includes an estimation unit that estimates at least one of an external force acting on the object and a torque generated at a joint based on the position dimension data expressed in the generalized coordinate system, velocity dimension data of the object obtained by differentiating the position dimension data, and acceleration dimension data converted by the conversion unit. The estimation unit estimates at least one of an external force acting on the object and a torque generated in a joint by performing one or both of a forward dynamics calculation and an inverse dynamics calculation.

[0003] In addition to the above-mentioned method using an inertial sensor (IMU, Inertial Measurement Unit), there are also methods using optical motion capture and multiple video cameras for detecting movement.

[0004] Japanese Patent Application Laid-Open No. 2020-201138

[0005] However, when an inertial sensor is used as in the information processing device described in Patent Document 1, the system includes many components, including the inertial sensor, and a large space is required to install the device during use. The same problem applies to methods using optical motion capture and multiple video cameras.

[0006] Therefore, an object of the present invention is to provide a motion analysis device, a motion analysis method, and a motion analysis program that have a simple system configuration and can save installation space.

[0007] The motion analysis device of the present invention includes an acquisition unit, a frame rate reduction unit, a posture estimation unit, a transformation matrix generation unit, a conversion unit, a position correction unit, a physical information storage unit, a height reference correction unit, and a gait variable calculation unit. The acquisition unit acquires video data captured from a fixed point of a subject whose body movement is to be analyzed. The frame rate reduction unit reduces the frame rate of the acquired video data. The posture estimation unit estimates a three-dimensional posture including first joint position coordinates, first joint rotation vectors, first skin vertex coordinates, floor normal vectors, and ground contact timings, each expressed in three dimensions, based on a plurality of chronologically ordered frame images obtained by reducing the frame rate, to generate three-dimensional posture time-series data. The transformation matrix generation unit generates a transformation matrix that causes the floor normal vector to point vertically upward. The conversion unit converts the first joint position coordinates, first joint rotation vectors, and first skin vertex coordinates into second joint position coordinates, second joint rotation vectors, and second skin vertex coordinates using the transformation matrix. The position correction unit corrects each of the second joint position coordinates and second skin vertex coordinates constituting the time series data to third joint position coordinates and third skin vertex coordinates so that the average value of the height of the lowest vertex among the second skin vertex coordinates of the frame images corresponding to the ground contact timings becomes 0. The physical information storage unit stores the height of the subject. The height reference correction unit corrects each of the third joint position coordinates and third skin vertex coordinates constituting the time series data to fourth joint position coordinates and fourth skin vertex coordinates based on correction information that corrects the calculated height calculated from the third skin vertex coordinates to the height of the subject. The gait variable calculation unit calculates gait variables for the subject based on the fourth joint position coordinates and fourth skin vertex coordinates.

[0008] The walking variables are preferably at least one of walking speed, step length, stride length, number of steps per unit time, maximum value of each joint angle, distance traveled by the center of gravity, and joint angle at a predetermined joint in a walking cycle.

[0009] It is preferable that the device further includes a contact timing calculation unit that calculates, from the time series data of the fourth joint position coordinate and the fourth skin vertex coordinate, the timing at which the height from the floor becomes equal to or less than a predetermined height threshold and the speed becomes equal to or less than a predetermined speed threshold as the heel contact timing, and the gait variable calculation unit calculates the gait variables from the contact timing.

[0010] Preferably, the conversion unit further converts the second joint rotation vector into a first joint Euler angle expressed by Euler angles. In this case, it is preferable to further include a time series data conversion unit that resamples time series data of the second joint rotation vector, the first joint Euler angle, the fourth joint position coordinate, and the fourth skin vertex coordinate for each of a plurality of chronologically ordered walking cycles obtained based on the ground contact timing, to generate time series data consisting of a predetermined constant number of third joint rotation vectors, the second joint Euler angles, the fifth joint position coordinates, and the fifth skin vertex coordinates.

[0011] It is preferable that the motion analysis device further includes a reference memory unit that stores reference gait variables that serve as evaluation criteria for gait variables, and a display control unit that compares the gait variables output from the gait variable calculation unit with the reference gait variables and generates an image showing the degree of difference from the reference gait variables.

[0012] The motion analysis method of the present invention includes an acquisition step, a frame rate reduction step, a posture estimation step, a transformation matrix generation step, a conversion step, a position correction step, a body information storage step, a height reference correction step, and a gait variable calculation step. The acquisition step acquires video data captured from a fixed point of a subject whose body movement is to be analyzed. The frame rate reduction step reduces the frame rate of the acquired video data. The posture estimation step estimates a three-dimensional posture including first joint position coordinates, first joint rotation vectors, first skin vertex coordinates, floor normal vectors, and ground contact timings, each expressed in three dimensions, based on a plurality of frame images in chronological order obtained by reducing the frame rate, to generate three-dimensional posture time-series data. The transformation matrix generation step generates a transformation matrix that directs the floor normal vector vertically upward. The conversion step transforms the first joint position coordinates, first joint rotation vectors, and first skin vertex coordinates into second joint position coordinates, second joint rotation vectors, and second skin vertex coordinates using the transformation matrix. The position correction step corrects the second joint position coordinates and the second skin vertex coordinates constituting the time series data to third joint position coordinates and third skin vertex coordinates so that the average value of the height of the lowest vertex among the second skin vertex coordinates of the frame images corresponding to the ground contact timings is 0. The physical information storage step stores the height of the subject. The height reference correction step corrects the third joint position coordinates and the third skin vertex coordinates constituting the time series data to fourth joint position coordinates and fourth skin vertex coordinates based on correction information that corrects the calculated height calculated from the third skin vertex coordinates to the height of the subject. The gait variable calculation step calculates gait variables for the subject based on the fourth joint position coordinates and the fourth skin vertex coordinates.

[0013] The motion analysis program of the present invention causes a computer to execute the above steps.

[0014] According to the present invention, the device configuration is simple and the installation space can be saved.

[0015] It is an explanatory diagram of a motion analysis system having a motion analysis device which is an embodiment. It is an explanatory diagram of an imaging method performed by an imaging device. It is a configuration diagram of a motion analysis device. It is an explanatory diagram of an image displayed on a client terminal. It is an explanatory diagram of an image displayed on a client terminal.

[0016] The motion analysis system 10 shown in FIG. 1 captures video of a subject (hereinafter simply referred to as the "subject") whose body movements are to be analyzed, and analyzes the subject's body movements based on the captured images. The motion analysis system 10 includes a user terminal 11 used by the subject, a motion analysis device 13, which is an example of an embodiment of the present invention, and a client terminal 15 that acquires analysis results and displays images showing the analysis results, etc. The user terminal 11, the client terminal 15, and the motion analysis device 13 communicate with each other via a communication network CN, i.e., send and receive various data. For example, the motion analysis system 10 captures video of the subject using an imaging device (camera) 17 (see FIG. 2) installed in the user terminal 11, and the motion analysis device 13 analyzes the subject's movements based on the captured video data (hereinafter referred to as video data) and transmits the analysis results, etc. to the client terminal 15. As a result, the analysis results, etc. are displayed on the display (display unit) of the client terminal 15, allowing the client to analyze the subject's movements.

[0017] The user terminal 11 includes an imaging device 17 that captures video of the subject as a subject, and a transmission unit (not shown) that sends the captured video data to the motion analysis device 13. In this example, a smartphone is used as the user terminal 11, but it is not limited to a smartphone as long as it is equipped with the imaging device 17 and transmission unit. One imaging device 17 is sufficient for one user. Although only one user terminal 11 is depicted in FIG. 1, there may be multiple user terminals 11 so that multiple users can use each of them.

[0018] The client terminal 15 is, for example, a terminal used by a trainer who instructs a subject on physical training, but the user of the client terminal 15 is not limited to this example. The motion analysis system 10 may not include a client terminal 15. For example, the user terminal 11 may include a display for displaying images and a receiving unit for acquiring (receiving) image data of the images, and the subject may use the user terminal 11 as the client terminal 15 when viewing analysis results related to his or her own motion. Although only one client terminal 15 is illustrated in FIG. 1 , there may be multiple client terminals 15, each used by multiple clients, such as trainers. The client terminal 15 may also be used as a management terminal for managing various settings and / or data of the motion analysis device 13. The management terminal may be provided separately from the client terminal 15.

[0019] The motion analysis device 13 is for analyzing the motion of the subject based on the acquired video data. Details of the motion analysis device 13 will be described later with reference to another drawing.

[0020] The user terminal 11, the client terminal 15, and the motion analysis device 13 are each composed of a computer. The user terminal 11 and the client terminal 15 may operate on a browser by receiving a program that runs on a browser from the motion analysis device 13, or may be equipped with predetermined application software and operate by executing the program. The motion analysis device 13 is equipped with a predetermined program, and by executing this program, it functions as each unit described below and performs predetermined processing. The motion analysis device 13 performs predetermined processing in response to input of each piece of information from the user terminal 11 and the client terminal 15, respectively.

[0021] The program incorporated into the motion analysis device 13 causes the computer to execute an acquisition step, a frame rate reduction step, a posture estimation step, a transformation matrix generation step, a transformation step, a position correction step, a body information storage step, a height reference correction step, and a gait variable calculation step. The acquisition step acquires video data captured from a fixed point of a subject whose body movement is to be analyzed. The frame rate reduction step reduces the frame rate of the acquired video data. The posture estimation step generates three-dimensional posture time-series data by estimating three-dimensional postures including first joint position coordinates, first joint rotation vectors, first skin vertex coordinates, and floor normal vectors, as well as ground contact timings, based on multiple frame images in chronological order obtained by reducing the frame rate. The transformation matrix generation step generates a transformation matrix that directs the floor normal vector vertically upward. The transformation step converts the first joint position coordinates, first joint rotation vectors, and first skin vertex coordinates into second joint position coordinates, second joint rotation vectors, and second skin vertex coordinates using the transformation matrix. The position correction step corrects the second joint position coordinates and the second skin vertex coordinates constituting the time series data to third joint position coordinates and third skin vertex coordinates so that the average value of the height of the lowest vertex among the second skin vertex coordinates of the frame images corresponding to the ground contact timings is 0. The physical information storage step stores the height of the subject. The height reference correction step corrects the third joint position coordinates and the third skin vertex coordinates constituting the time series data to fourth joint position coordinates and fourth skin vertex coordinates based on correction information that corrects the calculated height calculated from the third skin vertex coordinates to the height of the subject. The gait variable calculation unit calculates gait variables for the subject based on the fourth joint position coordinates and the fourth skin vertex coordinates.

[0022] The imaging device 17 is installed with a fixed position and orientation. As a result, the video is obtained as a fixed-point video captured from a fixed point. The imaging device 17 is positioned so that the entire body of the moving subject is captured during the analysis time, i.e., from the start to the end of the analysis period. In this example, as shown in FIG. 2 , a linear walking line (walkway) WL is set on the floor as the area in which the subject P moves, and the subject P's walking from one end to the other along this walking line WL is treated as an example of the subject P's movement. The user terminal 11 is positioned so that the imaging optical axis L of the imaging device 17 intersects with the walking line WL when viewed from above. The height of the imaging device 17 from the floor is set within a range of approximately 0.6 m to 1.2 m, e.g., 1 m, so that the entire body of the subject P is captured. In this way, the entire body of the subject P, from their feet to their head, is captured during the analysis time as they walk along the walking line WL. In this example, the imaging device 17 is positioned at the center of the walking line WL so that the walking line WL and the imaging optical axis L are perpendicular to each other. However, if the entire body of the subject P is to be imaged, the imaging device 17 may be positioned so that the imaging optical axis L intersects with the walking line WL at an angle other than 90° at a position offset from the center of the walking line WL, as shown by the two-dot chain line in Figure 2.

[0023] 3, the motion analysis device 13 includes an acquisition unit 21, a calibration processing unit 22, a preprocessing unit 23, a silhouette image generation unit 26, a two-dimensional posture estimation unit 27, a three-dimensional posture estimation unit 28, a transformation matrix generation unit 31, a conversion unit 32, a position correction unit 33, a height reference correction unit 36, a subject data storage unit 37, and a gait variable calculation unit 41. The motion analysis device 13 preferably further includes a ground contact timing calculation unit 42, a gait cycle calculation unit 43, a gait cycle extraction unit 46, a resampling unit 47, a reference storage unit 51, a display control unit 53, etc., and this is also the case in this example.

[0024] The acquisition unit 21 acquires video data sent from the imaging device 17. The video data is made up of a plurality of frame images arranged in chronological order.

[0025] Since the imaging device 17 is fixed as described above and images are captured, distortion of the subject image changes depending on the position of the subject within the imaging range. The manner in which this distortion changes varies depending on the performance of the lens of the imaging device 17 and also on the posture (height, direction) of the imaging device 17. Therefore, before starting to capture images of the subject P (see FIG. 2 ), the imaging device 17 captures a video for correcting the distortion, and the video data is sent to the acquisition unit 21 to calibrate the internal parameters of the imaging device 17. An example of a video for performing calibration is a video in which a checkerboard, with alternating white and black rectangular areas formed as the subject, is captured while moving across the entire imaging range.

[0026] While the resolution and frame rate when capturing images using the imaging device 17 are not particularly limited, it is preferable to use the same imaging method, including the resolution and frame rate, for the calibration imaging and the imaging of the subject P, in order to prevent internal parameters from changing due to differences in the imaging method used by the imaging device 17, in order to use the same internal parameters. In this example, the calibration imaging and the imaging of the subject P are both set to the same resolution of 4K (2160p) and the same frame rate of 120 fps. When capturing the image of the subject P, the resolution may be set according to the distance between the subject P and the imaging device 17. For example, when the distance is greater, such as 10 m, it is preferable to set the resolution to at least 4K. When capturing the image of the subject P, the frame rate may be set higher to suppress blurring; in this example, the frame rate may be set to 240 fps instead of 120 fps. When the acquisition unit 21 acquires video data for calibration, it sends the acquired video data to the calibration processing unit 22. When the acquisition unit 21 acquires video data of the subject, it sends the acquired video data to the pre-processing unit 23.

[0027] If the next image capture is performed using the same image capture method, such as the image capture device 17 used, the attitude of the image capture device 17, and the settings of the image capture device 17 (for example, resolution and angle of view), calibration does not need to be performed the next time. This is because the internal parameters do not change. In this example, while the client terminal 15 is logged in and connected to the motion analysis device 13, a calibration execution instruction button that instructs the client terminal 15 to perform calibration is displayed on the touch panel display provided as a display of the client terminal 15. Touching this button is configured to obtain video data of an image of a checkerboard and perform calibration by the calibration processing unit 22.

[0028] The calibration processing unit 22 determines the internal parameters of the camera serving as the image capture device 17 based on the video data sent to it, and outputs the determined internal parameters to the three-dimensional posture estimation unit 28. When the pre-processing unit 23 performs processing to reduce the resolution as described below, the calibration processing unit 22 converts the resolution of the multiple frame images making up the sent video data to the same resolution as the frame images sent from the pre-processing unit 23, and then determines the internal parameters of the image capture device 17. This prevents the internal parameters from changing, and also maintains the accuracy of the estimation by the three-dimensional posture estimation unit 28.

[0029] The preprocessing unit 23 is an example of a frame rate reduction unit. When video data is sent from the acquisition unit 21, the preprocessing unit 23 performs an FPS reduction process in response to this input, reducing the frame rate (hereinafter referred to as FPS, where FPS stands for frames per second)—the number of frames (images) per second. Reducing the FPS improves the processing efficiency of subsequent processes. The FPS value and the number of frame images resulting from the reduction are not particularly limited. In this example, the FPS is reduced to 30 fps, and the FPS of the video pre-trained by the AI ​​(Artificial Intelligence) used in the three-dimensional posture estimation unit 28 is also 30 fps. In this way, the preprocessing unit 23 reduces the FPS of the video to the same as the FPS of the video pre-trained by the AI ​​used in the three-dimensional posture estimation unit 28, thereby maintaining the accuracy of estimation by the three-dimensional posture estimation unit 28.

[0030] Depending on the resolution of the acquired video, i.e., the resolution of each frame image constituting the acquired video data, the preprocessing unit 23 may further perform resolution reduction processing to reduce the resolution of each frame image, from the perspective of improving the processing efficiency of subsequent processes. In this example, the preprocessing unit 23 reduces the resolution to full HD (high definition, 1080p). Even at full HD resolution, it has been confirmed that accuracy in subsequent processes, such as the calculation of joint position coordinates (described below), is maintained unless the predetermined processing, such as estimation, performed by the silhouette image generating unit 26 and the two-dimensional posture estimating unit 27 is significantly inappropriate. For example, it has been confirmed that the error in joint position coordinates in global space when the pelvis is not fixed is 26.00 cm. Furthermore, it has been confirmed that the error when the pelvis is fixed is 12.02 cm. Note that it is preferable not to perform resolution reduction processing when the resolution is relatively insufficient, such as when the subject image of the subject P is excessively small in the image. In such a case, the silhouette image generating unit 26 or the two-dimensional posture estimating unit 27 may not properly perform the predetermined processing such as estimation, which may result in a lower accuracy or even make analysis impossible. In this way, it is advisable to determine whether or not to perform resolution reduction processing depending on the balance between the size of the subject image of the subject P in the image and the set resolution. Therefore, in this example, the resolution reduction processing is not performed when the distance between the subject P and the imaging device 17 exceeds approximately 10 m.

[0031] The pre-processing unit 23 may perform either the resolution reduction process or the FPS reduction process first. The pre-processing unit 23 outputs the time-series data of the frame images obtained through the resolution reduction process and the FPS reduction process to the silhouette image generation unit 26 and the two-dimensional posture estimation unit 27.

[0032] When time-series data of frame images is input, silhouette image generation unit 26 generates a silhouette image of the subject, who is the subject, for each frame image in response to this input and outputs the generated silhouette image to three-dimensional posture estimation unit 28. A silhouette image is an image in which the area inside the outline of the subject image representing the subject is treated as the foreground and this foreground is classified and extracted from the area outside the outline as the background. However, instead of an image in which the foreground is extracted, the silhouette image may be an image in which the foreground is shown in a different color from the background, or an image in which the foreground is shown in a different pattern (such as hatching) from the background.

[0033] When time-series data of frame images is input, the two-dimensional posture estimation unit 27 responds to this input by estimating the coordinates of the subject's two-dimensional joint positions (hereinafter referred to as two-dimensional joint positions) for each frame image as a two-dimensional posture (hereinafter referred to as two-dimensional posture), and outputs the time-series data of the estimated two-dimensional joint position coordinates to the three-dimensional posture estimation unit 28. The two-dimensional joint position coordinates are the two-dimensional position coordinates of multiple joints in the body, such as elbows, wrists, neck, knees, and hip joints. In other words, the two-dimensional posture estimation unit 27 estimates two-dimensional joint position coordinates for each joint. Therefore, two-dimensional joint position coordinates are estimated for each of a plurality of pre-set joints per frame.

[0034] The two-dimensional joint position coordinates are expressed in the image coordinate system, where the vertex located at the leftmost and topmost corner of the image area defined on the xy plane is set as the origin (0,0), and the direction to the right is treated as the positive direction of the x-axis (positive values ​​increase gradually), and the direction to the bottom is treated as the positive direction of the y-axis (positive values ​​increase gradually).

[0035] The method for estimating the two-dimensional joint positions is not particularly limited, and any known method can be used. For example, the two-dimensional joint positions can be determined by inputting time-series data of frame images constituting previously acquired video data into a convolutional neural network and estimating each joint point on an image coordinate system by regression, as in DeepPose (described in the following Non-Patent Document 1: A. Toshev and C. Szegedy, "DeepPose: Human Pose Estimation via Deep Neural Networks," 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, Ohio, USA, 2014, pp. 1653-1660.). Two-dimensional joint positions may be estimated using an improved method that improves estimation accuracy by representing the probability distribution of joint positions as a heat map, such as OpenPose (described in the following non-patent document 2: Z. Cao, G. Hidalgo, T Simon, S. Wei, and Y. Sheikh. 2021. “OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields,” IEEE Trans. Pattern Anal. Mach. Intell. 43, 1 (Jan. 2021), 172-186.), or by grouping candidate joint points.

[0036] Since the two-dimensional posture estimation unit 27 obtains the two-dimensional posture from a video captured by only one imaging device 17, the only equipment that needs to be installed under the subject is a single smartphone as the user terminal 11. Therefore, the device configuration is simple, and space-saving installation is achieved.

[0037] The three-dimensional posture estimation unit 28 incorporates a posture estimation model (not shown) and AI. When the three-dimensional posture estimation unit 28 receives time-series data of silhouette images from the silhouette image generation unit 26 and time-series data of two-dimensional joint positions from the two-dimensional posture estimation unit 27, the three-dimensional posture estimation unit 28 uses the posture estimation model to estimate a three-dimensional posture (hereinafter referred to as the three-dimensional posture) using AI in response to these inputs. In this example, the AI ​​uses a conditional variation autoencoder (CVAE) that can impose conditions on the generated data. This AI learns time-series data of three-dimensional postures calculated from optical motion capture data or the like using a skeletal model and inverse kinematics, for example, using a dataset consisting of paired information in which the three-dimensional posture at a certain time point t on the time axis is input information and the three-dimensional posture at time t+1 is output information, thereby estimating the three-dimensional posture at time t+1 from the three-dimensional posture at time t. At this time, the internal parameters obtained by the calibration processing unit 22 are used to project the three-dimensional joint positions onto a two-dimensional plane on the image coordinate system, and an optimal three-dimensional posture is estimated so as to minimize the error with respect to the two-dimensional joint positions obtained by the two-dimensional posture estimation unit 27. The silhouette image can be used as a basis for determining that the two-dimensional joint positions obtained by the two-dimensional posture estimation unit 27 are erroneously estimated and ignoring the estimated data if they are located outside the outline of the silhouette shown in the silhouette image. However, the method for acquiring the three-dimensional posture data when creating the data set does not matter.

[0038] The posture estimation model has multiple vertices set on the skin (body surface) of the body, and three adjacent vertices form multiple triangular polygons, which are expressed as body surface data. The number of vertices is not particularly limited, and is set to 6,890 in this example. The vertices are identified by IDs (identification information) ranging from 0001 to 6,890, and some of these vertices are associated with joint positions. By specifying (identifying) the IDs, unique skin vertex coordinates can be obtained for any posture. The three-dimensional posture includes at least the first joint position coordinates, first joint rotation vector, first skin vertex coordinates, floor position and normal vector, and the ground contact timing of the subject on the floor, all of which are expressed in three dimensions. The three-dimensional posture estimation unit 28 estimates these as the three-dimensional posture. The first skin vertex coordinates are the coordinates of the 6,890 vertices on the body surface, as described above. The floor normal vector is an upward unit vector that originates on the floor and is perpendicular to the floor.

[0039] The first joint position coordinates are three-dimensional position coordinates of multiple joints in the body, such as elbows, wrists, neck, knees, and hip joints. That is, the three-dimensional posture estimation unit 28 estimates three-dimensional joint position coordinates for each joint. Therefore, the first joint position coordinates are estimated for each of multiple joints set in advance per frame.

[0040] The first joint position coordinate and the first joint rotation vector are expressed in a generalized coordinate system. The generalized coordinate system is a coordinate system capable of expressing the three-dimensional posture of a modeled subject using variables corresponding to the degrees of freedom of the model. In this embodiment, the generalized coordinate system is similar to the generalized coordinate system described in the aforementioned Patent Document 1, in which, for example, the head, chest, abdomen, pelvis, left and right thighs, left and right shins, and left and right feet are defined as segments SG, and joints JT(i) connecting the segments are defined (where i = 1 to N). Of the segments SG, the segment SG(B) corresponding to the pelvis is defined as the base segment SG(B). The six-degree-of-freedom displacements of segments SG other than the base segment SG(B) relative to the origin of the absolute coordinate system are not variables of the generalized coordinate system, and only the base segment SG(B) is treated as displaceable with six degrees of freedom. The six degrees of freedom include three translational directions (XYZ) and three rotational directions (yaw, roll, and pitch). The three translation directions are time-series data of the first joint position coordinates of the ID associated with the pelvis, and the three rotation directions are the first joint rotation vectors of the ID associated with the pelvis.

[0041] Three-dimensional posture estimation unit 28 outputs the floor normal vector to transformation matrix generation unit 31, the first joint position coordinate, the first joint rotation vector, and the first skin vertex coordinate to transformation unit 32, and the ground contact timing to position correction unit 33. As described above, silhouette image generation unit 26, two-dimensional posture estimation unit 27, and three-dimensional posture estimation unit 28 constitute posture estimation unit 29 that estimates a three-dimensional posture including the three-dimensional first joint position coordinate, the three-dimensional first joint rotation vector, the three-dimensional first skin vertex coordinate, the three-dimensional position and inclination of the floor, and the ground contact timing of contact with the floor, based on a plurality of frame images in chronological order obtained by reducing the number of frame images.

[0042] When a floor normal vector is input, the transformation matrix generation unit 31 generates a transformation matrix in response to this input such that the floor normal vector points vertically upward. For example, for the floor normal vector N, first, the X-axis unit vector e in the absolute coordinate system is ｘ By calculating the cross product of N and e ｘ and a unit vector e perpendicular to ｙ ' is obtained. Furthermore, eｙ By calculating the cross product of ' and N, ｙ A unit vector e perpendicular to ' and N ｘ At this time, e ｘ ',e ｙ ', N are mutually orthogonal, and the matrix M[e ｘ ',e ｙ By constructing

[0047] , a transformation matrix is ​​generated that rotates the direction of the estimated normal vector N of the floor surface by an amount that matches the positive direction of the Z axis of the absolute coordinate system. The transformation matrix generation unit 31 outputs the generated transformation matrix to the conversion unit 32.

[0043] When the conversion unit 32 receives time-series data of the first joint position coordinates, the first joint rotation vector, and the first skin vertex coordinates from the three-dimensional posture estimation unit 28 and a transformation matrix from the transformation matrix generation unit 31, the conversion unit 32 converts only those of the first joint position coordinates and the first joint rotation vector associated with the pelvis and the first skin vertex coordinates using the transformation matrix to convert them into second joint position coordinates, the second joint rotation vector, and the second skin vertex coordinates. This conversion process converts the position coordinates of the entire body so that the head faces upward in three-dimensional space, and the Z axis of the absolute coordinate system coincides with the direction of gravity. In this way, time-series data of the second joint position coordinates, the second joint rotation vector, and the second skin vertex coordinates are generated. The conversion unit 32 outputs the generated time-series data of the second joint position coordinates and the second skin vertex coordinates to the position correction unit 33 and outputs the second joint rotation vector to the resampling unit 47 and the display control unit 53.

[0044] When an image indicating the joint angle is displayed on the client terminal 15 using the display control unit 53, it is preferable that the conversion unit 32 further converts the second joint rotation vector into a first joint Euler angle indicated by Euler angles. In this case, the conversion unit 32 outputs the obtained time-series data of the first joint Euler angle to the resampling unit 47 and the display control unit 53.

[0045] When the position correction unit 33 receives the ground contact timing from the three-dimensional posture estimation unit 28 and the time-series data of the second joint position coordinates and the second skin vertex coordinates from the conversion unit 32, the position correction unit 33 identifies a frame image corresponding to the ground contact timing, i.e., the second skin vertex coordinate of the frame image showing the ground contact timing, from the time-series data of the second skin vertex coordinates. Then, the position correction unit 33 identifies the lowest coordinate (bottom coordinate), i.e., the second skin vertex coordinate corresponding to the lowest vertex with the smallest Z coordinate, from the identified second skin vertex coordinates, and then calculates the average value of the plurality of bottom coordinates at the ground contact timing. The average value of the bottom coordinate is the average value of the Z coordinates of the plurality of bottom coordinates. That is, the average value of the bottom coordinate indicated by the average Z coordinate is calculated by summing the Z coordinates of the plurality of bottom coordinates and dividing this sum by the number of bottom coordinates. Furthermore, the position correction unit 33 obtains the XY coordinates of the pelvis position at the initial time in the time series from the second joint position coordinates, and sets these as the initial pelvis XY coordinates. By subtracting the average values ​​of the lowest coordinates and the initial XY coordinates of the pelvis from the second joint position coordinates and second skin vertex coordinates at all time points, the second joint position coordinates and second skin vertex coordinates constituting the time series data are corrected to third joint position coordinates and third skin vertex coordinates so that the distance between the average value of the lowest coordinates and the floor surface, where the Z coordinate is 0, becomes 0 and the initial XY coordinates of the pelvis become 0. This prevents the subject P from being slightly floating above the floor surface or being buried in the floor surface, resulting in data indicating walking with the feet in contact with the floor surface, and furthermore, time series data of the third joint position coordinates and third skin vertex coordinates is generated so that the subject P starts walking from the center of the absolute coordinate system. The position correction unit 33 outputs each of these time series data to the height reference correction unit 36.

[0046] The subject data storage unit 37 is an example of a physical information storage unit, and stores the subject's height, and in this example, also stores the subject's weight. The height and weight are stored in the subject data storage unit 37 by a central control unit (not shown) that centrally controls each unit of the motion analysis device 13 in response to an input operation on at least one of the client terminal 15 and the user terminal 11.

[0047] When time-series data of the third joint position coordinates and the third skin vertex coordinates are input from the position correction unit 33, the height reference correction unit 36 ​​calculates the height from the third skin vertex coordinates in response to these inputs, and reads the subject's height from the subject data storage unit 37. The height reference correction unit 36 ​​stores correction information for correcting the calculated height to the subject's height, and based on this correction information, the height reference correction unit 36 ​​corrects the third joint position coordinates and the third skin vertex coordinates constituting the time-series data to fourth joint position coordinates and fourth skin vertex coordinates, respectively. The three-dimensional posture estimation unit 28 also estimates the height and reflects the height in estimating the first skin vertex coordinates. Therefore, the calculated height is also reflected in the height, but the estimation result contains errors. This correction by the height reference correction unit 36 ​​brings the analysis results of the gait variables (described below) calculated by the gait variable calculation unit 41 closer to true values, improving the reliability of the obtained analysis results.

[0048] The correction method is not particularly limited. For example, when the calculated height is MH and the subject's height is SH, the ratio calculated using the SH / MH correction formula is used as correction information, and correction can be performed by multiplying each third joint position coordinate constituting the time-series data and each third skin vertex coordinate constituting the time-series data by the correction information. According to this method, if the calculated height is 180 cm and the subject's height is 175 cm, the ratio calculated using the correction formula (175 / 180) is used as correction information, and is multiplied by each of the multiple third joint position coordinates and each of the multiple third skin vertex coordinates to obtain the fourth joint position coordinate and the fourth skin vertex coordinate. As a result, walking variables such as walking speed, step length, and stride length calculated based on a calculated height of 180 cm are obtained as smaller values ​​corresponding to the subject's height of 175 cm. The time-series data of the fourth joint position coordinate and the time-series data of the fourth skin vertex coordinate obtained in this manner are output to the ground contact timing calculation unit 42, the link sampling unit 47, and the display control unit 53, respectively.

[0049] When time series data of the fourth joint position coordinate and time series data of the fourth skin vertex coordinate are input, the ground contact timing calculation unit 42 calculates ground contact timing in response to these inputs and outputs the calculated timing to the gait variable calculation unit 41 and the gait cycle calculation unit 43. As described above, the ground contact timing is estimated by the three-dimensional posture estimation unit 28, and the estimation result has sufficient accuracy to be used for processing in the position correction unit 33. Therefore, the ground contact timing calculation unit 42 may not be provided, and the time series data of the fourth joint position coordinate and the time series data of the fourth skin vertex coordinate may be output from the height reference correction unit 36 ​​to the gait variable calculation unit 41 and the gait cycle calculation unit 43. However, the estimation result estimated by the three-dimensional posture estimation unit 28 may contain a slight error. This is thought to be because the AI ​​estimates a ground contact probability corresponding to movements such as walking from a latent space held by the AI, and determines that ground contact has occurred when the ground contact probability is equal to or greater than a certain level. Therefore, the ground contact timing calculation unit 42 calculates the timing of ground contact based on the fourth joint position coordinate and the fourth skin vertex coordinate obtained by correcting the Z coordinate in the vertical direction by the position correction unit 33 and the height reference correction unit 36. By recalculating the ground contact timing in this manner, the ground contact timing at which the lowest point of the body surface is positioned on the floor can be obtained efficiently and with greater accuracy.

[0050] The method for recalculating the contact timing is, for example, as follows. First, specific skin vertices, such as the heel and / or toe, are selected and stored from the vertices indicated by the fourth skin vertex coordinates as contact points that contact the floor surface. By specifying these IDs, time-series data of the position coordinates of the contact points is acquired. Furthermore, the velocity of the contact points is calculated based on the time-series data of the coordinates of the contact points. Then, from this time-series data, contact points whose distances between the coordinates of the contact points and the floor surface are equal to or less than a predetermined threshold (hereinafter referred to as the height threshold) and whose velocity is equal to or less than a predetermined velocity threshold are identified as being in a contact state, and the identified temporal position (point in time) is set as the contact timing. While the height threshold and velocity threshold are not particularly limited, in this example, the height threshold is set to 0.05 m and the velocity threshold is set to 0.8 m / s. The highly accurate contact timing is confirmed by the highly accurate floor reaction force calculated based on the contact timing.

[0051] When time series data of the fourth joint position coordinate and time series data of the fourth skin vertex coordinate, and in this example, ground contact timing, are input, the gait variable calculation unit 41 calculates gait variables in response to these inputs and outputs them to the display control unit 53. The gait variables may be at least one of walking speed, step length, stride length, cadence (number of steps per unit time), maximum value of each joint angle, distance traveled by the center of gravity, and joint angle in one walking cycle. Of these, the gait variable calculation unit 41 calculates walking speed, step length, stride length, cadence (number of steps per unit time), maximum value of each joint angle, and distance traveled by the center of gravity. The distance traveled by the center of gravity may be at least one of the maximum distance traveled by the subject P in the left-right and up-down directions and the total distance traveled in the left-right and forward-backward directions (length of the trajectory) for each walking cycle.

[0052] The walking speed is calculated by dividing the amount of movement of the pelvis in the direction of travel within the analysis time by the analysis time. The step length is calculated as the distance between the contact position of one heel and the contact position of the other heel. The stride length is calculated as the distance from the contact position of each heel to the contact position at the next contact timing. The cadence is calculated by dividing the number of steps within the analysis time by the analysis time. The maximum value of the joint angle is calculated by comparing multiple joint angles that make up the time-series data and finding the largest value. The joint angle in one walking cycle is the second joint Euler angle, which will be described later, and is calculated by the resampling unit 47.

[0053] The position of the center of gravity is calculated from the position of the center of gravity and the mass fraction of each segment included in the skeletal model. The mass fraction of a segment can be obtained from a known mass fraction table (described in the following non-patent document 3: Winter, DA (2009). Biomechanics and motor control of human movement (4th ed.). (USA) Wiley & Sons, Inc.). The position of the center of gravity of the entire body is then calculated by multiplying the position of the center of gravity of each segment by the mass fraction of the segment to obtain a product, and then summing the product values ​​obtained for all segments.

[0054] The walking speed is calculated based on the time-series data of the fourth joint position coordinate of the ID associated with the pelvis by dividing the amount of movement of the fourth joint position coordinate along a flat floor surface (the norm of the amount of movement in planar walking) by the movement time, which is the time to be analyzed. For example, if the amount of movement PL (unit: km) of the fourth joint position coordinate along the floor surface is 0.005 and the time to be analyzed T (unit: h) is 0.001, the walking speed (unit: km / h) is calculated as 5.0 km / h using the PL / T calculation formula. The stride length is the distance La1 from the first contact position where the heel of one foot touches the floor to the second contact position where the heel of the other foot then touches the floor, the distance Lb1 from the second contact position to the third contact position where the heel of the one foot touches the floor, the distance La2 from the third contact position to the fourth contact position where the heel of the other foot touches the floor, and so on. For example, the stride lengths are calculated as La1 = 0.778 m, Lb1 = 0.772 m, La2 = 0.781 m, etc. The stride length is time-series data within the analysis time T of the distance L1 from the first contact point of one heel with the floor to the aforementioned third contact point where the heel of the same foot next contacts the floor, and similarly the distance L2 from the third contact point to the fifth contact point, etc. For example, the stride lengths are calculated as L1 = 1.550 m, L2 = 1.553 m, etc.

[0055] The gait cycle calculation unit 43 calculates all gait cycles within the analysis time T and outputs them to the gait cycle extraction unit 46. The gait cycle is defined as the time from when the heel of one foot touches the ground to when the heel of the same foot touches the ground again, with 100% being the gait cycle. For example, within the analysis time T, the timings of the right heel touching the ground are, in order, i.e., chronologically, ta1, ta2, ta3, ta4, and ta5, and the timings of the left heel touching the ground are, in order, tb1 (where tb1 is after ta1 and before ta2), tb2, tb3, and tb4. In this case, the time from ta1 to ta2, the time from ta2 to ta3, the time from ta3 to ta4, the time from ta4 to ta5, the time from tb1 to tb2, the time from tb2 to tb3, and the time from tb3 to tb4 are each defined as 100%.

[0056] The gait cycle extraction unit 46 identifies and extracts one gait cycle closest to the center of the aforementioned imaging range from all gait cycles within the analysis time T, and outputs the extracted gait cycle as the extracted gait cycle to the resampling unit 47 along with all gait cycles within the analysis time. This extraction process is intended to extract the one gait cycle within the analysis time T as the gait cycle corresponding to the group of frame images with the smallest distortion. In the following description, the extracted gait cycle will be referred to as the extracted gait cycle. The center of the imaging range is defined as the coordinate system where the X and Y coordinates of the image coordinate system are half the width and height of the image resolution. The gait cycle closest to the center of the imaging range is defined as the gait cycle in which the two-dimensional joint position corresponding to the pelvis at the 50% point of each gait cycle is closest to the center of the imaging range. For example, in the above example, if the two-dimensional joint position corresponding to the pelvis at the median time between tb2 and tb3 is closest to the center of the imaging range, the gait cycle at that time is defined as the extracted gait cycle. However, this process assumes movement within the imaging range. When subject P is imaged on the walking line WL (see Figure 2), the pelvis is always located near the center of the imaging range, and so even in this case, the gait cycle closest to the center of the imaging range is extracted using the method described above. However, in this example, as described above, the entire body of subject P is imaged while walking, subject P is located at an analyzable distance, and is imaged at a resolution that allows analysis even at that distance, so this type of situation is avoided.

[0057] The resampling unit 47 is an example of a time-series data converter that resamples the time-series data of the second joint rotation vector and the first joint Euler angles input from the converter 32 and the time-series data of the fourth joint position coordinate and the fourth skin vertex coordinate input from the height-reference correction unit 36 ​​for each of all gait cycles in chronological order input from the gait cycle extraction unit 46 to generate time-series data consisting of a predetermined number of pieces of data, and converts the data into time-series data of the third joint rotation vector, the second joint Euler angles, the fifth joint position coordinate, and the fifth skin vertex coordinate. The number of pieces of information constituting the time-series data obtained by resampling is not particularly limited, and in this example, it is set to 101. That is, for each gait cycle, the time-series data is composed of 101 third joint rotation vectors in chronological order, 101 second joint Euler angles in chronological order, 101 fifth joint position coordinates in chronological order, and 101 fifth skin vertex coordinates in chronological order. In this way, for each gait cycle, time series data consisting of 101 third joint rotation vectors, time series data consisting of 101 second joint Euler angles, time series data consisting of 101 fifth joint position coordinates, and time series data consisting of 101 fifth skin vertex coordinates are generated. Note that the time intervals for all 101 chronologically ordered data are equal. As described above, the resampling process ensures that the number of first joint Euler angle information pieces is the same for all gait cycles. Therefore, subsequent processing involves comparisons between the same number of data pieces, resulting in efficient processing and rapid analysis results. The same applies to the second joint rotation vectors, fourth joint position coordinates, and fourth skin vertex coordinates. As described above, the second joint Euler angle is a joint angle during one gait cycle, which is one of the gait variables. Therefore, the resampling unit 47 also functions as a gait variable calculation unit.

[0058] The second joint rotation vector, the first joint Euler angle, the fourth joint position coordinate, and the fourth skin vertex coordinate are subjected to resampling processing to be acquired as the third joint rotation vector, the second joint Euler angle, the fifth joint position coordinate, and the fifth skin vertex coordinate, respectively, and are further output to the display control unit 53, thereby being used to display an image on the client terminal 15.

[0059] The reference storage unit 51 stores reference gait variables that serve as evaluation standards when evaluating the gait variables of the subject P. The reference gait variables are the same variables as the above-mentioned gait variables for the subject P, among walking speed, step length, stride length, cadence (number of steps per unit time), maximum value of each joint angle, distance traveled by the center of gravity, and joint angles in one gait cycle, and may be, for example, standard values ​​that are publicly known in papers or the like, i.e., average values, or past values ​​for the subject P. It is more preferable that the joint angles in one gait cycle stored in the reference storage unit 51 are also resampled, or the number of pieces of information is the same as the certain number set in the resampling unit 47 as described above. In this example, the number of pieces of information for standard values ​​that are publicly known in papers or the like is the same as the certain number set in the resampling unit 47 as described above.

[0060] When the second joint rotation vector, the third joint rotation vector, the fourth joint position coordinates, the fifth joint position coordinates, the fourth skin vertex coordinates, the fifth skin vertex coordinates, the first joint Euler angles, the second joint Euler angles, and the gait variables are input, the display control unit 53 generates an image showing at least one of a three-dimensional animation and an analysis result such as a graph based on at least a part of these, and sends it to the client terminal 15 for display. For example, based on the second joint rotation vector, the fourth joint position coordinates, the fourth skin vertex coordinates, the first joint Euler angles, and the gait variables, the display control unit 53 generates an image of a three-dimensional animation and an analysis result such as a graph on the time axis before resampling and sends it to the client terminal 15. Based on the third joint rotation vector, the fifth joint position coordinates, the fifth skin vertex coordinates, the second joint Euler angles, and the gait variables, the display control unit 53 generates an image of a three-dimensional animation and an analysis result such as a graph on the gait cycle time axis after resampling and sends it to the client terminal 15.

[0061] Furthermore, when the display control unit 53 of this example receives the gait variables output from the gait variable calculation unit 41, it reads out the reference gait variables corresponding to the input gait variables from the reference storage unit 51 in response to this input, compares the gait variables from the gait variable calculation unit 41 with the reference gait variables, and generates an image showing the degree of difference from the reference gait variables. The degree of difference is not particularly limited as long as it indicates the degree of difference, and may be shown numerically or as a graph, etc.

[0062] When data of an image to be displayed is input from the display control unit 53, the client terminal 15 displays the image G on the display 15a as shown in Fig. 4. The image G has a moving image section Ga showing a moving image and a result image section Gb showing the analysis results such as graphs and / or numerical values. However, the moving image section Ga and the result image section Gb may be configured to be different images G, allowing for switching between the images.

[0063] The moving image section Ga displays a moving image including a three-dimensional human body model MA represented as a subject P (see FIG. 2 ), and the movement of the three-dimensional human body model MA in the moving image reflects the body movement of the subject P. In the example shown in FIG. 4 , a moving image is displayed in which the three-dimensional human body model MA is shown walking from right to left on the page of FIG. 4 as the subject P walking on a flat floor. The moving image shows at least the walking start to the walking end, and the progress of the moving image is indicated by a progress bar BP extending linearly from "START" indicating the start to "END" indicating the end. The image G includes a pause button B1 for pausing the moving image in progress, a stop button B2 for stopping the moving image, a progress button (not shown) for progressing the moving image, and the progress of the moving image is turned on and off in response to touch operations on these buttons.

[0064] In this example, the three-dimensional human body model MA is generated by estimating the three-dimensional posture. The orientation of the three-dimensional human body model MA can be changed by swiping the client's finger while touching it on the video image section Ga of the display unit 15a. Furthermore, the foot contact timing and foot contact position are also identified in three-dimensional space. These time-series data are obtained in association with time-series data on the joint positions and joint angles of the entire body. Therefore, regardless of the orientation of the three-dimensional human body model MA, the walking direction can be displayed in a video, and the soles of the feet are displayed in a state that is in alignment with the floor. Thus, the motion analysis system 10 (see FIG. 1 ) not only has a simple system configuration and space-saving features, but also reflects the subject P's body movements and displays them on the client terminal 15. The display is three-dimensional, variable, and in alignment with the floor. As a result, the client using the client terminal 15 can repeatedly visually confirm the subject P's body movements and deepen their understanding of the body movements. Furthermore, because analysis results for the extracted gait cycle are obtained, and resampling processing is performed on the extracted gait cycle, and the evaluation criteria in the reference memory unit 51 are configured with the same number of pieces of information as those set in the resampling unit 47, the analysis results can be displayed in the result image unit Gb, and the current analysis results for subject P and the evaluation criteria can be displayed in the same range within the same graph, and the processing time required to generate the image is very short. As a result, despite the simple system configuration, precise analysis results are quickly displayed on the client terminal 15, and the client can easily understand the current analysis results for subject P and compare the analysis results with the evaluation criteria.

[0065] In the example shown in FIG. 4 , the result image section Gb includes, for "Left-knee," which indicates the left knee joint, a graph D1 of the joint angle of the left knee joint in the extracted gait cycle, and a numerical display D2 indicating the minimum value of the joint angle, "maximum flexion," and the maximum value, "maximum extension." The graph D1 displays a curve L1 of the joint angle of the left knee joint for subject P, superimposed on a curve L2 of the joint angle of the left knee joint, which is a reference gait variable. Near the horizontal axis of the graph D1, a movement operation section B3, indicated by an arrow, for example, is provided for sliding a straight line L3, indicated by a dotted line, extending vertically in a direction along the horizontal axis. When the client touches the movement operation section B3 on the display 15a with a finger and moves it to a predetermined position on the horizontal axis, the straight line L3 is displaced to that position, and the joint angle of the left knee joint indicated by the curve L1 that intersects with the displaced position and the gait cycle corresponding to this joint angle are displayed. For example, in FIG. 4, the joint angle and the corresponding walking cycle are displayed as "17° / 28%."

[0066] The result image section Gb is provided with a pull-down switching operation section B4 for switching the joint to be displayed. When the client uses this switching operation section B4 to select a joint other than the left knee joint, the result image section Gb having a graph D1 and a numerical display D2 for the selected joint is displayed on the client terminal 15.

[0067] Image G further includes a switch button B5 for displaying the analysis results for the entire walking cycle, i.e., the entire duration of the image capture. When the client touches this switch button B5, the result image section Gb is switched to a result image section Gc showing the analysis results for the entire walking cycle, as shown in FIG. 5. The result image section Gc includes a graph D3 of the joint angle of the left knee joint over the entire walking cycle, and numerical displays D4 showing the walking speed, step length, stride length, cadence, and distance traveled by the center of gravity. Similar to the result image section Gb (see FIG. 4), the result image section Gc also includes a switch operation section B4 and a switch button B5. The switch button B5 is used to switch to the result image section Gb showing the analysis results for the extracted walking cycle.

[0068] Graph D3 shows a curve L1 of the joint angle of the left knee joint for the subject P over an entire gait cycle, and a curve L2 of the joint angle of the left knee joint, which is the reference gait variable, over an entire gait cycle, superimposed on the curve L1. Graph D3 is provided with a movement operation unit B3, similar to graph D1 (see FIG. 4). In this way, the analysis results for the entire gait cycle are also displayed on the client terminal 15.

[0069] REFERENCE SIGNS LIST 10 Motion analysis system 11 User terminal 13 Motion analysis device 15 Client terminal 17 Imaging device 21 Acquisition unit 22 Calibration processing unit 23 Preprocessing unit 26 Silhouette image generation unit 27 Two-dimensional posture estimation unit 28 Three-dimensional posture estimation unit 29 Posture estimation unit 31 Transformation matrix generation unit 32 Transformation unit 33 Position correction unit 36 ​​Height reference correction unit 37 Subject data storage unit 41 Gait variable calculation unit 42 Landing timing calculation unit 43 Gait cycle calculation unit 46 Gait cycle extraction unit 47 Resampling unit 51 Reference storage unit 53 Display control unit

Claims

a frame rate reducing unit which reduces a frame rate of the acquired video data; a posture estimating unit which generates three-dimensional posture time series data by estimating a three-dimensional posture including a first joint position coordinate, a first joint rotation vector, a first skin vertex coordinate, a floor normal vector, and a ground contact timing of a floor surface based on a plurality of chronologically ordered frame images obtained by reducing the frame rate; a transformation matrix generating unit which generates a transformation matrix such that the floor normal vector faces vertically upward; a conversion unit which converts each of the first joint position coordinate, the first joint rotation vector, and the first skin vertex coordinate into a second joint position coordinate, a second joint rotation vector, and a second skin vertex coordinate using the transformation matrix; and a position correcting unit which corrects each of the second joint position coordinate and the second skin vertex coordinate constituting the time series data to a third joint position coordinate and a third skin vertex coordinate so that an average value of a height of the lowest vertex of the second skin vertex coordinate of the frame images corresponding to the ground contact timing becomes 0; a height reference correction unit that corrects each of the third joint position coordinates and the third skin vertex coordinates that constitute time series data to fourth joint position coordinates and fourth skin vertex coordinates based on correction information that corrects a calculated height calculated from the third skin vertex coordinate to the height of the subject; and a gait variable calculation unit that calculates gait variables for the subject based on the fourth joint position coordinates and the fourth skin vertex coordinate.

2. The motion analysis device according to claim 1, wherein the walking variables are at least one of walking speed, stride length, stride length, number of steps per unit time, maximum value of each joint angle, distance traveled by the center of gravity, and joint angle at a specified joint during a walking cycle.

3. The motion analysis device of claim 1 or 2, further comprising a contact timing calculation unit that calculates the timing at which the height from the floor surface becomes equal to or less than a predetermined height threshold and the speed becomes equal to or less than a predetermined speed threshold from each time series data of the fourth joint position coordinate and the fourth skin vertex coordinate, as the heel contact timing, wherein the gait variable calculation unit calculates the gait variables from the contact timing.

4. The motion analysis device according to claim 1 or 2, further comprising a time series data conversion unit which converts the second joint rotation vector into a first joint Euler angle represented by Euler angles, and resamples time series data of the second joint rotation vector, the first joint Euler angle, the fourth joint position coordinate, and the fourth skin vertex coordinate for each of a plurality of walking periods in chronological order obtained based on the ground contact timing, into a third joint rotation vector, the second joint Euler angle, a fifth joint position coordinate, and a fifth skin vertex coordinate, which are time series data composed of a predetermined fixed number.

5. The motion analysis device according to claim 1 or 2, further comprising: a reference memory unit that stores reference gait variables that serve as evaluation criteria for the gait variables; and a display control unit that compares the gait variables output from the gait variable calculation unit with the reference gait variables and generates an image showing the degree of difference from the reference gait variables.

6. An acquisition step of acquiring video data captured from a fixed point of a subject whose body movement is to be analyzed; a frame rate reduction step of reducing a frame rate of the acquired video data; a posture estimation step of estimating a three-dimensional posture including a first joint position coordinate, a first joint rotation vector, a first skin vertex coordinate, a floor normal vector, and a ground contact timing of the subject based on a plurality of frame images in chronological order obtained by reducing the frame rate, thereby generating three-dimensional posture time series data; a transformation matrix generation step of generating a transformation matrix such that the floor normal vector faces vertically upward; a transformation step of converting each of the first joint position coordinate, the first joint rotation vector, and the first skin vertex coordinate into a second joint position coordinate, a second joint rotation vector, and a second skin vertex coordinate using the transformation matrix; a position correction step of correcting each of the second joint position coordinate and the second skin vertex coordinate constituting the time series data to a third joint position coordinate and a third skin vertex coordinate so that an average value of the height of the lowest vertex of the second skin vertex coordinates of the frame images corresponding to the ground contact timing becomes 0; a height reference correction step of correcting each of the third joint position coordinates and the third skin vertex coordinates constituting time series data to fourth joint position coordinates and fourth skin vertex coordinates based on correction information for correcting a calculated height calculated from the third skin vertex coordinate to the height of the subject; and a gait variable calculation step of calculating gait variables for the subject based on the fourth joint position coordinates and the fourth skin vertex coordinate.

7. An acquisition step of acquiring video data captured from a fixed point of a subject whose body movement is to be analyzed; a frame rate reduction step of reducing a frame rate of the acquired video data; a posture estimation step of estimating a three-dimensional posture including a first joint position coordinate, a first joint rotation vector, a first skin vertex coordinate, a floor normal vector, and a ground contact timing of the subject based on a plurality of frame images in chronological order obtained by reducing the frame rate, thereby generating three-dimensional posture time series data; a transformation matrix generation step of generating a transformation matrix such that the floor normal vector faces vertically upward; a transformation step of converting each of the first joint position coordinate, the first joint rotation vector, and the first skin vertex coordinate into a second joint position coordinate, a second joint rotation vector, and a second skin vertex coordinate using the transformation matrix; a position correction unit that corrects each of the second joint position coordinate and the second skin vertex coordinate constituting the time series data to a third joint position coordinate and a third skin vertex coordinate so that an average value of the height of the lowest vertex of the second skin vertex coordinates of the frame images corresponding to the ground contact timing becomes 0; a body information storing step of storing a height of the subject; a height reference correcting step of correcting each of the third joint position coordinates and the third skin vertex coordinates constituting time series data to fourth joint position coordinates and fourth skin vertex coordinates based on correction information for correcting a calculated height calculated from the third skin vertex coordinate to the height of the subject; and a gait variable calculating step of calculating gait variables for the subject based on the fourth joint position coordinates and the fourth skin vertex coordinate.

Images