Exercise evaluation device, exercise evaluation method, and exercise evaluation program

Abstract

The motion evaluation device (11) includes a reference storage unit (23), a posture estimation unit (15), a body internal / external information calculation unit (19), an evaluation unit (25), and a display control unit (27). The reference storage unit (23) stores reference information that serves as the basis for evaluation. The posture estimation unit (15) estimates the position of each of a plurality of joints of the person being evaluated based on video data of the person being evaluated for body movement. The body internal / external information calculation unit (19) uses the height and weight of the person being evaluated and model data to determine at least one of muscle activity, joint load, center of pressure, and reaction force for the person being evaluated as body internal / external information. The evaluation unit (25) evaluates the body movement of the person being evaluated based on the reference information, the positions of the plurality of joints, and the body internal / external information. The display control unit (27) generates an image showing the evaluation result.

Description

[Technical field]

[0001] The present invention relates to an exercise evaluation device, an exercise evaluation method, and an exercise evaluation program. [Background technology]

[0002] For example, in order to understand the physical condition of a physically disabled person, an evaluation (diagnosis, etc.) is performed by visual inspection or palpation, and the results of this evaluation are recorded in a medical record. However, such evaluations are based on the subjectivity of the person performing the evaluation, and the evaluation results recorded for the person being evaluated are less objective. Therefore, there is a demand for evaluations that have objectivity.

[0003] Therefore, Patent Document 1 proposes an exercise evaluation system for evaluating physical exercise, and this exercise evaluation system includes a reference value storage unit, a body part identification unit, and an evaluation unit. The reference value storage unit stores a reference value related to the position of at least one body part. The body part identification unit analyzes an image of the body to identify the body part. The evaluation unit compares the position of the body part in the image with the reference value to determine an evaluation value for the exercise. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-43264 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the exercise evaluation system described in Patent Document 1 has limitations in grasping the physical state.

[0006] Therefore, an object of the present invention is to provide an exercise evaluation device, an exercise evaluation method, and an exercise evaluation program that enable a more accurate understanding of the physical condition of the person being evaluated. [Means for solving the problem]

[0007] The motion evaluation device of the present invention includes a reference memory unit, a posture estimation unit, a body internal and external information calculation unit, an evaluation unit, and a display control unit. The reference memory unit stores reference information that serves as the basis for evaluation. The posture estimation unit estimates the position of each of a plurality of joints of the person being evaluated based on video data of the person being evaluated for body movement. The body internal and external information calculation unit uses the height and weight of the person being evaluated and model data to determine at least one of muscle activity, joint load, center of pressure, and reaction force for the person being evaluated as body internal and external information. The evaluation unit evaluates the body movement of the person being evaluated based on the reference information, the plurality of joint positions, and the body internal and external information. The display control unit generates an image showing the evaluation result obtained by the evaluation unit. The body internal and external information calculation unit performs a scaling conversion of the model data, in which weight allocations of segments and center of gravity positions for a certain height and weight are set, based on the height and weight of the person to be evaluated, to obtain model data for the person to be evaluated that is tailored to the person to be evaluated, and obtains body internal and external information using the obtained model data for the person to be evaluated. The evaluation unit evaluates the body movement of the person to be evaluated by comparing the body internal and external information and the positions of multiple joints from the first key frame image to the second key frame image selected from the time series data constituting the video image data with reference information.

[0008] It is preferable that the evaluation unit evaluates the body movement of the person to be evaluated by resampling the body internal and external information and the multiple joint positions from the first key frame image to the second key frame image and comparing them with the reference information. The evaluation unit compares the resampled body internal and external information and the multiple joint positions with the resampled reference information. It is preferable.

[0009] The video data is Appraiser The data may be moving image data captured from a fixed point with the subject being a moving image.

[0011] The exercise evaluation method of the present invention includes a reference storage step, a posture estimation step, a body internal and external information calculation step, an evaluation step, and a display control step. A motion evaluation device including a reference storage unit, a posture estimation unit, an internal and external body information calculation unit, an evaluation unit, and a display control unit, The step of estimating the posture of the subject is performed by storing reference information that serves as a reference for evaluation. The posture estimation unit: The step of estimating the positions of each of a plurality of joints of the person being evaluated based on the video data of the person being evaluated for the body movement includes: The body internal and external information calculation unit: Using the height and weight of the person to be evaluated and the model data, at least one of muscle activity, joint load, center of pressure, and reaction force is obtained as internal and external body information for the person to be evaluated. The evaluation step includes: The evaluation unit: The body movement of the person being evaluated is evaluated based on the reference information, the positions of multiple joints, and the internal and external body information. Display Control The steps include: An image showing the evaluation result obtained in the evaluation step is generated. The body internal and external information calculation step performs a scaling conversion of the model data, in which weight allocations of segments and center of gravity positions for a certain height and weight are set, based on the height and weight of the person to be evaluated, to obtain model data for the person to be evaluated that is tailored to the person to be evaluated, and obtains body internal and external information using the obtained model data for the person to be evaluated. The evaluation step evaluates the body movement of the person to be evaluated by comparing the body internal and external information and the positions of multiple joints from the first key frame image to the second key frame image selected from the time-series data constituting the video data with reference information.

[0012] The exercise evaluation program of the present invention comprises: A reference storage step, a posture estimation step, a body internal and external information calculation step, an evaluation step, and a display control step. Each step is performed by a computer. The reference storage step stores reference information that serves as a reference for evaluation. The posture estimation step estimates the joint positions of the multiple joints of the person being evaluated based on the video data of the person being evaluated for body movement. The body internal and external information calculation step uses the height and weight of the person being evaluated and the model data to determine at least one of muscle activity, joint load, center of pressure, and reaction force as body internal and external information. The evaluation step evaluates the body movement of the person being evaluated based on the reference information, the multiple joint positions, and the body internal and external information. The display control step generates an image showing the evaluation result obtained in the evaluation step. The body internal and external information calculation step obtains model data of the person being evaluated that is tailored to the person being evaluated by performing a scaling conversion of the model data in which the weight allocation of segments and the center of gravity position for a certain height and weight are set based on the height and weight of the person being evaluated, and obtains body internal and external information using the obtained model data of the person being evaluated. The evaluation step evaluates the body movement of the person being evaluated by comparing internal and external body information and multiple joint positions from a first key frame image to a second key frame image selected from time series data constituting the video data with reference information. Effect of the Invention

[0013] According to the present invention, physical information of the person being evaluated can be grasped more accurately. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is an explanatory diagram of an exercise evaluation system including an exercise evaluation device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is an explanatory diagram of an imaging method performed by a camera. [Figure 3A] 1 is a configuration diagram of an exercise evaluation device according to an embodiment of the present invention. [Figure 3B] FIG. 13 is a configuration diagram of an exercise evaluation device according to another embodiment. [Figure 4] FIG. 11 is an explanatory diagram showing an example of an image displayed on the terminal device. [Diagram 5] FIG. 13 is a configuration diagram of an exercise evaluation device according to another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] In FIG. 1, the exercise evaluation system 10 is for evaluating the body movement of the person to be evaluated who is the subject of the evaluation. The exercise evaluation system 10 includes an exercise evaluation device 11 and a terminal device 12, which are capable of transmitting and receiving various data via a communication network CN, as is the case in this example. The terminal device 12 is, for example, a terminal device used by the person to be evaluated, and may be any device equipped with a camera 12a (see FIG. 2) that captures an image of the person to be evaluated, and may be, for example, a smartphone, which will be described later. The exercise evaluation system 10 evaluates the body movement of the person to be evaluated who is the subject of the evaluation, using the exercise evaluation device 11, based on a moving image captured by the terminal device 12, and displays the evaluation result on a display 29 (see FIG. 3A), which is the display unit of the terminal device 12.

[0016] The exercise evaluation system 10 may further include a client terminal (not shown). The client terminal is capable of transmitting and receiving various data to and from the exercise evaluation device 11 via the communication network CN, and the evaluation results are displayed on a display serving as a display unit of the client terminal. The client terminal is a terminal device used by a client, such as a trainer, physical therapist, or osteopath who provides guidance on physical training for the person to be evaluated, and is not particularly limited as long as it is a computer equipped with a display unit, such as a smartphone or a mobile terminal. The terminal device 12 and the client terminal may operate on a browser by receiving a program that operates on a browser from the exercise evaluation device 11, or may be incorporated with a specific application software and operate by executing the program of this application software.

[0017] The terminal device 12 is installed in a state where the position and the orientation are fixed. As a result, the moving image captured by the camera installed in the terminal device 12 is obtained as a fixed-point captured moving image captured from a fixed point. The terminal device 12 is preferably arranged so that the whole body of the person to be evaluated moving during the evaluation target time, that is, from the start to the end of the evaluation target time, is captured, and this is also done in this example. In this example, as shown in FIG. 2, a walking line (walkway) WL extending linearly on the floor surface is set as the area in which the person to be evaluated P moves, and the walking of the person to be evaluated P from one end to the other end on this walking line WL is taken as an example of the movement of the person to be evaluated P. The terminal device 12 is arranged so that the imaging optical axis L of the camera 12a intersects with the walking line WL when viewed from above. The height of the camera 12a from the floor surface is set within a range of approximately 0.6 m to 1.2 m from the floor surface so that the whole body of the person to be evaluated P is captured, for example, 1 m. In this way, the whole body of the person P walking on the walking line WL is imaged from the feet to the head during the evaluation time. In this example, the terminal device 12 is positioned so that the walking line WL and the imaging optical axis L are perpendicular to each other at the center of the walking line WL. However, as long as the whole body of the person P is imaged, the terminal device 12 may be positioned so that the imaging optical axis L intersects with the walking line WL at an angle other than 90° at a position shifted from the center of the walking line WL, as shown by the two-dot chain line in FIG. 2.

[0018] The motion evaluation device 11 is for evaluating the physical movement of the person to be evaluated. The motion evaluation device 11 is an example of an embodiment of the present invention, and each part shown as an example in FIG. 3A can be omitted or replaced with another part as appropriate.

[0019] Exercise evaluation device 11 includes a pre-processing unit 13, a posture estimation unit 15, a first post-processing unit 17, an internal and external body information calculation unit 19, a second post-processing unit 21, a reference storage unit 23, an evaluation unit 25, and a display control unit 27. Exercise evaluation device 11, together with a display 29 of the terminal device 12, constitutes an exercise evaluation system 10 (see FIG. 1 ).

[0020] The pre-processing unit 13 includes a frame rate (hereinafter, referred to as FPS, where FPS is an abbreviation for frames per second) adjustment unit 31 and a color correction unit 32. The pre-processing unit 13 preferably further includes an acquisition unit (not shown) and a calibration processing unit (not shown), and this example also includes these.

[0021] The video data 33 input to the pre-processing unit 13 may be either video data captured with the subject as the subject, or video data generated from a plurality of still images captured with the subject as the subject. Video data captured with the subject as the subject includes, for example, data captured with a monocular camera provided on a smartphone, and in this example, a monocular camera provided on a smartphone as the terminal device 12 (see Figs. 1 and 2) is used as the camera 12a (see Fig. 2) described above, and the video captured by this is used as the video data 33.

[0022] Since the camera 12a is fixed as described above and images are captured, the distortion of the subject image changes depending on the position of the subject within the imaging range. The manner in which this distortion changes varies depending on the performance of the lens of the camera 12a and also varies depending on the attitude (height, direction) of the camera 12a. Therefore, it is preferable to use the camera 12a to capture a moving image for correcting the distortion and send it to the above-mentioned acquisition unit before starting to capture the image of the subject P (see FIG. 2), and to calibrate the internal parameters of the camera 12a. As an example of the moving image for calibration, there is a moving image of a checkerboard in which white and black rectangular areas are alternately formed as a subject, the checkerboard being captured while moving across the entire imaging range.

[0023] The resolution and FPS when imaging with the camera 12a are not particularly limited, but it is preferable to use the same imaging method including the resolution and FPS for imaging for calibration and imaging of the person being evaluated P in order to prevent internal parameters from changing due to differences in imaging methods by the camera 12a, in terms of using the same internal parameters. In this example, the imaging for calibration and imaging of the person being evaluated P are set to the same resolution of 4K (2160p) and the same FPS of 120 fps. In imaging of the person being evaluated P, the resolution may be set according to the distance between the person being evaluated P and the camera 12a, and it is preferable to set the resolution to at least 4K when the distance is relatively large, for example, 10 m. In imaging of the person being evaluated P, the FPS may be set higher in order to suppress the occurrence of blur, and in this example, the FPS may be set to 240 fps instead of 120 fps. When the acquisition unit acquires moving image data for calibration, the acquisition unit sends the acquired moving image data 33 to the above-mentioned calibration processing unit, and when the acquisition unit acquires moving image data 33 in which the person being evaluated P is imaged, the acquisition unit sends the acquired moving image data 33 to the FPS adjustment unit 31.

[0024] If the next shooting is performed using the same shooting method such as the camera 12a used, the attitude of the camera 12a, and the settings of the camera 12a (for example, resolution and angle of view), there is no need to perform calibration in this next shooting. This is because the internal parameters do not change. In this example, while the above-mentioned terminal device 12 is logged in and connected to the motion evaluation device 11, a calibration execution instruction button for instructing to perform calibration is displayed on the touch panel display provided as the display of the terminal device 12. A configuration is made such that touching this button causes acquisition of a moving image of an image of a checkerboard and execution of calibration by a calibration processing unit.

[0025] The calibration processing unit finds the internal parameters of the camera as camera 12a based on the received video data 33, and sends the found internal parameters to the posture estimation unit 15 via the FPS adjustment unit 31 and the color correction unit 32. When the pre-processing unit 13 performs processing to reduce the resolution as described below, the calibration processing unit finds the internal parameters of camera 12a after converting the resolution of the received video data to the same resolution as the frame images sent from the pre-processing unit 13 as described below. This prevents the internal parameters from changing, and also maintains the accuracy of the estimation by the posture estimation unit 15.

[0026] Depending on the resolution of the acquired video data 33, the pre-processing unit 13 may further perform a resolution reduction process to reduce the resolution of each frame image in order to improve the processing efficiency of the next process and subsequent processes. The pre-processing unit 13 in this example reduces the resolution to full HD (high definition, 1080p), and it has been confirmed that even with full HD resolution, the accuracy in the next process and subsequent processes, such as the accuracy in calculating the joint position described below, is maintained. For example, it has been confirmed that the error in the joint position in the global space and when the position of the pelvis is not fixed is 26.00 cm. It has also been confirmed that the error in the case of fixing the position of the pelvis is 12.02 cm. Note that, in cases where the subject image of the person to be evaluated P (see FIG. 2) is excessively small in the video and the resolution is relatively insufficient, it may be preferable not to perform the resolution reduction process, so it is performed as necessary. In this way, it is advisable to determine whether or not to perform the resolution reduction process depending on the balance between the size of the subject image of the person to be evaluated P in the video and the set resolution. Therefore, in this example, when the distance between the person being evaluated P and the camera 12a exceeds about 10 m, the resolution reduction process is not performed.

[0027] When the FPS adjustment unit 31 acquires the moving image data 33 of the subject, the FPS adjustment unit 31 performs an FPS adjustment process to reduce the number of frames (number of images) per second in response to the acquisition, thereby shortening the processing time in the subsequent process. The subsequent process includes, for example, an estimation process or a calculation process in the body internal and external information calculation unit 19. The FPS value and the number of frame images due to the reduction in FPS are not particularly limited. In this example, the FPS is reduced to 30 fps, and the FPS of the moving image previously trained by the AI ​​(Artificial Intelligence) used in the posture estimation unit 15 is also 30 fps. In this way, the FPS adjustment unit 31 reduces the FPS of the moving image to be the same as the FPS of the moving image trained by the AI ​​used in the posture estimation unit 15, thereby maintaining the accuracy of the estimation in the posture estimation unit 15. The FPS-adjusted moving image data obtained by the FPS adjustment unit 31 is output to the color correction unit 32.

[0028] When performing the resolution reduction process, the pre-processing unit 13 may perform either the resolution reduction process or the FPS reduction process first. The pre-processing unit 13 outputs the time-series data of the frame images obtained through the resolution reduction process and the FPS reduction process to the color correction unit 32.

[0029] The color correction unit 32 performs color correction processing such as brightness, contrast, and white balance on the FPS adjusted video data input from the FPS adjustment unit 31. This allows, for example, an AI (Artificial Intelligence) used in the body internal and external information calculation unit 19 to more reliably recognize the subject image as a person, for example, when the image is too dark depending on the shooting conditions. The color corrected video data obtained by the color correction unit 32 is output to the posture estimation unit 15 and the display control unit 27.

[0030] The posture estimation unit 15 incorporates a posture estimation model M1 and an AI, estimates the positions of each of the joints of the person being evaluated (hereinafter referred to as joint positions) from the color-corrected video data input from the color correction unit 32, and outputs them in three-dimensional coordinates. The coordinate system is the world coordinate system. The estimation process is performed using a preset posture estimation model, and the posture estimation model M1 has a graph structure with vertices and edges, and the joints are associated with IDs (identification information), which will be referred to as joint IDs in the following explanation. The time-series data of the joint positions obtained by the posture estimation unit 15 is output to a low-pass filter 35 of the first post-processing unit 17.

[0031] The first post-processing unit 17 includes a low-pass filter 35, a joint angle calculation unit 36, a velocity calculation unit 37, and a virtual marker addition unit 39. The low-pass filter 35 performs frequency filtering on the time-series data of the joint position to remove high-frequency components and leave only low-frequency components, thereby making the time-series data smooth in the displacement of the joint position over time. The time-series data of the joint position with smooth displacement by the low-pass filter 35 (hereinafter referred to as joint position filter data) is output to each of the joint angle calculation unit 36, the velocity calculation unit 37, the data selection unit 41 and the data analysis unit 42 of the evaluation unit 25, the virtual marker addition unit 39, and the body internal / external information calculation unit 19. The low-pass filter 35 may be replaced with either a missing value complementation unit that complements missing values ​​or an abnormal value correction unit that corrects abnormal values.

[0032] When the joint position filter data is input, the joint angle calculation unit 36 ​​calculates the angle between a vertex of the posture estimation model and a neighboring vertex connected by an edge as a joint angle. The joint angle calculation unit 36 ​​is connected to the velocity calculation unit 37, and a data selection unit 41 and data analysis unit 42 (described later) of the evaluation unit 25, and outputs the calculated joint angle to the velocity calculation unit 37, the data selection unit 41 of the evaluation unit 25, and the data analysis unit 42.

[0033] The velocity calculation unit 37 calculates at least one of the joint position velocity and the joint angular velocity. The velocity calculation unit 37 calculates the velocity at which the joint position is displaced as the joint position velocity based on the joint position filter data input from the low-pass filter 35. The velocity calculation unit 37 calculates the joint angular velocity based on the joint angle input from the joint angle calculation unit 36. That is, the joint angular velocity is calculated by first-order differentiation of the time-series data of the joint angle.

[0034] The virtual marker adding unit 39 calculates a virtual marker based on the joint positions. Specifically, the virtual marker adding unit 39 generates virtual markers that maintain a certain relative distance from the positions of two or more determined joint IDs, i.e., the joint positions, and generates a plurality of virtual markers using combinations of a plurality of joint positions.

[0035] The body internal and external information calculation unit 19 includes a joint load calculation unit 45, a muscle activity estimation unit (not shown), a reaction force estimation unit (not shown), a COP (abbreviation of Center of pressure, also called center of pressure or center of foot pressure) estimation unit (not shown), and a control unit (not shown). The control unit comprehensively controls the joint load calculation unit 45, the muscle activity estimation unit, the reaction force estimation unit, and the COP estimation unit. The control unit switches between the joint load calculation unit 45, the muscle activity estimation unit, the reaction force estimation unit, and the COP estimation unit according to input information, for example, and outputs the input information to the switched functional unit to process it. Any of the joint load calculation unit 45, the muscle activity estimation unit, the reaction force estimation unit, and the COP estimation unit may be replaced with the body internal and external information calculation unit 19.

[0036] In this example, a joint load calculation unit 45 is used, and the muscle activity calculation unit, reaction force calculation unit, and COP calculation unit are described later. The joint load calculation unit 45 calculates (calculates) the load on the joint (hereinafter referred to as joint load) as internal and external body information based on the joint position filter data from the low-pass filter 35 and multiple virtual markers from the virtual marker addition unit 39. The joint load calculation unit 45 incorporates model data M2, and the model data is set with weight allocation and center of gravity position of segments (muscles and bones and markers) for a certain height and weight. When the height 46 and weight 47 of the subject and the joint position filter data are input, the model data is scaled and converted based on them to convert it into model data (hereinafter referred to as subject model data) that matches the skeleton of the subject. The joint load calculation estimation unit 45 performs inverse kinematics using the assessee model data obtained by scaling conversion and the joint position filter data to which the joint position filter data virtual markers have been added, and calculates the time-series joint angles of the assessee model data. The joint load (joint torque) is calculated using inverse dynamics as a result of the inverse kinematics calculation, and is output to the second post-processing unit 21. In this way, the joint load calculation estimation unit 45 of this example calculates and finds the value of the joint load.

[0037] The height 46 and the weight 47 are the measurement data of the assessee at the time of the assessment. The height 46 and the weight 47 may be stored in advance in a data storage unit at the time of the assessment (not shown) at the time of the measurement, or may be input to the joint load calculation estimation unit 45 by an input operation from the terminal device 12.

[0038] The second post-processing unit 21 is a low-pass filter, and this low-pass filter is configured in the same manner as the low-pass filter 35. When time-series data of any of the joint load, muscle activity, reaction force, and COP (in this example, time-series data of the joint load) is input from the body internal and external information calculation unit 19, the second post-processing unit 21 performs frequency filtering processing to remove high-frequency components and leave only low-frequency components, thereby making the time-series data smooth in the change of the joint load over time. The time-series data of the joint load that changes smoothly by the second post-processing unit 21 (hereinafter referred to as joint load filter data) is output to the data selection unit 41 and data analysis unit 42 of the evaluation unit 25.

[0039] The standard storage unit 23 stores at least one of the standard data and history data, which is various past data of the person being evaluated, as standard data. The standard data includes various average values ​​calculated from data of multiple people sampled as persons who serve as standards for evaluation and a numerical range including this average value (a range from (a first threshold value smaller than the average value) to (a second threshold value larger than the average value)), which are used as standard information. The various average values ​​are, for example, the average values ​​of height and weight, color-corrected moving image data, joint position filter data, time-series data of joint angles, joint angular velocity, joint position velocity, joint load, joint load filter data, etc. The history data may be various past data of other people different from the person being evaluated.

[0040] The evaluation unit 25 includes the data selection unit 41 and the data analysis unit 42. When at least one of the joint position filter data as an output value from the posture estimation unit 15, the joint angle from the joint angle calculation unit 36, the joint angular velocity and joint position velocity from the velocity calculation unit 37, and the joint load filter data as an output value from the body internal / external information calculation unit 19 is input, the data selection unit 41 selects a frame (referred to as a key frame) to be evaluated from the movement of the person to be evaluated based on the input data and the motion type 49 previously confirmed in the motion type storage unit (not shown), and also selects a similar key frame from the reference storage unit 23. A plurality of frames may be selected. The frames are numbered, and the data selection unit 41 outputs the number (key frame number) of the key frame selected as a feature point selection result to the data analysis unit 42 and the display control unit 27. The motion type 49 is, for example, walking, batting swing, etc. The above-mentioned movement type storage unit may be provided outside of exercise evaluation device 11, or may be provided in exercise evaluation device 11 as a functional unit constituting exercise evaluation device 11.

[0041] When evaluating the body movement of walking, for example, a frame image showing the heel touching down on one side is specified and selected as a first key frame image, and a frame image showing the next heel touching down on the same side is specified and selected as a second key frame image. Then, all of the internal and external body information such as joint angles and / or joint torques from the first key frame image to the second key frame image are resampled to generate a predetermined number of frame images, which makes it easy to compare with the reference data. Similarly, for the reference data, it is preferable to specify and select the first key frame image and the second key frame image from the time series data previously acquired, and resample the data from the first key frame image to the second key frame image to generate the above-mentioned predetermined number of frame images. This allows time series data with the same number of frame images to be compared, which further improves processing efficiency and provides highly accurate evaluation results.

[0042] When evaluating a baseball batting swing as a body movement, multiple key frames can be defined, such as the time when the front foot takes off from the stance, the time when the knee is at its highest point, the time when the front foot touches the ground, the time when the upper limbs face forward, and the time when the rotation of the upper body including the trunk and upper limbs stops and the swing ends. By selecting and resampling these key frames, it is possible to synchronize and compare the operation of the reference memory unit 23 with the time series. The reference data may be publicly known data from a paper or the like, or may be past data about the person being evaluated.

[0043] When a frame with a key frame number is input from the data selection unit 41, the data analysis unit 42 calculates the following items 1 to 3. 1. The difference between the reference data from the reference storage unit 23 and various data obtained for the subject (color-corrected video data, joint position filter data, joint angle, joint angular velocity, joint position velocity, joint load filter data), and the difference is obtained for each type of data. 2. Evaluation value for each of the above data based on 1 above, expressed as a decimal point from 0 to 5 3. An overall rating of 0 to 5 based on the evaluation scores in 2 above.

[0044] Specifically, the data analysis unit 42 calculates the loss value L for each of the joint position, joint angle, joint angular velocity, joint position velocity, etc., for the frame indicated by the key frame number, using the following formula (A). The symbols in formula (A) have the following meanings.

[0045] i : Number of joints or internal and external body information (i = an integer between 1 and n, n is the total number found) w i ; i is the weight when calculating the loss value L of the part indicated by i, and is set in advance for each type of operation.

[0046]

number

[0047] For the evaluation value F, a function of f(L) is prepared and the value is set to 0 to 5 with one decimal place, i.e., 0.0 to 5.0. The overall evaluation is the same as above, where all the joints, the information inside and outside the body, and the losses of all the frames are biased with a certain weight, and the function g is used to make it a value of 0 to 5 with one decimal place.

[0048] As described above, the data analysis unit 42 determines the above items 1 to 3 based on the joint positions and the body internal and external information determined by the posture estimation unit 15 and the body internal and external information calculation unit 19, i.e., from the joint position filter data calculated based on the joint positions, the joint angles, the joint position velocities, and the joint load filter data calculated based on the body internal and external information. In this way, the evaluation unit 25 evaluates the body movement of the person to be evaluated based on the reference information, the multiple joint positions, and the body internal and external information.

[0049] The display control unit 27 generates an image that displays at least one of the evaluation results 1 to 3 above, and sends it to the display 29. The display control unit 27 may generate the image by further adding data from the reference storage unit 23, color-corrected image data, a key frame number, or a time of a moving image corresponding to the key frame number, or a key frame indicated by the key frame number, and this is also done in this example.

[0050] In this example, the display 29 displays color-corrected moving image data, joint position filter data which is time-series data, and the like, together with history data. The motion evaluation device 11 is configured to display a radar chart, a graph having a vertical axis indicating the value of the display object and a horizontal axis which is the time axis, text display of numerical values ​​and characters, and a table in combination or by switching between them, depending on the display object. The display 29 displays an image acquired from the display control unit 27.

[0051] The motion evaluation device 11 is composed of a computer, and functions as each of the above-mentioned parts by executing a motion evaluation program. The motion evaluation program causes the computer to execute a reference storage step, a posture estimation step, a body internal and external information calculation step, an evaluation step, and a display control step. The reference storage step stores reference information that serves as the basis of evaluation. The posture estimation step estimates the joint positions of the multiple joints of the person being evaluated based on the video data of the person being evaluated for their body movements. The body internal and external information calculation step uses the height and weight of the person being evaluated and model data to determine at least one of muscle activity, joint load, center of pressure, and reaction force for the person being evaluated as body internal and external information. The evaluation step evaluates the body movements of the person being evaluated based on the reference information, the multiple joint positions, and the body internal and external information. The display control unit generates an image showing the evaluation results obtained in the evaluation step.

[0052] The image displayed on the display 29 shows data based on the joint load calculated by the body internal and external information calculation unit 19 as the evaluation result. Therefore, the physical condition of the person being evaluated can be grasped more accurately. In addition, according to the exercise evaluation device 11, by having the posture estimation unit 15, the body internal and external information calculation unit 19, etc., the physical condition of the person being evaluated is quantified and obtained more objectively, and information on the physical condition that cannot be grasped by visual inspection is also made apparent. Therefore, the evaluation result is obtained as quantitative information, and a certain degree of reliability is ensured for the evaluation result. As a result, for example, when advice is given based on the obtained evaluation result, information with a certain degree of reliability is shared between multiple staff members who rehabilitate a patient or multiple trainers who instruct a person playing sports, and the occurrence of a gap in recognition when, for example, information is passed on to each other is suppressed. In addition, information can be shared remotely, and more detailed and accurate advice can be derived between staff members or trainers.

[0053] In the above example, the posture estimation model M1 and the posture estimation unit 15 incorporating AI are used to estimate the positions of the joints of the subject (hereinafter referred to as joint positions) and output them in three-dimensional coordinates, but this is not limited thereto. For example, the color correction unit 32, the posture estimation unit 15, the first post-processing unit 17, and the body internal and external information calculation unit 19 may be replaced with the following silhouette image generation unit 71, two-dimensional information estimation unit 72, three-dimensional information estimation unit 73, derivation unit 74, and body internal and external information calculation unit 77, as shown in FIG. 3B. The body internal and external information calculation unit 77 includes an external force estimation unit 81, a dynamics analysis unit 82, and a muscle information estimation unit 83. The two-dimensional information estimation unit 72, the three-dimensional information estimation unit 73, and the derivation unit 74 constitute a posture estimation unit 85. In this example, a pre-processing unit 86 having a silhouette image generation unit 71 is provided instead of the above-mentioned color correction unit 32, and the speed calculation unit 37 of the first post-processing unit 17 is used.

[0054] When the time series data of the frame images obtained through the resolution reduction process and the FPS reduction process is input, the silhouette image generating unit 71 generates a silhouette image of the subject, who is the subject, for each frame image in response to this input, and outputs it to the three-dimensional information estimating unit. The silhouette image classifies the inside of the outline of the subject as the foreground and the outside of the outline as the background. The silhouette image may be any of an image in which the foreground is extracted, an image in which the foreground is shown in a different color from the background, and an image in which the foreground is shown in a different pattern from the background, and in this example, an image in which the foreground is extracted is used.

[0055] The two-dimensional information estimation unit 72 estimates time series data (hereinafter referred to as two-dimensional time series data) of the subject's two-dimensional joint positions (hereinafter referred to as two-dimensional joint positions) as two-dimensional information based on the time series data of the frame images.

[0056] The two-dimensional joint positions are the two-dimensional positions of multiple joints, such as elbows, wrists, finger joints, neck, and knees, in each frame image of the time-series data.

[0057] The two-dimensional joint positions are expressed in an image coordinate system. In the image coordinate system, the vertex located at the leftmost and uppermost corner of the image area defined by the xy plane is set as the origin (0,0), and the direction toward the right is treated as the positive direction of the x axis (positive values ​​gradually increase), and the direction toward the bottom is treated as the negative direction of the y axis (negative values ​​gradually decrease). The two-dimensional information estimation unit 72 outputs the estimated two-dimensional information to the three-dimensional information estimation unit 73.

[0058] The method of estimating the two-dimensional information is not particularly limited, and a known method can be used. For example, the two-dimensional joint position can be estimated using DeepPose (described in the following non-patent document 1. Non-patent document 1: A. Toshev and C. Szegedy, "DeepPose: Human Pose Estimation via Deep Neural Networks," 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 2014, pp. 1653-1660.), time series data of frame images constituting previously acquired video data can be input into a convolutional neural network, and each joint point in the image coordinate system can be estimated by regression. Two-dimensional joint positions can be estimated using an improved method that represents the probability distribution of joint positions as a heat map, or improves estimation accuracy by grouping joint candidate points, as in OpenPose (described in the following non-patent document 2: Non-Patent Document 2: Z. Cao, G. Hidalgo, T Simon, S. Wei, and Y. Sheikh. 2021. “OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields,” IEEE Trans. Pattern Anal. Mach. Intell. 43, 1 (Jan. 2021), 172-186.).

[0059] The two-dimensional information estimation unit 72 obtains two-dimensional information from a moving image captured by only one camera 12a, so the only equipment to be installed under the person to be evaluated P is the terminal device 12. This saves space and simplifies the device and system configurations. In addition, since it is only necessary to capture the movement of the person to be evaluated P as a moving image by capturing an image with the camera 12a of the terminal device 12, the preparation for starting the evaluation is also labor-saving.

[0060] When the silhouette image time series data is input from the silhouette image generating unit 71 and the two-dimensional information is input from the two-dimensional information estimating unit 72, the three-dimensional information estimating unit 73 responds to this input by estimating motion information and the position information and angle of the floor surface based on the two-dimensional information. The motion information includes first data of a position dimension and second data of a velocity dimension, which are smoothly displaced along a time series and expressed in a generalized coordinate system, for the assessee P. For example, when the three-dimensional information estimating unit 73 receives time series data of a two-dimensional joint position as two-dimensional information, it generates first data of a position dimension and second data of a velocity dimension, which are smoothly displaced along a time series and expressed in a generalized coordinate system. The three-dimensional information estimating unit 73 estimates the three-dimensional position and angle (angle with respect to the horizontal plane) of the floor surface based on the motion information. The first data and the second data are time series data, and the first data is calculated based on the q data described later. m The second data is q m The three-dimensional information estimation unit outputs the first data and second data obtained by estimation to the derivation unit 74, and outputs the three-dimensional position and angle of the floor surface to the external force estimation unit 81.

[0061] Here, the generalized coordinate system will be described. The generalized coordinate system is a coordinate system capable of expressing the posture of the assessee P by variables according to the degrees of freedom of the model when the assessee P is modeled. In this embodiment, the generalized coordinate system is similar to the generalized coordinate system described in JP 2020-201138 A, for example, in which the head, chest, abdomen, pelvis, left and right thighs, left and right shins, and left and right feet are each defined as a segment SG, and joints JT(i) connecting the segments are defined (where i = 1 to N). Among the segments SG, the segment SG(B) corresponding to the pelvis is defined as the base segment SG(B). The six-degree-of-freedom displacement of the segments SG other than the base segment SG(B) with respect to the absolute coordinate system origin is not a variable of the generalized coordinate system, and only the base segment SG(B) is treated as being displaceable with six degrees of freedom. The six degrees of freedom include three translational directions (XYZ) and three rotational directions (yaw, roll, pitch).

[0062] The posture of the person being evaluated P expressed in the generalized coordinate system (hereafter referred to as the "generalized position") q m is a collection of variables with a position dimension, and is expressed, for example, by the following formula (1). In the following explanation, ' indicates a first-order differential, and '' indicates a second-order differential. In formula (1), q0 is a vector with a total of seven elements, consisting of four elements: the X coordinate, Y coordinate, and Z coordinate of the base segment SG(B), and a quaternion representing the orientation. q i is a vector of rotation angles with a number of dimensions according to the degrees of freedom of each joint JT(i) (where i = 1 to N). For example, a joint that corresponds to a knee is defined as having two degrees of freedom, the bending and straightening direction of the knee and the twisting direction of the shin, and is therefore expressed as a two-dimensional vector. A joint that corresponds to a hip joint is defined as having three degrees of freedom, and is therefore expressed as a three-dimensional vector.

[0063]

number

[0064] The three-dimensional information estimation unit 73 is, for example, a machine learning model that generates positional dimensional data for the assessee P expressed in a three-dimensional generalized coordinate system based on the two-dimensional joint positions input from the two-dimensional information estimation unit 72. The three-dimensional information estimation unit 73 estimates the generalized positions q so as to match the acquired two-dimensional joint positions by using a data generation method such as CVAE (conditional variational autoencoder). m and q m The CVAE illustrated here generates three-dimensional motion information of the subject P smoothly over time and in consistency with the floor surface. Here, the motion information q m and q m Let ' be the state variable x.

[0065]

number

[0066] CVAE is a machine learning model that generates new data based on information about a certain condition. In this example, the state variable x at time t is x t To generate x, the state variable x at time t-1 is used. t-1 In other words, CVAE is x t-1 Given the input, the next state variable x with the highest probability t is generated as the motion state. This probability can be obtained from a probability distribution obtained by learning a dataset containing time series data of multiple motions (time series motion data). This probability distribution is a priori probability distribution learned in the latent space of machine learning, and the latent variable z t Therefore, the probability distribution of the learning parameter θ has a mean value μ θ and standard deviation σ θ Using the above, it can be expressed as the Gaussian distribution formula of the following formula (3): In formula (3) and formulas (11) and (12) described later, N (italic letters in the formulas) represents the Gaussian distribution.

[0067]

number

[0068] More x t and z t The relationship between x and t-1 From x t Using the change amount Δθ from

[0069]

number

[0070] Therefore, x t-1 Since Δθ is derived from the information of x t Since x0, which is the motion information at t=0, cannot be derived from the above formula (4), it is preferable to generate it by creating a probability distribution of x at t=0 using a Gaussian mixture model.

[0071] There are many x t and x t-1 The pair of q and q is used as a data set of motion data (motion data set). For example, the motion data is a human q expressed in a generalized coordinate system calculated using inverse kinematics calculation based on marker information acquired by optical motion capture. m and q m It may also be created using a method that uses an IMU (Inertial measurement unit) or a video camera. good.

[0072] To train CVAE, we search for a learning parameter θ that minimizes the loss function L. The loss function L can be expressed by the following equation (5).

[0073]

number

[0074] Lrec is the training data x t and x t-1 and x reconstructed based on t This is the L2 norm with the hat, and is expressed by equation (6).

[0075]

number

[0076] L joint is the training data x t The joint position J of the rigid link model obtained from t and the reconstructed x t Joint position J of the rigid link model obtained from the hat t The L2 norm with the hat is expressed by Equation 7. t is q at t m q mt are the X, Y, and Z coordinate components of the

[0077]

number

[0078] L contact is a loss function that evaluates the contact point of the foot with the floor and the ground contact state with the floor. Here, CVAE is x t At the same time, a variable c indicates the probability that the contact point is in contact with the floor surface. t The training data for the ground state, which is binary data, c t [0,1] is x t It is determined from the position and speed data of the contact point obtained from the above. It is preferable to set a first threshold value TH1 of the height of the contact point from the floor surface and a second threshold value TH2 of the speed of the contact point in advance, and determine that the contact point has touched the ground when the height of the contact point from the floor surface is equal to or less than TH1 and the speed of the contact point is equal to or less than TH2. There are no particular limitations on either TH1 or TH2, but in this example, the contact point is determined to have touched the ground by setting TH1 to, for example, 0.05 m and the second threshold value to 0.8 m / s. contact L BCE , Lvel and can be expressed by equation (8). L BCE is calculated by using binary cross entropy to find the training data c t and reconstructed c t This improves the accuracy of CVAE's estimation of the ground condition. vel is expressed by equation (9), and the velocity of the contact point in contact state v t It works to suppress the increase of hat.

[0079]

number

[0080]

number

[0081] A motion data set similar to that of the CVAE can be used to train the Gaussian mixture model. Here, x is calculated based on the two-dimensional joint positions obtained from the two-dimensional information estimation unit 72. t A method for estimating the hat will now be described.

[0082] The trained CVAE is a time series of 3D motion information x 0:T This is a deep learning model that can generate x0 from a Gaussian mixture model. t-1 When you enter z, t is sampled x t Therefore, when x0 is input to the CVAE, z1 is sampled in response to this input and x1 is generated. By repeating this until time T, the latent variable z and state variable x from t = 1 to t = T, z 1:T is sampled x 1:T At this point, x 0:T does not match the observed 2D joint position information and generates an error. Therefore, we choose x0 and z 1:TThe final estimated motion information x is obtained by performing an optimization calculation to find the optimal solution of x. Furthermore, since the observed two-dimensional joint positions do not include information about the position of the floor, the floor position dimensional data g is also optimized in the same way. The entire objective function can be expressed by the following equation (10).

[0083]

number

[0084]

number

[0085] ε mot is expressed by equation (11), and ε CVAE and ε init As expressed in the following equation (12), the goal is to maximize the log likelihood for the probability distribution of the CVAE and the Gaussian mixture model, and the closer to the center of the probability distribution, the smaller the optimization cost. Hereinafter, λ is the weight for each term.

[0086]

number

[0087] ε in Eq. (10) data evaluates the loss of observed data. data is expressed by the following equation when the two-dimensional joint positions are input to the three-dimensional information estimation unit 73.

[0088]

number

[0089] In equation (13), when "j" is the joint position, P t jis the three-dimensional position of the joint j, and Π represents the pinhole projection, which is a function that projects the joint j into a two-dimensional image coordinate system using the position and focal length information of the camera 12a (see FIG. 2) used for the image capture. t j is the two-dimensional joint position on the observed image coordinate system. Therefore, the joint position information calculated from the motion information generated by the optimization calculation is projected two-dimensionally, and the equation (13) works to minimize the error with the two-dimensional joint position obtained by the two-dimensional information estimation unit 72. σ t j is the reliability of the estimated 2D joint positions, and more reliable observation data is weighted more heavily. ρ is the Geman-McClure function, which is used to improve robustness against outliers.

[0090] ε in Eq. (10) reg It consists of the following elements:

[0091]

number

[0092] ε skel and ε env is a term that is set so that the movement information generated from the CVAE is smooth in time and is more consistent with the floor surface.

[0093]

number

[0094] l t i and t-l i is the length of the segment calculated from the generated motion information, and acts to minimize the fluctuation from time t to time t-1.

[0095]

number

[0096] c t is the contact point p generated by the CVAE t j The first term of equation (16) is p t j The second term of equation (16) suppresses the movement of the contact point p t j When is in ground state, p t j The height z from the floor of the object is made lower than a threshold value δ.

[0097] P02A-0064 ε gnd is the initial value g in optimizing the floor position data init The change from the initial value g is suppressed by the following equation (17). init In this example, it is a horizontal plane with a height of 0m.

[0098]

number

[0099] By solving the optimization problem using the objective function described above, we obtain three-dimensional time-series motion information x 0:T Estimate.

[0100] When the derivation unit 74 receives the second data from the three-dimensional information estimation unit 73, the derivation unit 74 first differentiates the second data in response to this input, thereby deriving third data of the dimension of acceleration expressed in the generalized coordinate system. This third data is time-series data. The derivation unit 74 in this example calculates the generalized velocity [q m ]' is differentiated first, and the generalized acceleration [q m The derivation unit 74 associates the first data and second data input from the three-dimensional information estimation unit 73 with the derived third data, and outputs the associated data to each of the external force estimation unit 81 and the dynamics analysis unit 82. The derivation unit 74 also outputs the first data and second data input from the three-dimensional information estimation unit 73 to the muscle information estimation unit 83.

[0101] The three-dimensional information estimation unit 73 incorporates model data of a posture estimation model and an AI. When the time series data of a silhouette image from the silhouette image generation unit 71 and the time series data of two-dimensional joint positions from the two-dimensional information estimation unit 72 are input, the three-dimensional information estimation unit 73 responds to these inputs and estimates a three-dimensional posture (hereinafter referred to as a three-dimensional posture) by an AI using the model data. In this example, the AI ​​generates data using a conditional variational autoencoder (CVAE) that can impose conditions on the generated data. This AI estimates a three-dimensional posture at time t+1 from the three-dimensional posture at time t by having the AI ​​learn the time series data of the three-dimensional posture calculated from optical motion capture data or the like using a skeleton model and inverse kinematics. At this time, the three-dimensional joint positions are projected onto a two-dimensional plane on the image coordinate system using the internal parameters obtained by the calibration processing unit described above, and an optimal three-dimensional posture is estimated so as to minimize the error with the two-dimensional joint positions obtained by the two-dimensional information estimation unit 72. Therefore, in this example, the calibration processing unit described above is used. If the two-dimensional joint positions obtained by the two-dimensional information estimation unit 72 are located outside the outline of the silhouette shown in the silhouette image, the silhouette image can be used as a judgment material for judging that the estimation data is an incorrect estimation and ignoring the estimation data. However, the method for acquiring the three-dimensional information data when creating the data set is not important.

[0102] The above-mentioned model data has 6890 vertices (hereinafter referred to as skin vertices) set on the skin (body surface) of the body, and is expressed as body surface data by forming multiple triangular polygons using three adjacent vertices among them. The vertices are identified by IDs (identification information) ranging from 0001 to 6890, and some of these vertices are associated with joint positions. This allows the IDs to be specified (specified), and as a result, unique skin vertex coordinates are obtained for any posture, and joint angles can be recalculated using inverse kinematics. The three-dimensional posture includes at least the coordinates of the joint positions, joint rotation vectors, skin vertex coordinates, the position and normal vector of the floor surface, and the timing of contact with the floor surface, which are shown in three dimensions, and the three-dimensional information estimation unit 73 estimates these as the three-dimensional posture and sets them as the above-mentioned three-dimensional information. The skin vertex coordinates are the coordinates of 6890 vertices on the body surface as described above. The floor surface normal vector is an upward unit vector that has its origin on the floor surface and is perpendicular to the floor surface.

[0103] When the first to third data are input from the derivation unit 74 and the three-dimensional position and angle of the floor surface are input from the three-dimensional information estimation unit 73, the external force estimation unit 81 responds to these inputs by kinetic analysis based on the first to third data, the three-dimensional position and angle of the floor surface, and the above-mentioned evaluation subject model data to estimate the external force from the floor surface that is in contact with the evaluation subject P (see FIG. 2). That is, the motion information is input to the above-mentioned evaluation subject model data, and whether or not each contact point is in contact with the floor surface is determined based on the speed and height from the floor surface of multiple contact points set and arranged on the soles. Specifically, if the speed of the contact point is equal to or less than the above-mentioned second threshold value TH2 and the height is equal to or less than the above-mentioned first threshold value, it is determined that the contact point is in contact, and otherwise it is determined that the contact point is not in contact. A floor reaction force is generated from the contact point that is determined to be in contact with the floor surface. Therefore, the external force estimation unit 81 performs kinetic analysis based on the motion information and estimates the external force from the contact point by optimization calculation so that the equation of motion of the whole body is established. In this example, the generalized position q generated above m , generalized speed q m ', generalized acceleration q m'', based on the three-dimensional position and angle of the floor surface and the subject model data, the external force is estimated as time series data.

[0104] The objective function can be expressed by the following equation (18), and an inverse dynamics calculation is performed.

[0105]

number

[0106] In formula (18), F i N on each foot f is the ground reaction force (vector) at the contact point i. m ) is the mass property vector, and C(q m ,q m ') is the centrifugal force vector and the Coriolis force vector, G(q m ) is the gravity vector. E is the external force vector, and F i This constraint ensures that no dynamic error occurs between the root joint, the pelvis, and the global space. F i is the maximum value F i max The maximum value F i max can be set to any value, and in this example, the weight BW is multiplied by the acceleration of gravity, and this multiplication value is further multiplied by 0.4. The value by which the multiplication value obtained by multiplying the weight BW by the acceleration of gravity is further multiplied is not limited to 0.4 as in this example, but is preferably within a range of 0.28 to 0.52, and by setting it within this range, the error variation is suppressed to be extremely small. It is particularly preferable that the value by which the multiplication value obtained by multiplying the weight BW by the acceleration of gravity is further multiplied by 0.4 as in this example, and it has been confirmed by sensitivity analysis that the change in error is reliably suppressed to less than 1% even if the value is changed by 30% up or down.

[0107] When the external force is input from the external force estimation unit 81 and the first to third data are input from the derivation unit 74, the dynamics analysis unit 82 performs dynamics analysis based on the external force, the first to third data, and the musculoskeletal model of the person to be evaluated in response to these inputs. The dynamics analysis unit 82 thereby calculates each joint torque of a plurality of joints connecting each of a plurality of segments.

[0108] In the dynamics analysis unit 82 of this example, the generalized force vector τ is calculated by inverse dynamics calculation using the following formula (19). The generalized force vector τ includes translational and rotational forces expressed in a generalized coordinate system, and the rotational force is the joint torque.

[0109]

number

[0110] When the first and second data are input from the derivation unit 74 and the joint torque is input from the dynamics analysis unit 82, the muscle information estimation unit 83 responds to these inputs and estimates at least one of muscle activity and muscle tension based on the first and second data, the assessee model data, and each joint torque. Both muscle activity and muscle tension are time-series data.

[0111] The joint torque is generated by the muscle tensions of the corresponding multiple muscle models, and the muscle information estimation unit 83 distributes the joint torque to the muscle tensions of each muscle model by optimization calculation. The optimization calculation is performed so as to minimize the sum of squares or sum of cubes of the muscle activity of the whole body under the constraints of the Hill-type muscle model. This results in the output of more optimal muscle activity and muscle tension. The Hill-type muscle model determines muscle tension from the increase or decrease in muscle activity, i.e., the magnitude of muscle activity that increases or decreases over time, depending on the muscle length and contraction speed (speed of change in muscle length) during exercise. The muscle length and contraction speed are determined from the first data and the second data.

[0112] Here, in equation (20), “j” is the degree of freedom of the joint. In this example, the generalized force vector τ j and muscle tension F m The relationship is expressed by the following equations (20) and (21). m is the muscle activity of muscle m, and F m 0 is the maximum muscle tension of muscle m, and l m is the muscle length, and v m is the contraction velocity of muscle m, and r m,j is the moment arm of muscle m at joint degree of freedom j.

[0113]

number

[0114]

number

[0115]

number

[0116]

number

[0117]

number

[0118] The muscle information estimation unit 83 solves an optimization problem using objective functions such as the above equations (22) to (24) to obtain a generalized force vector τ jto muscle tensions. J denotes an objective function, and minJ denotes minimization of J. p is an arbitrary exponent that is set in advance. It is preferable to set p within the range of 2 to 4. A value of 2 or more ensures that the accuracy of estimating muscle activity is improved compared to a value less than 2, and a value of 4 or less ensures that the accuracy of estimating muscle activity is improved compared to a value exceeding 4.

[0119] The body internal and external information calculation units 19 and 77 can re-output the joint angles, and either the re-output joint angles or the joint angles output from the three-dimensional information estimation unit 73 may be used.

[0120] At least one of the muscle activity and muscle tension estimated in the above manner is transmitted to the terminal device 12 or the like as described above in this example, and is displayed on the display 29, which is the display unit of the terminal device 12. As described above, in this example, even though only one camera 12a is used, the joint torque and muscle tension can be estimated. Furthermore, with the above configuration, even though only one camera 12a is used, the three-dimensional positions of the joint positions and the like are generated (estimated) with high accuracy. Furthermore, since the external force from the floor surface is estimated after estimating the three-dimensional position and angle of the floor surface, the joint torque and muscle tension can be estimated with high accuracy. Furthermore, with the above configuration, the three-dimensional positional relationship between the floor surface and the leg is also estimated, so that the joint torque, muscle tension, muscle activity, and the like can be obtained with extremely high accuracy.

[0121] When data of an image to be displayed is input from the display control unit 27, the terminal device 12 displays the image G on the display 29 as shown in Fig. 4. The image G has a moving image section Ga showing a moving image, and a result image section Gb showing an evaluation result such as a graph and / or numerical values. However, the moving image section Ga and the result image section Gb may be configured to form different images G, and the images may be switched between them.

[0122] The moving image section Ga displays a moving image including a three-dimensional human body model MA shown as the person being evaluated P (see FIG. 2), and the movement of the three-dimensional human body model MA in the moving image reflects the body movement of the person being evaluated P. In the example shown in FIG. 4, a moving image is displayed in which the three-dimensional human body model MA walks from the right side to the left side of the page in FIG. 4 as the person being evaluated P walking on a flat floor. The moving image shows at least the walking start to the walking end, and the progress of the video is shown by a progress bar BP that extends in a line from "START" indicating the start time to "END" indicating the end time. The image G includes a pause button B1 for pausing the video in progress, a stop button B2 for stopping it, a progress button (not shown) for progressing the video, and the progress of the video is turned on and off according to touch operations on these.

[0123] In this example, the three-dimensional human body model MA is generated by estimating three-dimensional information, so the orientation of the three-dimensional human body model MA can be changed by a swipe operation in which a finger is moved while touching the moving image portion Ga of the display 29. In addition, the ground contact timing and ground contact position are also specified in three-dimensional space, and these time series data are obtained in association with each time series data of the joint positions and joint angles of the whole body, so that regardless of the orientation of the three-dimensional human body model MA, the walking direction can be displayed in a moving image, and the sole of the foot is displayed in a state that is aligned with the floor surface. In this way, according to the exercise evaluation system 10 (see FIG. 1), in addition to the system configuration being simple and space-saving being realized, the three-dimensional human body model MA reflects the body movement of the person to be evaluated P and displays it on the terminal device 12, and the display is variable in three dimensions and aligned with the floor surface. As a result, the person to be evaluated P, the operator such as a trainer, who uses the terminal device 12, can repeatedly visually confirm the body movement of the person to be evaluated P, and can deepen his / her consideration of the body movement. Also, as described above, the evaluation result for one walking cycle from the first frame image to the second frame image is obtained from the time series data from the start of walking to the end of walking, and the resampling process is performed on this walking cycle, and the reference information is also composed of the same number of frame images by the resampling process. Therefore, the evaluation result can be displayed in the result image section Gb, and the current result of the person to be evaluated P and the reference information are displayed in the same range in the same graph, and the processing time required to generate the image is also very short. As a result, although the system configuration is simple (simple), precise evaluation results are quickly displayed on the terminal device 12, and it is easy to understand the current evaluation result of the person to be evaluated P and to compare the evaluation result with the reference information.

[0124] The video image section Ga may display a video image including the three-dimensional human body model MAp together with a video image including the three-dimensional human body model MAp. The movement of the three-dimensional human body model MAp reflects the body movement at the time of the past evaluation of the assessee P, "2023.9.30". In contrast, the current evaluation time is shown in FIG. 4 as "2023.10.23". The three-dimensional human body model MA and the three-dimensional human body model MAp may be displayed by enlarging or reducing the time of either one so as to match the grounding timing of each other, or may be displayed together. In this example, the grounding timing and the grounding position are specified in a three-dimensional space, so that either one can be displayed. In the example shown in FIG. 4, the three-dimensional human body model MA and the three-dimensional human body model MAp are displayed side by side in the horizontal direction, but the positional relationship between the two may be other modes, such as being vertically aligned. In the example shown in FIG. 4, the three-dimensional human body model MA and the three-dimensional human body model MAp are displayed at approximately the same size, but one of them may be displayed smaller than the other. Furthermore, an operation button (not shown) for switching the display of the three-dimensional human body model MAp on and off may be provided, and the three-dimensional human body model MAp may be switched between displayed and hidden by, for example, a touch operation on this operation button.

[0125] In the example shown in FIG. 4, the result image section Gb includes a graph D1 of the joint angle of the left knee joint in one walking cycle for "Left-knee" indicating the left knee joint, and a numerical display D2 indicating the minimum value of the joint angle, "maximum flexion", and the maximum value, "maximum extension". The graph D1 is displayed by superimposing a curve L1 of the joint angle of the left knee joint for the assessee P and a curve L2a of the joint angle of the left knee joint, which is a reference walking variable (shown by a two-dot chain line in FIG. 4). The reference walking variable is an example of the standard data among the reference data. In the example shown in FIG. 4, the curve L2b of the joint angle of the left knee joint, which is history data (shown by a one-dot chain line in FIG. 4), and the joint angle of the right knee joint, which is the opposite side, are further displayed by superimposing the curve L1. This makes it easy to compare the past and present of the assessee P. The curve L2b showing the history data is generated and displayed in conjunction with the evaluation time "2023.9.30" of the three-dimensional human body model MAp shown in the moving image section Ga, but is not limited to this. The curve L2c (shown by a dashed line in FIG. 4) of the joint angle of the right knee joint on the opposite side may also be displayed superimposed on the curve L1. In this example, all of these curves L1 to L4 are displayed superimposed, which allows the body movement of the subject to be considered from various perspectives. A movement operation unit B3 shown by an arrow, for example, for sliding the straight line L3 shown by a chain line extending vertically in the direction along the horizontal axis is provided near the horizontal axis of the graph D1. When the client touches this movement operation unit B3 on the display 11a with a finger and moves it to a predetermined position on the horizontal axis, the straight line L3 is displaced to that position, and the joint angle of the left knee joint shown in the curve L1 that intersects with the curve L1 at the position after the displacement and the walking period corresponding to this joint angle are displayed. For example, in FIG. 4, the joint angle and the corresponding walking cycle are displayed as "17° / 28%."

[0126] The result image section Gb is provided with a switch operation section B4 in a pull-down format for switching the joint to be displayed. When the operator of the terminal device 12 uses this switch operation section B4 to select a joint other than the left knee joint, the result image section Gb including a graph D1 and a numerical display D2 for the selected joint is displayed on the terminal device 12. The result image section Gb in this example is also provided with a "waveform selection" button B6, which allows a graph showing another evaluation result for the left knee joint to be displayed.

[0127] The image G further includes a switch button B5 for displaying the evaluation results for the entire walking cycle, i.e., the entire duration of the image capture. When the operator touches the switch button B5, the result image section Gb is switched to a result image section (not shown) showing the evaluation results for the entire walking cycle. In this manner, the evaluation results for the entire walking cycle may be displayed in the same manner as the evaluation results for one walking cycle.

[0128] 5 differs from the motion evaluation device 11 (see FIG. 1) in that it does not include a virtual marker adding unit 39, in that color corrected moving image data obtained by the color correction unit 32 is sent to the body internal and external information calculation unit 19, and in the processing performed by the body internal and external information calculation unit 19. Description of the same functional units and processing as those of the motion evaluation device 11 will be omitted for the motion evaluation device 51.

[0129] The color correction unit 32 sends the color corrected video data to the joint load calculation estimation unit 45 of the body internal / external calculation unit 19. This joint load calculation estimation unit 45 estimates the load on the joint (hereinafter referred to as joint load) as body internal / external information based on the color corrected video data. The joint load calculation estimation unit 45 incorporates model data M3 different from the model data M2 and AI, and outputs the estimated joint load value to the second post-processing unit 21. In this way, the joint load calculation estimation unit 45 of this example estimates and finds the joint load value. As described above, the joint load calculation estimation unit 45 performs a calculation process to calculate or estimate the joint load value.

[0130] In the motion evaluation device 51 as well, the image displayed on the display 29 shows data based on the joint load calculated by the body internal and external information calculation unit 19 as the evaluation result. This allows the physical condition of the person being evaluated to be grasped more accurately, and the above-mentioned effects obtained by the motion evaluation device 11 can be obtained in the same way.

[0131] The body internal and external information calculation unit 19 in each of the above examples may output at least one of muscle activity, joint load, COP, and reaction force. The body internal and external information calculation unit 19 may perform calculations to estimate or calculate the joint load, COP, and reaction force, and these calculation results may be used when estimating or calculating the muscle activity. Other output values ​​may also be used when estimating or calculating the joint load, COP, and reaction force. In addition, when using joint positions in the estimation calculation of the body internal and external information calculation unit 19, AI is used, which is different from the case of calculation calculation.

[0132] Similarly, the muscle activity estimation calculation unit, reaction force estimation calculation unit, and COP estimation calculation unit described above perform either a calculation process to calculate a predetermined value or a calculation process to estimate. When performing a calculation, the muscle activity estimation calculation unit calculates and outputs the muscle activity of each muscle using a plurality of virtual markers added by the virtual marker addition unit 39, joint position filter data, height 46 and weight 47, and model data M2. The muscle activity of each muscle is calculated by performing the same calculation as that performed by the joint load calculation estimation unit 45 shown in Fig. 3A, followed by optimization calculation (static optimization) based on the joint load.

[0133] The muscle activity estimation calculation unit incorporates AI for estimation calculations, and when video data is input, it estimates muscle activity from the video data, and when joint position filter data is input, it estimates muscle activity from the data.

[0134] When performing calculations, the reaction force estimation calculation unit calculates and outputs a reaction force (floor reaction force) using a plurality of virtual markers added by the virtual marker addition unit 39, joint position filter data, height 46, weight 47, and model data M2. The calculation is performed as follows. That is, first, a plurality of contact points are assigned to the contact surface of the object (the person being evaluated in this example) that contacts the ground (including the floor), and an actuator is placed at each contact point. After that, when the distance between the object and the ground becomes equal to or less than a certain value, the force at each contact point is calculated using the above-mentioned calculation method performed by the joint load calculation estimation unit 45 in the motion evaluation device 11. When performing estimations, the reaction force estimation calculation unit incorporates AI, and when video data is input, it estimates the reaction force from the video data, and when joint position filter data is input, it estimates the reaction force from the data.

[0135] When performing calculations, the COP estimation calculation unit calculates and outputs the COP using the joint position filter data of the multiple virtual markers added by the virtual marker addition unit 39, height 46, weight 47, and model data M2. The COP is found using the reaction force of each contact point, the moment calculated from the reaction force, and the world coordinates of the contact surface. When performing estimation calculations, the reaction force estimation calculation unit incorporates AI, and when video data is input, it estimates the reaction force from the video data, and when joint position filter data is input, it estimates the COP from that data.

[0136] When the body internal / external information calculation unit performs estimation calculation using joint position filter data, two-dimensional joint position filter data may be used. Also, the posture estimation unit 15 may output two-dimensional coordinates. [Explanation of symbols]

[0137] 11, 51 Exercise evaluation device 13, 86 Pretreatment section 15, 85 Posture estimation section 17 First post-processing section 19, 77 Internal and external body information calculation unit 21 Second post-processing section 23 Reference storage section 25 Evaluation Section 27 Display control section 29 Display 31 FPS adjustment section 32 Color Correction Section 33 Video data 35 Low-pass filter 36 Joint angle calculation unit 37 Speed ​​calculation section 39 Virtual marker addition section 41 Data Selection Section 42 Data Analysis Department 45 Joint Load Calculation and Estimation Unit 46 Height Weight 47 49 Action Types M1 Pose Estimation Model M2, M3 model data

Claims

1. a standard storage unit that stores standard information that serves as a standard for evaluation; a posture estimation unit that estimates a joint position of each of a plurality of joints of a subject to be evaluated based on video data of the subject to be evaluated for body movement; a body internal / external information calculation unit that uses the height and weight of the person to be evaluated and model data to obtain at least one of muscle activity, joint load, center of pressure, and reaction force as body internal / external information for the person to be evaluated; an evaluation unit that evaluates the body movement of the person to be evaluated based on the reference information, the joint positions, and the internal and external body information; a display control unit that generates an image showing the evaluation result obtained by the evaluation unit; Equipped with The body internal and external information calculation unit performs a scaling conversion of the model data, in which weight allocations of segments and center of gravity positions for a certain height and weight are set, based on the height and weight of the person to be evaluated, to obtain model data for the person to be evaluated that is tailored to the person to be evaluated, and calculates the body internal and external information using the obtained model data for the person to be evaluated; The evaluation unit is a motion evaluation device that evaluates the physical movement of the person being evaluated by comparing the internal and external body information and the multiple joint positions from a first keyframe image to a second keyframe image selected from time series data constituting the video image data with the reference information.

2. The motion evaluation device of claim 1, wherein the evaluation unit resamples the internal and external body information and the multiple joint positions from the first key frame image to the second key frame image and evaluates the physical movement of the person being evaluated by comparing it with the reference information.

3. A motion evaluation device as described in Claim 2, wherein the evaluation unit compares the resampled internal and external body information and the multiple joint positions with the resampled reference information.

4. 3. The exercise evaluation device according to claim 1, wherein the moving image data is data of moving images captured from a fixed point with the person being evaluated as a subject.

5. A motion evaluation device including a reference memory unit, a posture estimation unit, a body internal / external information calculation unit, an evaluation unit, and a display control unit, the reference memory unit storing reference information that serves as a basis for evaluation; a posture estimation step in which the posture estimation unit estimates a joint position of each of a plurality of joints of the subject to be evaluated based on video data of the subject to be evaluated for body movement; a body internal and external information calculation step in which the body internal and external information calculation unit calculates at least one of muscle activity, joint load, center of pressure, and reaction force of the person to be evaluated as the body internal and external information by using the height and weight of the person to be evaluated and model data; an evaluation step in which the evaluation unit evaluates the body movement of the person to be evaluated based on the reference information, the joint positions, and the internal and external body information; a display control step of generating an image showing the evaluation result obtained in the evaluation step by the display control unit; having The body internal and external information calculation step performs a scaling transformation of the model data, in which weight allocations of segments and center of gravity positions for a certain height and weight are set, based on the height and weight of the person to be evaluated, to obtain model data for the person to be evaluated that is tailored to the person to be evaluated, and calculates the body internal and external information using the obtained model data for the person to be evaluated; The evaluation step is a movement evaluation method for evaluating the body movement of the person being evaluated by comparing the internal and external body information and the multiple joint positions from a first keyframe image to a second keyframe image selected from time series data constituting the video image data with the reference information.

6. a reference storage step of storing reference information serving as a reference for evaluation; a posture estimation step of estimating a joint position of each of a plurality of joints of a subject to be evaluated based on video data of the subject to be evaluated for body movement; a body internal and external information calculation step of calculating at least one of muscle activity, joint load, center of pressure, and reaction force of the subject as body internal and external information using the height and weight of the subject and model data; an evaluation step of evaluating the body movement of the person to be evaluated based on the reference information, the joint positions, and the internal and external body information; a display control step of generating an image showing the evaluation result obtained in the evaluation step; on the computer, The body internal and external information calculation step performs a scaling transformation of the model data, in which weight allocations of segments and center of gravity positions for a certain height and weight are set, based on the height and weight of the person to be evaluated, to obtain model data for the person to be evaluated that is tailored to the person to be evaluated, and calculates the body internal and external information using the obtained model data for the person to be evaluated; The evaluation step is an exercise evaluation program that evaluates the physical movement of the person being evaluated by comparing the internal and external body information and the multiple joint positions from a first key frame image to a second key frame image selected from time series data constituting the video image data with the reference information.

Images