Analysis support device, analysis support method, and analysis support program

Abstract

Provided are an analysis support device, an analysis support method, and an analysis support program capable of more easily analyzing movement of a body. An analysis support device 13 comprises a moving image storage unit 41 and a display control unit 42. The moving image storage unit 41 stores a moving image obtained by imaging a subject person being analyzed with respect to motion as a subject, or a moving image of a three-dimensional human body model being displayed as the subject person. The display control unit 42 generates a graph indicating a temporal analysis result on the basis of the analysis result with respect to the motion of the subject person, and simultaneously displays the moving image and a graph having a progress marker indicating the progress status of the moving image.

Description

Analysis support device, analysis support method, and analysis support program

[0001] The present invention relates to an analysis support device, an analysis support method, and an analysis support program.

[0002] There is a technology for estimating and analyzing human body movements using motion capture. For example, Patent Document 1 discloses a three-dimensional motion analysis device including a photographing means, a marker coordinate calculation means, a marker coordinate storage means, a marker positioning relationship storage means, a reference frame marker ID setting means, and a marker ID setting means. The photographing means photographs multiple markers attached to a subject and outputs video data. The marker coordinate calculation means extracts the positions of the markers in the image data from the image data of each frame of the video data output from the photographing means and calculates the three-dimensional coordinate positions of each extracted marker as marker coordinate data for each frame. The marker coordinate storage means stores the marker coordinate data for each frame calculated by the marker coordinate calculation means. The marker positioning relationship storage means stores the marker IDs of each marker corresponding to predetermined attachment positions of the multiple markers and marker positioning relationship data indicating the positioning relationship of each marker corresponding to these marker IDs. The reference frame marker ID setting means selects, from the marker coordinate data of each frame, marker coordinate data in which the coordinate position of each marker exists in a layout relationship corresponding to the layout relationship of each marker in the marker layout relationship data, sets a marker ID corresponding to the coordinate position of each marker in the marker coordinate data of the reference frame based on the layout relationship of the coordinate positions of each marker in the marker coordinate data of the reference frame and the layout relationship of the marker corresponding to each marker ID in the marker layout relationship data. The marker ID setting means sets marker IDs to coordinate positions corresponding to each marker in the marker coordinate data of a frame other than the reference frame, based on the marker coordinate data of the reference frame in which the marker IDs have been set corresponding to the coordinate positions of each marker by the reference frame marker ID setting means.

[0003] Another example of a method for detecting movement is to use an inertial measurement unit (IMU).

[0004] Japanese Patent Application Laid-Open No. 2004-344418

[0005] However, the three-dimensional motion analysis device described in Patent Document 1 can only partially grasp the body posture corresponding to the analysis results, and there is room for improvement in the ease of analyzing body movements.

[0006] Therefore, an object of the present invention is to provide an analysis support device, an analysis support method, and an analysis support program that make it easier to analyze body movements.

[0007] The analysis support device of the present invention includes a video storage unit and a display control unit. The video storage unit stores video images of a subject whose motion is to be analyzed, or video images of a three-dimensional human body model displayed as the subject. The display control unit generates a graph showing the analysis results over time based on the analysis results of the subject's motion, and simultaneously displays the video and a graph with a progress marker showing the progress of the video.

[0008] The apparatus may further include a three-dimensional human body model storage unit that stores the three-dimensional human body model, and the display control unit may generate a moving image of the three-dimensional human body model based on the analysis results, and the moving image storage unit may store the moving image of the three-dimensional human body model.

[0009] The exercise may include repeated movements. In this case, it is preferable to further include an extraction unit that extracts any repeated unit of movement as a unit exercise, and it is preferable that the display control unit generates a unit motion image and a unit graph that are a motion image and a graph of the unit exercise, and simultaneously displays the unit motion image and the unit graph with a progress marker. The exercise may be walking, and the unit exercise may be a walking cycle.

[0010] Preferably, the extraction unit divides the unit movement into smaller movement phases over time, and the display control unit displays a progress bar extending in one direction indicating the progress of the unit video image in association with the movement phases. In this case, the movement may be walking, the unit movement may be a walking cycle, and the movement phase may be a walking phase.

[0011] It is preferable that the display control unit, in response to a selective display request requesting display with either the right leg or the left leg selected, generates and displays a unit graph with the gait cycle of the selected leg as the unit movement.

[0012] The display control unit preferably displays the selected one of the moving image and the individual moving image in response to a selective display request for displaying either the moving image or the individual moving image in a selected state.

[0013] The analysis support method of the present invention includes a moving image storage step and a display control step. The moving image storage step stores moving images captured of a subject whose motion is to be analyzed as a subject, or moving images of a three-dimensional human body model displayed as the subject. The display control step generates a graph showing the analysis results over time based on the analysis results of the subject's motion, and simultaneously displays the moving image and the graph having a progress marker that indicates the progress of the moving image.

[0014] The analysis support program of the present invention causes a computer to execute the above steps.

[0015] According to the present invention, it is easier to analyze body movements.

[0016] It is an explanatory diagram of a motion analysis system having an analysis support device which is an embodiment. It is an explanatory diagram of an image displayed on a client terminal. It is an explanatory diagram of an image displayed on a client terminal. It is an explanatory diagram of an image displayed on a client terminal. It is a configuration diagram of an analysis support analysis device. It is a processing flow of the analysis support system.

[0017] The analysis support system 10 shown in FIG. 1 analyzes the body movements of a subject (hereinafter simply referred to as the "subject") and displays the analysis results on a display unit (display) of a terminal device. The analysis support system 10 includes a detection unit 11, an analysis support device 13, and a client terminal 15, which is an example of a terminal device. The detection unit 11, the analysis support device 13, and the client terminal 15 communicate with each other, i.e., send and receive various data, via a communication network CN. For example, the detection unit 11 detects the subject's body movements while in motion and transmits motion information data indicating analysis results calculated based on the detection results to the analysis support device 13 via the communication network CN. The transmitted analysis results may include at least a portion of the detected detection results. The analysis support device 13 acquires motion information as an analysis result of analyzing the subject's body movements and transmits data indicating the analysis results to the client terminal 15. As a result, the analysis results are displayed on the display unit 15a (see FIG. 2) of the client terminal 15, and the client using the client terminal 15 can understand and analyze the analysis results.

[0018] The detection unit 11 detects the spatial position of the body and each body part of a subject who is exercising, and the detection results and analysis results are used as exercise information regarding the body movement. One detection unit 11 is sufficient for one subject. Although only one detection unit 11 is depicted in FIG. 1, there may be multiple detection units 11 so that multiple subjects each use one. Furthermore, multiple subjects may share one detection unit 11.

[0019] The client terminal 15 is a terminal used by a client, such as a physical therapist or a trainer who provides physical training guidance to a subject. The client is not limited to these and may also be the subject being analyzed. The client terminal 15 is an example of a display terminal that displays moving images, etc., and is composed of a display unit 15a that displays moving images and graphs, etc., described below, transmitted from the analysis support device 13, and an input unit 15b (see FIG. 4) that allows input operations such as a pause button B2 (see FIG. 2), described below. In this example, the input unit is a touch panel display, which also functions as a display unit. However, the input unit may be a known input device capable of various input operations, such as a mouse or keyboard. The client terminal 15 may be any of a mobile terminal, a smartphone, a personal computer, etc., and is not particularly limited. While FIG. 1 illustrates only one client terminal 15, the number of client terminals 15 may be multiple, so that each client can use one. The client terminal 15 may also be used as a management terminal for managing various settings and / or data of the analysis support device 13. The management terminal may be provided separately from the client terminal 15 .

[0020] The analysis support device 13 performs predetermined processing in response to input of each piece of information from the detection unit 11 and the client terminal 15, and transmits information as a processing result to the client terminal 15 for display. For example, in response to input of exercise information from the detection unit 11, the analysis support device 13 generates various data for displaying on the client terminal 15 an analysis result relating to the body movement indicated by the input exercise information, and displays the data on the display unit of the client terminal 15. Details of the analysis support device 13 will be described later using another drawing.

[0021] The analysis support device 13 and the client terminal 15 are each composed of a computer. The client terminal 15 may operate on a browser in response to a program sent from the analysis support device 13 that runs on the browser, or may be equipped with predetermined application software and operate by executing the program. The analysis support device 13 has a predetermined program installed, and by executing this program, it functions as each unit described below and performs predetermined processing.

[0022] The program causes a computer to execute a moving image storage step and a display control step. The moving image storage step stores moving images captured of a subject whose motion is to be analyzed as a subject, or moving images of a three-dimensional human body model displayed as the subject. The display control step generates a graph showing the analysis results over time based on the analysis results of the subject's motion, and simultaneously displays the moving image and the graph with a progress marker showing the progress of the moving image in a single image.

[0023] As shown in FIG. 2 , the display unit 15a of the client terminal 15 displays an image G having a moving image portion Ga and a graph image portion Gb showing the analysis results. The moving image portion Ga displays a moving image including a three-dimensional human body model MA displayed as the subject. The graph image portion Gb displays the analysis results, such as a graph DG. In this manner, the display unit 15a simultaneously displays a moving image of the three-dimensional human body model MA and a graph in a single image G (hereinafter referred to as "simultaneous display"). The simultaneous display of the moving image and the graph showing the analysis results allows the client to compare the subject's posture and its changes with the analysis results related to the subject's body movements, making it easier for the client to analyze the movement and facilitate consideration and analysis. Furthermore, because a graph is displayed as the analysis results, the analysis results related to the body movements can be understood as being based on a predetermined method, regardless of the client's level of experience, knowledge, or insight.

[0024] In the example shown in FIG. 2 , the moving image section Ga and the graph image section Gb are arranged vertically, with the graph image section Gb disposed below the moving image section Ga. However, the positional relationship between the moving image section Ga and the graph image section Gb in the vertical direction may be reversed. Also, instead of the vertical arrangement, they may be arranged horizontally. In the case of horizontal arrangement, the left-right positional relationship between the moving image section Ga and the graph image section Gb is not particularly limited. Also, the moving image section Ga and the graph image section Gb may be superimposed on a partial area of ​​the other.

[0025] The movement of the three-dimensional human body model MA in the moving image of the moving image section Ga reflects the movement of the subject's body. In the example shown in Fig. 2, the subject is shown as a three-dimensional human body model MA walking on a flat floor surface FS, and a moving image of the three-dimensional human body model MA walking from right to left on the page of Fig. 2 is displayed. Instead of the three-dimensional human body model MA, captured images of the subject exercising may be displayed as moving images.

[0026] The video images are either an overall video image showing the start and end of walking detection detected by the detection unit 11 (see FIG. 1 ), a right walking cycle video image showing a representative walking cycle of the right leg (described later), or a left walking cycle video image showing a representative walking cycle of the left leg. The video image section Ga is provided with a switching operation section B1, and an operation such as a touch operation on this switching operation section B1 constitutes a selective display request operation requesting the display of one of the overall video image, the right walking cycle video image, or the left walking cycle video image. In response to this operation, the selected one is displayed in the video image section Ga. In this way, the display in the video image section Ga is a switching display in which one of the overall video image, the right walking cycle video image, and the left walking cycle video image is selectively displayed. Note that in FIG. 2 , "Overall Video Image" is a selective display request button requesting the display of the overall video image, "Right Walking Cycle Image" is a right walking cycle image, and "Left Walking Cycle Image" is a selective display request button requesting the display of the left walking cycle image. FIG. 2 shows an example in which "Overall Video Image" is selected.

[0027] In the video section Ga, the progress of the video is indicated by a progress bar (progress bar) BP extending horizontally. When the entire video is displayed as a video, the left end of the progress bar BP indicates the start of the video, corresponding to the start of detection, and the right end indicates the end of the video, corresponding to the end of detection. The video section Ga includes a pause button B2 for pausing the video in progress and a progress button (not shown) for progressing the video, and the progress of the video is turned on and off in response to operations on these buttons. The tip B3 of the progress bar, which slides to the right depending on the progress, functions as an operation unit for specifying the temporal position in the time series of the video. The temporal position of the displayed video changes in response to an operation on this tip B3. An operation on the tip B3 is, for example, an operation of sliding it left and right while touching it. The video section Ga also includes an operation button B4 for selecting the progression speed of the video. By selecting this operation button B4, the video is displayed at a progression speed selected from multiple preset progression speeds. The progression speed is set as a magnification of the detected speed, with 1 being the same as the speed detected by the detection unit 11 (hereinafter referred to as the detected speed). In Fig. 2, "x2.00" is displayed near the operation button B4 to indicate a progression speed that is 2.00 times the detected speed, and in this case, the moving image is displayed at a progression speed of 2.00 times the detected speed.

[0028] The moving image portion Ga preferably shows the center of gravity position and its trajectory D1 together with the three-dimensional human body model MA, and this is also the case in this example. The display of the moving image center of gravity position and its trajectory D1 can be switched on and off by, for example, touching the operation button B5 that switches the display on and off.

[0029] The video unit Ga preferably includes a floor reaction force display D2 showing the floor reaction force along with the three-dimensional human body model MA. It is more preferable that the floor reaction force display be displayed as a floor reaction force vector. The starting point of the vector indicates the center of foot pressure, which is the origin of the floor reaction force, its length indicates the magnitude of the floor reaction force, and its direction indicates the direction in which the floor reaction force acts. This allows the client to more easily understand, for example, how weight is distributed left and right and front and back on the body. Furthermore, depending on where the floor reaction force vector passes on the three-dimensional human body model MA, it becomes easy to identify the part of the body that is under load and the posture as the timing of the load, and it also becomes easy to analyze the location where joint torque is generated and the magnitude of the joint torque. Furthermore, the position of the starting point of the floor reaction force vector makes it easy to understand the center of foot pressure. In this way, the floor reaction force display makes analysis easier for the client. The floor reaction force display can be switched on and off by, for example, touching the operation button B6 that switches the display on and off.

[0030] The moving image unit Ga preferably displays a contact position marker D3 on the floor surface FS, indicating the contact position between the foot and the floor surface. The shape of the contact position marker D3 is not particularly limited, and in this example, the contact position marker D3 is a line in a moving image in which the floor surface FS is viewed from the side as shown in FIG. 2, and a foot shape in a moving image in which the floor surface FS is viewed from above or below. In this example, the contact position marker D3 is displayed at all of the determined contact positions so that the chronological history can be grasped, but only some of them may be displayed. The display of the contact position marker D3 can be switched on and off by, for example, a touch operation on the operation button B7 that switches the display on and off.

[0031] The moving image unit Ga preferably includes an operation button B8 for changing the orientation of the displayed three-dimensional human body model MA. In this example, the orientation of the three-dimensional human body model MA can be changed by a swipe operation in which the client touches and moves their finger on the operation button B8.

[0032] In this example, only the three-dimensional human body model MA whose position and posture correspond to the progression of the video is shown in the video section Ga (depicted by a solid line), and the three-dimensional human body models MA before and after it are not displayed, as shown by the imaginary two-dot chain lines in Figure 2. That is, in this example, a single three-dimensional human body model MA is displayed as the video progresses. However, three-dimensional human body models MA before and after it may also be displayed in the video section Ga in addition to the three-dimensional human body model MA whose position and posture correspond to the progression, so that they can be distinguished from the three-dimensional human body model MA. In this way, the client terminal 15 displays the analysis results of the subject's body movements using video including the three-dimensional human body model MA.

[0033] The three-dimensional human body model MA in this example is composed of skeletal structure information indicating the skeleton; mesh part information indicating mesh parts that schematically show the shapes of muscles, skin, bones, etc.; texture information indicating texture (feel) using color (hue, thickness, brightness), line type, line thickness, etc.; and link information associating the skeletal structure parts with the mesh parts. The three-dimensional human body model MA is a rigid link model with the pelvis as the root joint. Note that the three-dimensional human body model MA displayed in the animation in this example is a schematic representation of muscles, skin, etc., but in Figure 2, to avoid cluttering the illustration, only the outline of the body surface is shown. The link information is information that associates which mesh parts move in response to the movement of any joint in the skeletal structure.

[0034] The graph image section Gb displays a graph DG of a selected analysis result from among the analysis results obtained based on the detection results detected by the detection unit 11. In the example shown in FIG. 2 , one graph DG is displayed, but multiple analysis results may be selectable, and the graphs of the selected analysis results may be displayed, for example, vertically aligned. The graph image section Gb is provided with a "waveform selection" button as an operation button B11 for selecting the analysis result to be displayed as a graph. In response to a touch operation on this operation button B11, a list of analysis results to be selected is displayed in a pull-down format on the operation button B11. The client selects at least one analysis result from this list by a touch operation, and the graph DG of the selected analysis result is displayed in the graph image section Gb.

[0035] The analysis results are not particularly limited, and in this example, they are as follows. All are angles (unit: °): 1. Pelvis (1) Pelvis Anterior Tilt (2) Pelvis Right List (3) Pelvis Left Rotation 2. Hip (1) L (Left) - Hip Extension (Left hip extension) (2) L - Hip Abduction (Left hip abduction) (3) L - Hip External Rotation (Left hip external rotation) (4) R (Right) - Hip Extension (Right hip extension) (5) R - Hip Adduction (Right hip adduction) (6) R - Hip Internal Rotation (Right hip internal rotation) 3. Spine (1) Spine1 Flexion (2) Spine1 Right Bending (3) Spine1 Left Rotation (4) Spine2 Flexion (5) Spine2 Right Bending (6) Spine2 Left Rotation 4. Knee (knee joint) (1) L-Knee Flexion (left knee joint flexion) (2) L-Knee Abduction (left knee joint abduction) (3) L-Knee External Rotation (left knee joint external rotation) (4) R-Knee Flexion (right knee joint flexion) (5) R-Knee Adduction (right knee joint adduction) (6) R-Knee Internal Rotation (right knee joint internal rotation) 5. 5. Ankle (1) L-Ankle Plantarflexion (left ankle dorsiflexion) (2) L-Ankle Pronation (left ankle pronation) (3) L-Ankle Abduction (left ankle abduction) (4) R-Ankle Plantarflexion (right ankle dorsiflexion) (5) R-Ankle Supination (right ankle supination) (6) R-Ankle Adduction (right ankle adduction) 6. Neck (1) Neck Flexion (neck flexion) (2) Neck Right Bending (right neck flexion)(3) Neck Left Rotation 7. Head (1) Head Flexion (2) Head Right Bending (3) Head Left Rotation 8. Shoulder (shoulder joint) (1) L-Shoulder Internal Rotation (left shoulder joint adduction) (2) L-Shoulder Abduction (left shoulder joint abduction) (3) L-Shoulder Extension (left shoulder joint extension) (4) R-Shoulder Internal Rotation (right shoulder joint adduction) (5) R-Shoulder Adduction (right shoulder joint adduction) (6) R-Shoulder Flexion (right shoulder joint extension) 9. Elbow (elbow joint) (1) L-Elbow Pronation (left elbow joint rotation) (2) L-Elbow Abduction (left elbow joint abduction) (3) L-Elbow Extension (left elbow joint extension) (4) R-Elbow Pronation (right elbow joint rotation) (5) R-Elbow Adduction (right elbow joint adduction) (6) R-Elbow Flexion (right elbow joint flexion) 10. Wrist (1) L-Wrist Pronation (left wrist joint rotation) (2) L-Wrist Extension (left wrist joint extension) (3) L-Wrist Ulnar Deviation (left wrist joint ulnar flexion) (4) R-Wrist Pronation (right wrist joint rotation) (5) R-Wrist Flexion (right wrist joint extension) (6) R-Wrist Radial Deviation (right wrist joint radial flexion)

[0036] In the graph DG, the vertical axis represents the analysis element showing the analysis result, and the horizontal axis represents time, with the analysis result over time being shown by a continuous curve D11. When the moving image displayed in the moving image section Ga is the entire video, the graph DG shows the analysis result from the start to the end of walking detection. A reference curve D12 showing a reference value may be displayed together with the analysis result. The reference value may be, for example, a standard value known to the public in a paper or the like, i.e., an average value, or may be a past analysis result for the subject. The display of the reference curve D12 can be switched on and off by, for example, touching the operation button B12 that switches the display on and off.

[0037] The graph DG has a progress marker D13 that indicates the progress of the video. The progress marker D13 moves to the right as the video progresses. As described above, when the temporal position of the video is changed by operating the tip B3, the position of the progress marker D13 in the horizontal direction also changes. In this way, the position of the progress marker D13 is linked to the progress of the video. This allows the client to compare the body and each body part shown in the video with the corresponding analysis results on the graph, making analysis easier. Note that the value of the analysis element corresponding to the position of the progress marker D13 may be displayed within or near the graph DG. For example, as shown in FIG. 2, if the analysis element is the "joint angle" of the "Right Knee," the value of the right knee joint angle may be displayed changing with the displacement of the progress marker D13.

[0038] In this example, furthermore, by moving the position of the progress marker D13 within the graph DG, the temporal position of the moving image can be changed at the time point designated as the destination, i.e., the moving image can be changed to the moving image at the time point corresponding to the progress marker D13. By positioning the progress marker D13 at a time point of interest within the graph, the position and posture of the body and each part of the body at that time point can be grasped, making analysis easier. The designation operation for designating the destination can be performed, for example, by a touch operation on any position within the graph DG, and the touched position can be designated as the destination.

[0039] When the video image in the video image section Ga is switched to the "right walking cycle" using the switching operation section B1, the video image section Ga displays a right walking cycle video as a video image, as shown in FIG. 3A . The right walking cycle video is a video image from the start of the representative walking cycle (initial foot contact, described below) to its end (the next initial foot contact of the same leg). In this case, the progress bar BP displays only the range from the start to the end of the representative walking cycle and includes a walking phase display D14. The walking phase display D14 indicates the walking phase identified in the representative walking cycle of the right leg, as described below, and is displayed in temporal association with the progress bar BP. The same applies when the video image is switched to the "left walking cycle." When the video image is switched to the "full video," the progress bar BP may include a walking phase display D14 for each of the left and right legs, allowing the walking phases on both sides to be individually confirmed. This allows the client to compare the posture of the three-dimensional human body model MA shown in the video image with the walking phase, making analysis easier.

[0040] When the moving image in the moving image section Ga is switched to "right walking cycle" using the switching operation section B1, the graph image section Gb displays a graph DG showing the analysis results for the representative walking cycle of the right leg, as shown in FIG. 3B . In this example, the left end of the horizontal axis represents the start of the representative walking cycle of the right leg, and the right end represents the end. In this way, the graph DG shown in the graph image section Gb switches display in conjunction with the moving image shown in the moving image section Ga. The same is true when the moving image is switched to "left walking cycle." This makes it easier to analyze the representative walking cycles of the right and left legs, and further allows for analysis by walking phase. Furthermore, since the moving image and graph DG are displayed by switching between the right walking cycle and the left walking cycle, it is easy to compare the right walking cycle and the left walking cycle, making analysis easier.

[0041] In FIG. 4 , the detection unit 11 calculates the three-dimensional joint angles (hereinafter referred to as three-dimensional joint angles), three-dimensional joint positions (hereinafter referred to as three-dimensional joint positions), joint torque (joint load), muscle activity, and floor reaction force as the aforementioned motion information. In this example, the center of foot pressure position is also calculated as motion information. As such, the motion information may be determined appropriately depending on the analysis elements displayed as analysis results in image G ( FIG. 2 ). The detection unit 11 may be configured with multiple detectors that calculate some of this motion information. The three-dimensional joint angles and three-dimensional joint positions can be calculated using, for example, a device that calculates them by performing inverse kinematics based on marker data obtained by optical motion capture, an inertial sensor, or the like, or a device that calculates them by three-dimensional posture estimation using AI. As such, the detection unit 11 does not need to directly detect motion information such as three-dimensional joint angles and three-dimensional joint positions, but may instead calculate the motion information based on the detected elements.

[0042] The three-dimensional joint angles may be expressed using any of a rotation vector, Euler angles, a rotation matrix, a quaternion, etc. Three-dimensional information such as three-dimensional joint angles and three-dimensional joint positions has the origin at (x, y, z) = (0, 0, 0) in three-dimensional space.

[0043] In this example, the three-dimensional joint angles and three-dimensional joint positions are expressed in a generalized coordinate system. The generalized coordinate system is a coordinate system capable of expressing three-dimensional information using variables corresponding to the degrees of freedom of a model in which a subject is modeled as having rigid segments and joints (articulations) connecting the segments. The generalized coordinate system in this embodiment is similar to the generalized coordinate system described in Japanese Patent Application Laid-Open No. 2020-201138, for example, in which the head, chest, abdomen, pelvis, left and right thighs, left and right shins, and left and right feet are each defined as segments SG, and joints JT(i) connecting the segments are defined (where i = 1 to N). Of the segments SG, the segment SG(B) corresponding to the pelvis is defined as the base segment SG(B). The six-degree-of-freedom displacement of segments SG other than the base segment SG(B) relative to the origin of the absolute coordinate system is not a variable in the generalized coordinate system, and only the base segment SG(B) is treated as displaceable with six degrees of freedom. The six degrees of freedom include three translational directions (XYZ) and three rotational directions (yaw, roll, and pitch). In this example, the three translational directions are time-series data of three-dimensional joint positions of identification information (ID) associated with the pelvis, and the three rotational directions are joint rotation vectors as joint angles of the ID associated with the pelvis.

[0044] The detection unit 11 in this example includes a subject skeletal model generation unit (not shown), which generates a subject skeletal model by applying at least the subject's height to the built-in general skeletal model. The general skeletal model is expressed in the generalized coordinate system, and segments and joints, the center of gravity of each segment (hereinafter referred to as the segment center of gravity), and the mass ratio of each segment for a certain height and weight are set. When at least the subject's height is input, the subject skeletal model generation unit performs scaling transformation based on the input data to generate a subject skeletal model tailored to the subject's skeleton. As a result, three-dimensional joint angles, three-dimensional joint positions, the center of gravity of each segment, and the mass ratio of each segment are calculated based on the subject skeletal model, for example, by performing inverse kinematics using marker data obtained by optical motion capture. However, the detection unit 11 may calculate the three-dimensional joint angles and three-dimensional joint positions using other methods. The subject's height can be input to the detection unit 11 from the client terminal 15, for example, by operating the client terminal 15. The mass proportions of the segments in the subject skeletal model may be the same as those in the general skeletal model. When the segment center of gravity positions and the segment mass proportions are determined by the detection unit 11 as in this example, the detection unit 11 sends the segment center of gravity positions and the segment mass proportions in addition to the three-dimensional joint angles and three-dimensional joint positions to the position calculation unit 58. The segment center of gravity positions and the segment mass proportions may be determined by the position calculation unit 58.

[0045] The skeletal structure of the general skeletal model and the three-dimensional human body model MA is the same, meaning that the number, configuration, and degrees of freedom of the joints are the same, and the positions (relative positions) of the joints on the body are also the same.

[0046] The floor reaction force and the center of foot pressure can be obtained, for example, using a commercially available force plate, which can be used as a detection section constituting the detection unit 11. The force plate includes, for example, a plate and four load cells arranged below the plate, and measures the force generated when the subject moves on the plate, making it possible to determine the center of foot pressure.

[0047] Instead of the detection unit 11, an estimation device that estimates the floor reaction force and the center of foot pressure in addition to estimating the three-dimensional posture using AI may be used as the detection device.

[0048] The joint torque is the torque of each of the multiple joints connecting each of the multiple segments, and is generated by the muscle tension of the corresponding multiple muscle models. The joint torque can be calculated by performing dynamics analysis based on the position, velocity, and acceleration dimensional information and the subject's skeletal model.

[0049] Muscle activity is an alternative expression of the action potential, which is a signal from the central nervous system to the muscle, or the calcium ion concentration that increases in the muscle due to the input of the action potential, and is related to the strength of muscle contraction. Muscle activity is calculated based on three-dimensional information such as three-dimensional joint position and three-dimensional joint angle, and joint torque.

[0050] The detection unit 11 is connected to the analysis support device 13 as an external device of the analysis support device 13 , but may also be a part of the analysis support device 13 .

[0051] The analysis support device 13 includes a video storage unit 41 and a display control unit 42. The analysis support device 13 preferably further includes a three-dimensional human body model storage unit 52, an acquisition unit 56, a position calculation unit 58, a gait variable calculation unit 61, a gait period extraction unit 63, a resampling unit 64, and a reference storage unit 65, and this is also the case in this example. The video storage unit 41 stores video of the three-dimensional human body model MA displayed as a subject engaged in exercise. As described above, when video captured with the subject as the subject is displayed in the video unit Ga, the video storage unit 41 stores video captured with the subject as the subject instead of video including the three-dimensional human body model MA.

[0052] The three-dimensional human body model storage unit 52 stores the three-dimensional human body model MA (see FIG. 2A ) displayed in the moving image. Therefore, as described above, when a moving image captured with the subject as the subject is displayed in the moving image unit Ga, the analysis support device 13 does not need to include the three-dimensional human body model storage unit 52.

[0053] The acquisition unit 56 acquires motion information about the subject during exercise, i.e., time-series data of three-dimensional joint angles, three-dimensional joint positions, floor reaction forces, joint torques, muscle activities, and foot pressure center positions, from the detection unit 11. The acquisition unit 56 in this example further acquires the center of gravity position of each segment and the mass proportion of each segment from the detection unit 11. The acquisition unit 56 sends the acquired time-series data of the motion information, the center of gravity position of each segment, and the mass proportion of each segment to the position calculation unit 58.

[0054] The position calculation unit 58 calculates time-series data of the subject's center of gravity position based on the time-series data of the three-dimensional joint positions among the motion information input from the acquisition unit 56. The position calculation unit 58 calculates a multiplication value by multiplying the segment center of gravity position by the mass fraction of the segment for each segment of the skeletal model described above when the detection unit 11 calculated the three-dimensional joint positions, based on the segment center of gravity positions and mass fractions of the segment input from the acquisition unit 56. The position calculation unit 58 then calculates the center of gravity position of the entire body by summing up the multiplication values ​​calculated for all segments.

[0055] The position calculation unit 58 further calculates the contact position of the foot on the floor surface FS (see FIG. 2) in chronological order based on the time-series data of the three-dimensional joint positions input from the acquisition unit 56. In this example, when the position of at least one of the multiple markers associated with the heel, toe, etc. and set as being in contact with the floor surface FS is within a range of 0.05 m from the floor surface FS and the speed of the marker is 0.8 m / s or less, the marker is defined as being in contact with the floor surface FS, and the contact position is calculated.

[0056] The position calculation unit 58 outputs motion information, which is obtained by adding time series data on the contact position and center of gravity position to time series data on the three-dimensional joint position, joint angle, floor reaction force, joint torque, muscle activity, and foot pressure center position, to the display control unit 42, the gait variable calculation unit 61, and the gait cycle extraction unit 63. Note that if the motion information acquired by the acquisition unit 56 from the detection unit 11 includes the contact position and center of gravity position, it is sufficient for the position calculation unit 58 to output the information only to the gait variable calculation unit and the gait cycle extraction unit 63, and it is not necessary to output the information to the display control unit 42.

[0057] When motion information, which is time-series data, is input, the gait variable calculation unit 61 calculates gait variables in response to these inputs and outputs them to the display control unit 42. In this example, the gait variables are the maximum value of each joint angle, the movement distance of the center of gravity, and the joint angle in one gait cycle. The movement distance of the center of gravity can be at least one of the maximum movement distance of the subject P in the left-right and up-down directions and the total movement distance (length of the trajectory) in the left-right and forward-backward directions for each gait cycle.

[0058] The maximum value of the joint angle is determined by comparing a plurality of joint angles constituting the time-series data and finding the largest value.

[0059] The position of the center of gravity is calculated from the position of the center of gravity and the mass fraction of each segment included in the skeletal model. The mass fraction of a segment can be obtained from a known mass fraction table (described in the following non-patent document 3: Winter, DA (2009). Biomechanics and motor control of human movement (4th ed.). (USA) Wiley & Sons, Inc.). The position of the center of gravity of the entire body is then calculated by multiplying the position of the center of gravity of each segment by the mass fraction of the segment to obtain a product, and then adding up the product values ​​obtained for all segments to obtain the position of the center of gravity of the entire body.

[0060] The walking cycle extraction unit 63 is an example of an extraction unit that extracts any repetitive unit of movement as a unit movement when the movement is a repetitive movement such as walking. However, the movement does not have to be composed of multiple unit movements, and the extraction unit may extract a movement of a specific duration from the movement to be analyzed as a unit movement.

[0061] When the movement to be analyzed is walking, the gait cycle extraction unit 63 calculates the gait cycles of the right and left legs within the analysis time T from the start of detection to the end of detection, then extracts any one gait cycle for each leg and outputs the extracted gait cycle to the resampling unit 64 as the representative gait cycle for that leg. A gait cycle, which is an example of a unit movement, is defined as the time from when the heel of one foot touches the ground to when the heel of the same foot next touches the ground, with 100% representing the time. For example, within the analysis time T, the heel contact times of the right foot are, in order, i.e., over time, ta1, ta2, ta3, ta4, and ta5, and the heel contact times of the left foot are, in order, tb1 (where tb1 is after ta1 and before ta2), tb2, tb3, and tb4. In this case, the time from ta1 to ta2, the time from ta2 to ta3, the time from ta3 to ta4, the time from ta4 to ta5, the time from tb1 to tb2, the time from tb2 to tb3, and the time from tb3 to tb4 are each set to 100%.

[0062] The gait cycle extraction unit 63 identifies and extracts any one gait cycle from all gait cycles within the analysis time T for each of the right leg and the left leg, and outputs the extracted gait cycle as a representative gait cycle to the resampling unit 64. In this extraction process, it is preferable to extract the gait cycle located at the most center on the time axis of the analysis time T as the representative gait cycle.

[0063] The gait cycle extraction unit 63 may divide the unit movement into smaller movement phases over time. In this example, where the unit movement is the gait cycle, the gait phase is the movement phase. The gait phase may be based on any known definition. In this example, the eight gait phases are defined in accordance with J. Perry's definition, which is said to be the most common in the field of physical therapy, making analysis easier for physical therapists and sports trainers. In J. Perry's definition, only the initial contact indicates a single point, and the others are indicated by specific times, as follows: 1. Initial Contact (hereinafter referred to as IC): When the heel of one foot first touches the ground 2. Loading Response (hereinafter referred to as LR): From IC to when the other leg leaves the floor (ground) 3. 3. Mid Stance (hereinafter referred to as MSt): From when the other leg leaves the ground to when the one leg leaves the floor 4. Terminal Stance (hereinafter referred to as TSt): From when the heel of the one foot leaves the floor to when the heel of the other foot touches the ground (IC) 5. Pre Swing (hereinafter referred to as PSw): From when the heel of the other foot touches the ground (IC) to when the toe of the one foot leaves the floor 6. Initial Swing (hereinafter referred to as ISw): From when the toe of the one foot leaves the floor to when both lower legs intersect in the sagittal plane (when the midpoints of the knee joints and ankle joints on both sides intersect in the direction of travel) 7. Mid swing (MSw): From when both lower legs cross in the sagittal plane to when one of the lower legs becomes perpendicular to the floor. 8. Terminal swing (TSw): From when the lower leg of one leg becomes perpendicular to the floor to when the heel of the next ipsilateral foot touches the ground (IC).

[0064] In this example, gait phases are identified using three-dimensional information such as three-dimensional joint positions and three-dimensional joint angles. Therefore, gait phases can be obtained more reliably than when gait phases are identified from, for example, images of the subject captured in a plane parallel to the sagittal plane, i.e., images of the subject captured from the side. Furthermore, when gait phases are identified from images of the subject captured from the side, the gait phase of the imaged front leg can be identified, but the gait phase of the imaged rear leg cannot be identified. In contrast, this example uses the above three-dimensional information, so gait phases can be identified for both the left and right legs, and simultaneously. As a result, analysis can be performed quickly and easily.

[0065] The gait phase can be identified as follows using the three-dimensional joint positions, the heel and toe contact positions and timing, etc. The above eight gait phases IC, LR, MSt, TSt, and PSw can be uniquely determined from the heel and toe contact timing. To define the sagittal plane direction required to identify ISw, MSw, and TSw, the direction of gait is defined as the X direction in three-dimensional space, the vertical upward direction as the Z direction, and the XZ plane as the sagittal plane. In this case, if the three-dimensional joint positions are not on the same coordinate system, the pelvic positions at any two points in time (P1 and P2, with P2 being the pelvic position at the later point in time) can be determined from the time-series data of the pelvic position, and the X vector VX can be obtained using the formula VX = P2 - P1. The three-dimensional joint position data can then be rotated by creating a transformation matrix. To create the transformation matrix, the Y vector is calculated from the cross product of the Z vector (vertically upward) and the X vector. The transformation matrix can then be created by using the X vector, Y vector, and Z vector as elements of the transformation matrix. The position of the lower leg is obtained by determining the midpoint between the knee joint position and the ankle joint position from the three-dimensional joint positions. When both lower legs, required for defining ISw, intersect in the sagittal plane, this can be defined as the time when the X component of the position of one of the lower legs exceeds the X component of the position of the other lower leg. When the lower leg, required for defining MSw, becomes perpendicular to the floor, this can be defined as the time when the X component of the position of one of the knee joints exceeds the X component of the position of one of the ankle joints. In the circular gait (walking with one leg swinging outward) seen in stroke patients, the lower leg is not necessarily perpendicular to the floor, but the above definition makes it possible to uniquely determine it. TSw can be uniquely determined from the time when the lower leg becomes perpendicular to the floor and the timing of the heel contact.

[0066] The walking cycle extraction unit 63 sends the representative walking cycle of each leg, each time series data of the motion information in the representative walking cycle, and the walking phase obtained for the representative walking cycle to the resampling unit 64.

[0067] The resampling unit 64 resamples each time-series data of the movement information and ground contact positions for each representative gait cycle of both legs input from the gait cycle extraction unit 63, converting it into time-series data consisting of a predetermined number of pieces. The number of pieces of information constituting the time-series data is not particularly limited, and in this example, it is 101. That is, for each representative gait cycle of both legs, all time-series data input from the gait cycle extraction unit 63 is converted into time-series data consisting of 101 pieces of information, such as 101 chronologically ordered three-dimensional joint positions, 101 chronologically ordered foot pressure center positions, and 101 chronologically ordered three-dimensional joint angles. The obtained time-series data is sent to the display control unit 42. Note that the time intervals for all 101 pieces of chronologically ordered information are all equal. As described above, the resampling process ensures that all pieces of movement information and ground contact positions have the same number of pieces of information. Therefore, subsequent processing is performed by comparing the same number of pieces of information, which allows for efficient processing and calculation, rapid display of analysis results, and easy analysis.

[0068] When the display control unit 42 receives time-series data of movement information from the position calculation unit 58 and gait variables from the gait variable calculation unit 61, it generates, based on these data, motion data as a moving image showing the movement of the three-dimensional human body model MA (see FIG. 2) corresponding to the analysis time T, and also generates analysis data of at least one of the floor reaction force, the center of gravity position, and the ground contact marker shown together with the three-dimensional human body model MA in the moving image. The display control unit 42 also generates, based on each time-series data, plot data that forms the basis of the curves displayed in each graph DG (see FIGS. 2 and 3B) for the analysis time T, and then generates a graph DG having a curve D11 (see FIGS. 2 and 3B) based on this plot data.

[0069] When the display control unit 42 receives time-series data of movement information for each representative walking cycle of the right leg and the left leg from the resampling unit 64, the display control unit 42 generates motion data as unit moving images showing the movement of the three-dimensional human body model MA (see FIG. 2 ) corresponding to the representative walking cycle based on the time-series data. The display control unit 42 further generates plot data for each representative walking cycle and generates a graph DG as a unit graph having a curve for each representative walking cycle based on the plot data.

[0070] The display control unit 42 sends the generated motion data, analysis data, curve data, etc. as image data to the client terminal 15 and displays it on the display unit 15 a. Prior to sending the motion data, analysis data, etc., the display control unit 42 also sends the three-dimensional human body model MA to the client terminal 15 and stores it in the three-dimensional human body model storage unit 15 c of the client terminal 15.

[0071] In this example, in response to a selective display request operation for selecting and requesting display of one of the overall video, right walking cycle video, and left walking cycle video, i.e., an operation on the switching operation unit B1 (see FIG. 2), analysis data, curve data, etc. for the selected video are transmitted to and displayed on the client terminal 15. However, the data for the overall video, right walking cycle video, and left walking cycle video may be transmitted together to the client terminal 15, and the selected video may be displayed on the client terminal 15 in response to the selective display request operation.

[0072] The client terminal 15 includes a display unit 15a, an input unit 15b, a three-dimensional human body model storage unit 15c that stores the acquired three-dimensional human body model MA, and a control unit 15d. The input unit 15b is used by the client to perform various input operations, such as a login operation, input operations for various buttons such as the pause button B1 (see FIG. 2) described above, and a login operation when starting use.

[0073] The three-dimensional human body model storage unit 15c of the client terminal 15 stores the three-dimensional human body model MA transmitted from the display control unit 42 of the analysis support device 13. It is preferable that the three-dimensional human body model MA be stored in the three-dimensional human body model storage unit 15c before motion data, analysis data, and the like are transmitted from the display control unit 42. For example, if the client terminal 15 operates on a browser as described above, the display control unit 42 may transmit the three-dimensional human body model MA to the client terminal 15 in response to a login operation and store it therein, or if the client terminal 15 operates by executing a predetermined application software program as described above, the three-dimensional human body model MA may be stored when the software is installed.

[0074] The control unit 15d comprehensively controls the display unit 15a, input unit 15b, three-dimensional human body model storage unit 15c, and other units of the client terminal 15. The control unit 15d displays the image G on the display unit 15a based on various data transmitted from the display control unit 42 of the analysis support device 13.

[0075] The operation of the above configuration will be described with reference to Fig. 5. When a client terminal 15 logs in, the client terminal 15 determines whether a three-dimensional human body model MA is already stored in the client terminal 15 (S1). If the three-dimensional human body model MA is not stored, the client terminal 15 sends a transmission request for the three-dimensional human body model MA to the analysis support device 13. When the display control unit 42 of the analysis support device 13 receives the transmission request from the client terminal 15, it transmits the three-dimensional human body model MA to the client terminal 15 in response to the transmission request (S2). When the three-dimensional human body model MA is input, the client terminal 15 stores the three-dimensional human body model MA in the three-dimensional human body model storage unit 15c under the control of the control unit 15d. If the client terminal 15 determines that the three-dimensional human body model MA is already stored, it sends an analysis result request requesting analysis results to the analysis support device 13. If the analysis result has not yet been obtained, the analysis support device 13 sends an analysis result request containing an analysis instruction for requesting the analysis result to the detection unit 11, and the detection unit 11 responds to this analysis result request to request the analysis result and transmits the obtained analysis result to the analysis support device 13. However, as mentioned above, the processing flow up to the storage of the three-dimensional human body model MA in the client terminal 15 is not limited to this example.

[0076] When the analysis support device 13 receives the analysis result request, the acquisition unit 56 acquires, as analysis results, time-series data of movement information such as three-dimensional joint positions, three-dimensional joint angles, floor reaction forces, and foot pressure center positions from the detection unit 11 (S3). In response to the acquisition of the time-series data of the three-dimensional joint positions and foot pressure center positions, the acquisition unit 56 outputs the time-series data to the position calculation unit 58.

[0077] In response to the input of the time-series data of the motion information, the position calculation unit 58 generates time-series data of the center of gravity position (S4a), and in this example, further calculates the trajectory of the center of gravity position based on this time-series data. In response to the input of the time-series data of the three-dimensional joint positions, the position calculation unit 58 identifies the timing of the contact points and calculates the contact points at each of these timings as coordinates (S4b). In the position calculation step S4 performed by the position calculation unit 58, either the first step S4a of calculating the center of gravity position or the second step S4b of calculating the contact points may precede the step S4. The position calculation unit 58 outputs the calculated time-series data of the center of gravity position, the trajectory of the center of gravity position, and the time-series data of the motion information including the contact points to the display control unit 42, the gait variable calculation unit 61, and the gait period extraction unit 63, respectively.

[0078] The gait variable calculation unit 61 calculates the gait variables and outputs them to the display control unit 42 (S5).

[0079] After extracting the walking cycle of the right leg and the walking cycle of the left leg, the walking cycle extraction unit 63 obtains a representative walking cycle and outputs it to the resampling unit 64 together with the time series data of the acquired movement information (S6).

[0080] The resampling unit 64 performs a resampling process on the time series data of each piece of acquired motion information, and sends the resampled time series data to the display control unit 42 (S7).

[0081] When the display control unit 42 receives the time-series data of the movement information from the position calculation unit 58 and the gait variables from the gait variable calculation unit 61, it generates a moving image corresponding to the analysis time T (S8a) and generates a graph (S8b) in response to these inputs. Either step S8a or step S8b may be performed first. When the display control unit 42 receives the time-series data from the resampling unit 64 and the gait variables from the gait variable calculation unit 61, it similarly receives the time-series data from the resampling unit 64 and generates a unit moving image corresponding to each representative gait cycle of the right leg and the left leg (S8c) and generates a unit graph (S8d) in response to these inputs. Either step S8c or step S8d may be performed first. The display control unit 42 sends the generated data as image data to the client terminal 15 together with the reference curve stored in the reference storage unit 65 (S9).

[0082] The client terminal 15 performs a process of merging the video data with the pre-stored three-dimensional human body model MA, and displays an image G including a video image portion Ga and a graph image portion Gb on the display unit 15a (S10).

[0083] 10 Analysis support system 11 Detection unit 13 Analysis support device 41 Video image storage unit 42 Display control unit 52 Three-dimensional human body model storage unit 63 Gait cycle extraction unit 64 Resampling unit

Claims

1. An analysis support device comprising: a video storage unit that stores video images of a subject whose movement is to be analyzed, or video images of a three-dimensional human body model displayed as the subject; and a display control unit that generates a graph showing the analysis results over time based on the analysis results of the subject's movement, and simultaneously displays the video and the graph having a progress marker that shows the progress of the video.

2. The analysis support device according to claim 1, further comprising a three-dimensional human body model storage unit that stores the three-dimensional human body model, wherein the display control unit generates a moving image of the three-dimensional human body model based on the analysis results, and the moving image storage unit stores the moving image of the three-dimensional human body model.

3. The analysis support device of claim 1 or 2, wherein the movement includes repeated actions, and further comprises an extraction unit that extracts any repeated unit of action as a unit movement, and the display control unit generates a unit moving image and a unit graph that are the moving image and graph of the unit movement, and simultaneously displays the unit moving image and the unit graph having the progress marker.

4. The analysis support device according to claim 3, wherein the movement is walking and the unit movement is a walking cycle.

5. The analysis support device of claim 3, wherein the extraction unit divides the unit movement into smaller movement phases over time, and the display control unit displays the movement phases in association with a progress bar extending in one direction that indicates the progress of the unit motion image.

6. The analysis support device according to claim 5, wherein the movement is walking, the unit movement is a walking cycle, and the movement phase is a walking phase.

7. The analysis support device according to claim 5, wherein the display control unit generates and displays the unit graph in which the gait cycle of the selected leg is the unit movement in response to a selective display request requesting display of either the right leg or the left leg in a selected state.

8. The analysis support device according to claim 3, wherein the display control unit displays the selected one of the moving image and the unit moving image in response to a selective display request requesting display in a selected state of either the moving image or the unit moving image.

9. An analysis support method comprising: a video storage step of storing video images of a subject whose movement is to be analyzed, or video images of a three-dimensional human body model displayed as the subject; and a display control step of generating a graph showing the analysis results over time based on the analysis results of the subject's movement, and simultaneously displaying the video and the graph having a progress marker that shows the progress of the video.

10. An analysis support program that causes a computer to execute: a video storage step for storing video images of a subject whose movement is to be analyzed, or video images of a three-dimensional human body model displayed as the subject; and a display control step for generating a graph showing the analysis results over time based on the analysis results of the subject's movement, and simultaneously displaying the video and the graph having a progress marker that shows the progress of the video.

Images