Motion analysis system

Abstract

To provide a technology capable of accurately analyzing a walking motion by suppressing the influence of noise even when the walking motion of a subject is captured by using an imaging device.SOLUTION: A motion analysis system according to the present invention determines the reliability of skeleton coordinates in each motion cycle of a subject, selects any motion cycle on the basis of the reliability, and analyzes a motion action using the motion cycle.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a technique for analyzing the walking motion of a subject. [Background technology]

[0002] It is known that motor dysfunction in elderly people is most evident in walking. Walking dysfunction is diagnosed by a specialist such as a doctor observing the subject's walking. On the other hand, with the advancement of various sensing technologies, technologies that perform mechanical diagnosis instead of such expert diagnosis have also been developed.

[0003] Patent Document 1 aims to "provide an estimation device, estimation method, and estimation program for accurately estimating whether a subject is frail or has mild cognitive impairment without the need for bothersome work," and describes a technology in which "estimation device 100 includes a position acquisition unit 133 that acquires position data of a body part including the subject's toes in chronological order, an extraction unit 134 that extracts features based on the position data acquired in chronological order, and a first estimation unit 135 that inputs the features into a learning model and estimates from the learning model that the subject is frail or has mild cognitive impairment" (see abstract).

[0004] Patent Document 2 describes a technology that aims to "track pedestrians with high accuracy" and "constructs an extended DeepSORT that adds similarity based on the walking state of the person to be tracked, thereby performing robust tracking of multiple people using walking state estimation. Since the walking state is easy to use for matching because the change in the posture of the person between frame images is small, it is possible to predict the posture at that time even if observation is impossible for a certain period of time due to occlusion, etc. Since the walking state has different properties from the rectangular motion information and appearance information, it is suitable as new information to supplement the rectangular motion information and appearance information. In addition, since the walking state is extracted from skeletal information based on the pedestrian's joints, the accuracy of the walking state estimation may decrease due to non-detection of necessary joints, etc., an index based on the proportion of undetected joints is set. The validity of the walking state is judged by the index, and the possibility of pedestrian matching is judged based on this (see abstract).

[0005] Patent Document 3 describes a technology in which "the measuring device has a first acquiring means for acquiring depth information indicating the depth for each two-dimensional position, and a measuring means for measuring the walking state of the subject using the depth information" (see abstract). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2022-092940 [Patent Document 2] Patent Publication No. 2022-152202 [Patent Document 3] WO2016 / 208289 Summary of the Invention [Problem to be solved by the invention]

[0007] In the conventional techniques such as Patent Documents 1 to 3, the walking motion of a subject is captured using a sensing device such as a camera, and the subject's skeletal coordinates are detected from the captured image, thereby analyzing the walking motion of the subject. However, the captured image of the walking motion contains a lot of noise, which reduces the accuracy of the gait analysis. In Patent Documents 1 to 3, it is considered that there is room for further consideration on how to process or remove such noise.

[0008] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a technology that can suppress the effects of noise and accurately analyze a subject's walking movement, even when the walking movement of the subject is captured using an imaging device. [Means for solving the problem]

[0009] The motion analysis system according to the present invention determines the reliability of skeletal coordinates in each motion cycle of a subject, selects one of the motion cycles based on the reliability, and analyzes the motion using that motion cycle. Effect of the Invention

[0010] According to the motion analysis system of the present invention, even when the walking motion of the subject is photographed using an imaging device, the influence of noise can be suppressed and the walking motion can be analyzed with high accuracy. Problems, configurations, effects, etc. other than those described above will be made clear through the description of the following embodiments. [Brief description of the drawings]

[0011] [Figure 1] 1 is a configuration diagram of a motion analysis system 1 according to a first embodiment. [Diagram 2] FIG. 11 is a diagram illustrating a method for determining reliability for each walking cycle. [Diagram 3] 4 is a flowchart illustrating an operation procedure of the motion analysis system 1 in the first embodiment. [Figure 4] FIG. 13 is a diagram illustrating a method for calculating the deviation of bone length from a normal value. [Diagram 5] 13A and 13B are diagrams illustrating another method for calculating the deviation of walking motion from natural movement. [Figure 6] 10 is a flowchart illustrating an operation procedure of the motion analysis system 1 in the second embodiment. [Figure 7] The process of searching for corrected skeleton coordinates that can minimize the sum of E1 and E2 is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] <Embodiment 1> 1 is a configuration diagram of a motion analysis system 1 according to a first embodiment of the present invention. The motion analysis system 1 is a system that analyzes the walking motion of a subject to determine, for example, whether or not the subject has a walking dysfunction. The motion analysis system 1 includes a skeleton recognition unit 11, a period extraction unit 12, a period reliability determination unit 13, and an analysis unit 14.

[0013] The skeleton recognition unit 11 recognizes the skeleton of the subject using a captured image of the walking motion of the subject. For example, the captured image is used to recognize the joints of the subject, and the skeleton of the subject can be recognized based on the result. The recognized skeleton can be recorded as data describing, for example, the positions of the joints and their changes over time.

[0014] The period extraction unit 12 extracts the period of the walking movement of the subject based on the change over time of the captured image of the walking movement of the subject (or the skeletal position recognized by the skeletal recognition unit 11, the same applies below). Specifically, one period is defined as a time period during which the foot of the subject on the same side touches the ground twice.

[0015] The cycle reliability determination unit 13 determines the reliability of the cycle extracted by the cycle extraction unit 12. The skeletal movement recognized based on the captured image of the walking movement contains various noises (e.g., skeletal coordinates that are incorrectly recognized). The cycle reliability determination unit 13 selects, from among the cycles of the subject's walking movement (time periods during which the same foot touches the ground twice), those that are considered to have relatively small noises as cycles with high reliability. A specific method for determining the reliability of a cycle will be described later.

[0016] The analysis unit 14 analyzes the walking motion of the subject using the captured images of the walking motion of the subject in the cycle selected by the cycle reliability determination unit 13. For example, it analyzes whether the walking function of the subject is lower than a reference level. Since any known technology can be used as the analysis method, it will not be described in detail here.

[0017] 2 is a diagram for explaining a method for determining the reliability of each gait cycle. The cycle reliability determination unit 13 quantitatively evaluates the stability of the skeletal coordinates of the subject to determine the reliability of the gait cycle. The quantitative evaluation here is performed using an index that quantifies the human-likeness of the skeletal coordinates of the subject.

[0018] The cycle reliability determination unit 13 determines the reliability of the cycle by calculating an index representing the human-likeness of the skeletal coordinates for each cycle extracted by the cycle extraction unit 12. Specifically, the length between the joints of the subject is calculated for each time, and the variance of the length in the cycle is calculated. If the joints (bones) are considered to be rigid bodies, the length is ideally constant, so if the variance is smaller than a predetermined threshold, the reliability of the cycle is determined to be high. In other words, this variance can be used as an index of the human-likeness.

[0019] The cycle reliability determination unit 13 selects the cycle with the highest reliability calculated according to the above method from among the cycles extracted by the cycle extraction unit 12. The analysis unit 14 uses the cycle to analyze the walking motion of the subject. The cycle with the highest reliability may be selected, for example, all cycles whose index value (variance of joint-to-joint length at each time) satisfies a condition, or the top index value (e.g., top 50%) that satisfies a condition, or the single cycle with the best index value. Selection may be based on other appropriate criteria. In addition, if there is no index value that satisfies the condition, the analysis of the walking motion of the subject is interrupted. In that case, it is desirable to store a record of the interruption of the analysis and the reason thereof as a log inside the system, or to notify the system user.

[0020] FIG. 3 is a flowchart for explaining the operation procedure of the motion analysis system 1 in the first embodiment. In S301, the skeleton recognition unit 11 acquires the skeleton coordinates of the subject. In S302, the cycle extraction unit 12 extracts a plurality of walking cycles of the subject. In S303, the cycle reliability determination unit 13 calculates the variance of the lengths between the joints of the subject according to the method described in FIG. 2. The cycle reliability determination unit 13 performs S303 for all walking cycles extracted by the cycle extraction unit 12 (S304). In S305, the cycle reliability determination unit 13 selects, for example, a walking cycle whose variance is smaller than a threshold value as one with high reliability.

[0021] <First embodiment: Summary> The motion analysis system 1 according to the first embodiment selects a walking cycle with a small variance in the lengths between the joints of the subject as a highly reliable walking cycle, and uses the selected walking cycle to analyze the walking motion of the subject. This makes it possible to automatically select a walking cycle with a higher reliability of the skeletal coordinates, and to improve the accuracy of the walking analysis.

[0022] <Embodiment 2> In the first embodiment, an index representing the reliability (human-likeness) of a walking cycle is calculated from an image capturing the walking motion of a subject, and the walking motion is analyzed using the walking cycle with high reliability. In the second embodiment of the present invention, an example of an operation is described in which the deviation degree of the walking motion from a natural movement is quantified based on the human-likeness index, and the skeletal coordinates are corrected to minimize the deviation degree. The configuration of the motion analysis system 1 is the same as that of the first embodiment.

[0023] FIG. 4 is a diagram for explaining a method for calculating the deviation of bone length from the normal value. The deviation E1 can be calculated by adding up the squares of the differences between the inter-joint (bone) lengths and the standard lengths of the bones for all bones to be analyzed. E1 represents the degree to which the bone lengths deviate from the normal value. The bones to be analyzed are not necessarily limited to the femur (Femur in FIG. 4) and the tibia (Leg in FIG. 4), but other bones whose skeletal coordinates change with walking may be analyzed. The same applies to FIG. 5.

[0024] Figure 5 is a diagram explaining another method for calculating the deviation of walking motion from natural movement. It is assumed that human joints change smoothly in the time direction. Under this premise, an index representing the deviation of a joint is calculated using the product of the velocity vectors of the skeletal coordinates (joint coordinates) at adjacent times. The deviation is calculated by summing up these indices for all joints.

[0025] Joint P i The velocity vector of the skeleton coordinates is, for example, the position P i,t+1 and position P at time t i,t The difference between the joint P and the joint P is calculated by dividing the difference between the joint P and the joint P by the time difference Δt. i The graph shows how the velocity vector of the joint changes over time. The top left of Fig. 5 shows how the movement changes over time when it is natural, and the top right shows how the movement changes over time when it is unnatural. The deviation degree E2, which is the product of the velocity vectors at two consecutive times, can be calculated, for example, using the formula shown in Fig. 5. The deviation degree E2 indicates the degree to which the movement of the joint deviates from natural movement.

[0026] It is desirable to construct a function for calculating E2 so that it becomes a large value when multiplied by a velocity vector with a different sign, and becomes a smaller value when multiplied by a velocity vector with the same sign. By using such a function, it is possible to appropriately evaluate whether the velocity vector is changing smoothly or not. The function shown in Figure 5 is an example.

[0027] Even when a person is moving naturally, the direction of the velocity vector may change to the opposite direction. For example, this occurs when the legs move up and down during walking. However, if the frame rate when shooting the subject is increased accordingly, the change in the velocity vector between the two times before and after the point in time when the direction of the velocity vector changes to the opposite direction is likely to be small. Therefore, in this case, the deviation E2 will be a small value, and there is little possibility that natural movement will be mistaken for unnatural movement.

[0028] The cycle reliability determination unit 13 obtains a combination of skeletal coordinates that minimizes the sum of the deviations E1 and E2, for example, and corrects the skeletal coordinates acquired from the walking image according to the result. The combination of skeletal coordinates that minimizes the sum of the deviations may itself be the corrected skeletal coordinates, or some correction value may be derived from the combination. The analysis unit 14 uses the corrected skeletal coordinates to analyze the walking movement of the subject.

[0029] FIG. 6 is a flow chart for explaining the operation procedure of the motion analysis system 1 in the second embodiment. In S601, the skeleton recognition unit 11 acquires the skeleton coordinates of the subject. In S602, the cycle extraction unit 12 extracts the walking cycle of the subject. In S603, the cycle reliability judgment unit 13 acquires the skeleton coordinates for one frame. The skeleton coordinates for one frame here correspond to the coordinates P at one time point described in FIG. 5. In S604, the cycle reliability judgment unit 13 calculates the deviations E1 and E2 described in FIG. 4 to FIG. 5, and corrects the skeleton coordinates using these. Only one of E1 or E2 may be used. The cycle reliability judgment unit 13 performs S603 to S604 for all frames (all times within the walking cycle extracted in S602) (S605).

[0030] The period extracted in S602 may be a highly reliable walking period selected by, for example, the method described in embodiment 1. Alternatively, an arbitrary walking period may be selected, and then the joint coordinates may be corrected by the method described in embodiment 2. In either case, the use of accurate joint coordinates can improve the accuracy of analysis of the walking motion.

[0031] FIG. 7 shows the process of searching for the corrected skeleton coordinates that can minimize the sum of E1 and E2. The minimization process can be performed using any search algorithm. For example, the coordinate values ​​are arbitrarily varied around the initial values ​​of the joint coordinates, and if E1 and E2 decrease, the coordinate values ​​are provisionally adopted. The above process is performed an arbitrary number of times, and the coordinate values ​​finally obtained are determined as the corrected coordinates. If the sum of E1 and E2 becomes equal to or less than a threshold before the above process is performed a predetermined number of times, the process may be interrupted and the coordinate values ​​obtained at that time may be determined as the corrected coordinates. Furthermore, if the sum of E1 and E2 does not become equal to or less than the threshold after the predetermined number of trials, the correction may be determined to have failed and the coordinate values ​​may be returned to the initial values. If the sum of E1 and E2 does not decrease but increases, it can also be determined to have failed. If the correction fails, it is desirable to store a record of the failure and its cause as a log inside the system, or to notify the system user.

[0032] <Embodiment 2: Summary> The motion analysis system 1 according to the second embodiment calculates at least one of the deviation degree E1 of the joint length and the deviation degree E2 from natural motion, and corrects the skeletal coordinates to minimize the deviation degree. This makes it possible to automatically correct skeletal coordinates with low reliability, thereby improving the accuracy of gait analysis.

[0033] <Embodiment 3> The deviation E1 described in the second embodiment may be calculated for a single bone. For example, E1 may be calculated for any one bone in the trunk portion that is assumed to be highly reliable. Alternatively, E1 may be calculated for any one bone in the limbs that are likely to vary in reliability. For example, when a photographing device with large noise is used, it is desirable to calculate E1 for bones in a highly reliable portion. This is because bones in a less reliable portion are considered to have too much noise. When a photographing device with small noise is used, E1 may be calculated for bones in a less reliable portion. This is because it is considered that if E1 can be minimized for the less reliable portion, the reliability of the more reliable portion will be further increased.

[0034] The deviation degree E1 described in the second embodiment may be calculated by further weighting the length deviation degrees calculated for each of a plurality of bones. When an imaging device with large noise is used, the weight may be larger for a bone with high reliability than for other bones, and smaller for a bone with low reliability than for other bones. When an imaging device with small noise is used, the opposite may be true. The reason is the same as when E1 is calculated for a single bone.

[0035] The deviation E2 described in the second embodiment may be calculated for a single joint. For example, E2 may be calculated for any one joint of the trunk portion that is assumed to be highly reliable. Alternatively, E2 may be calculated for any one joint of the limbs whose reliability is likely to vary. The reason is the same as when E1 is calculated for a single bone.

[0036] The deviation degree E2 described in the second embodiment may be calculated by further weighting the deviation degrees from natural motion calculated for each of a plurality of joints. The weight may be larger for joints with high reliability than for other joints, and smaller for joints with low reliability than for other joints. Or the opposite may be true. The reason is the same as when E1 is calculated for a single bone.

[0037] In the second embodiment, it has been described that the skeletal coordinates are corrected so as to minimize the sum of E1 and E2. The sum of E1 and E2 may be a weighted addition. When the accuracy of bone length is emphasized in gait analysis, the weight of E1 is increased. When the smoothness of joint movement is emphasized in gait analysis, the weight of E2 is increased. Since E1 and E2 are not necessarily compatible, it is considered that which is emphasized depends on the characteristics of the movement and the part to be analyzed.

[0038] <Fourth embodiment> In the second embodiment, when the period reliability determination unit 13 corrects the skeletal coordinates, the three-axis coordinates (XYZ coordinates) of all joints are corrected simultaneously in principle. However, in order to improve the convergence of the calculation or to shorten the calculation time, the correction process can be performed individually for each processing unit as follows.

[0039] Instead of correcting all the joints at the same time, the joints may be corrected one by one. For example, it is possible to first correct the joints of the trunk one by one, and then correct the joints of the limbs one by one. Alternatively, it is possible to first correct the joints of the limbs one by one, and then correct the joints of the trunk one by one. The cycle reliability determination unit 13 recalculates the deviation degree every time the coordinates of one joint are corrected, and corrects the coordinates of the other joints again using the recalculated deviation degree (including the determination of whether correction is necessary, the same applies to the method described below). For example, inside S604 in FIG. 6, a loop is run to correct each joint as described above (the same applies to the method described below). It is considered that the order of the joints to be corrected is preferably, but not limited to, a position where the coordinate information is relatively stable (small noise) first. The same applies to the correction for each joint group and the correction for each coordinate axis described below.

[0040] Instead of correcting all the joints at the same time, correction may be performed for each joint group. For example, correction may be performed for each adjacent joint group, such as first correcting the joint group in the trunk, then correcting the joint group in the upper right half of the body, etc. Each time the coordinates of one joint group are corrected, the period reliability determination unit 13 recalculates the deviation degree, and then uses the recalculated deviation degree to recorrect the coordinates of the other joint groups.

[0041] Instead of correcting the XYZ coordinates simultaneously, correction may be performed for each coordinate axis. For example, correction is performed first for the X coordinate, then for the Y coordinate, and finally for the Z coordinate. The coordinate axes do not have to be in this order. The cycle reliability determination unit 13 recalculates the deviation degree every time the coordinate is corrected for one coordinate axis, and corrects the coordinates for the other coordinate axes again using the recalculated deviation degree. For example, when a subject is photographed using a two-dimensional camera, it is considered that noise in the depth direction is relatively large. Therefore, it is desirable to correct the coordinate axis in the depth direction last.

[0042] In the second embodiment, it has been described that the skeletal coordinates are corrected so as to minimize the sum of E1 and E2. The sum of E1 and E2 may be a weighted addition, or the weight may be dynamically changed. For example, in the swing phase, the coordinate change is large, so it is possible to relatively increase the weight of E2, and relatively increase the weight of E1 in the stance phase. Depending on the characteristics of the part to be analyzed, the opposite may be possible. For example, it is possible to relatively increase the weight of E1 for the trunk part, and relatively increase the weight of E2 for the limb parts.

[0043] <Modifications of the present invention> The present invention is not limited to the above-described embodiment, and includes various modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and it is not necessary to include all of the configurations described. In addition, a part of an embodiment can be replaced with a configuration of another embodiment. In addition, a configuration of another embodiment can be added to a configuration of an embodiment. In addition, a part of the configuration of each embodiment can be added to, deleted from, or replaced with a part of the configuration of another embodiment.

[0044] Each functional unit (skeleton recognition unit 11, period extraction unit 12, period reliability judgment unit 13, and analysis unit 14) of the motion analysis system 1 can be configured by hardware such as a circuit device that implements these functions, or can be configured by a calculation device such as a CPU (Central Processing Unit) executing software that implements these functions.

[0045] In the above embodiment, the subject of the walking analysis is assumed to be a human being, but the method of the present invention can be applied to other animals whose joints move with walking. Furthermore, even if the movement is other than walking, the method of the present invention can be applied to the movement cycle or the movement of the joints with the movement with periodic movement. [Explanation of symbols]

[0046] 1: Motion analysis system 11: Skeleton recognition unit 12: Period extraction part 13: Periodic reliability judgement unit 14:Analysis Department

Claims

1. a skeleton recognition unit for acquiring skeleton coordinate information of a target; a period extraction unit that extracts a plurality of motion periods of the object based on the skeletal coordinate information; a cycle reliability determination unit that determines the reliability of the skeletal coordinate information in the extracted multiple movement cycles; an analysis unit that selects at least one of the movement cycles based on the reliability, and analyzes the movement of the subject based on the skeletal coordinate information in the selected movement cycle; Equipped A motion analysis system comprising:

2. The cycle reliability determination unit calculates a length between the joints of the target for each time period, The cycle reliability determination unit determines the reliability using a variance of the length calculated for each time.

2. The motion analysis system according to claim 1.

3. the period reliability determination unit calculates an index representing a degree of deviation of the skeletal coordinate information from a resemblance to a human body; The cycle reliability determination unit calculates a correction value that can reduce the deviation by correcting the skeletal coordinate information based on the index, The analysis unit analyzes the motion of the object using the skeletal coordinate information to which the correction value has been applied.

2. The motion analysis system according to claim 1.

4. The cycle reliability determination unit determines the reliability by using the skeleton coordinate information to which the correction value has been applied.

4. The motion analysis system according to claim 3.

5. The period reliability determination unit calculates a first index value representing a deviation of the bone length of the subject from a normal value; The cycle reliability determination unit calculates a second index value representing a deviation of the motion of the joint of the subject from a normal value; The cycle reliability determination unit calculates the correction value that minimizes the sum of the first index value and the second index value.

4. The motion analysis system according to claim 3.

6. The cycle reliability determination unit calculates a length between the joints of the object, The period reliability determination unit sums up the length deviation calculated using the difference between the calculated length and a reference length for each joint, The period reliability determination unit uses the sum of the length deviations as the first index value.

6. The motion analysis system according to claim 5.

7. The cycle reliability determination unit acquires a velocity vector of the target joint at each time point; the cycle reliability determination unit sums, for each joint, a motion deviation calculated using a product of the velocity vectors at adjacent times in the same joint; The motion analysis system according to claim 5 , wherein the cycle reliability determination unit uses the sum of the motion deviations as the second index value.

8. The cycle reliability determination unit calculates the first index value for any one of the bones of the target; or The period reliability determination unit calculates the deviation of the bone length from a normal value for each of the plurality of bones of the subject, and calculates the first index value by performing a weighted sum of the calculation results.

6. The motion analysis system according to claim 5.

9. The cycle reliability determination unit calculates the second index value for any one of the target joints, or The cycle reliability determination unit calculates the deviation of each of the joints of the target from a normal value of the movement, and calculates the second index value by performing a weighted addition of the calculation results.

6. The motion analysis system according to claim 5.

10. The cycle reliability determination unit calculates the correction value that minimizes a weighted addition of the first index value and the second index value.

6. The motion analysis system according to claim 5.

11. The period reliability determination unit calculates values ​​of each coordinate axis of the correction value in a three-dimensional coordinate system for all joints of the target, The cycle reliability determination unit simultaneously corrects the skeletal coordinate information of each of the coordinate axes for all of the joints by collectively applying the correction value for each of the coordinate axes for all of the joints.

5. The motion analysis system according to claim 4.

12. The period reliability determination unit calculates the correction value for any one of the target joints and applies the calculated correction value to the joint for which the correction value was calculated; The cycle reliability determination unit minimizes the deviation by repeatedly calculating the index while changing the joint for which the correction value is calculated and applied.

5. The motion analysis system according to claim 4.

13. The period reliability determination unit calculates the correction value for one or more groups of the target joints, and applies the calculated correction value to the joints belonging to the group; The cycle reliability determination unit minimizes the deviation by repeatedly calculating the index while changing the group for which the correction value is calculated and applied.

5. The motion analysis system according to claim 4.

14. the cycle reliability determination unit calculates a value of the correction value for any one coordinate axis in a three-dimensional coordinate system, and applies the calculated correction value to the coordinate axis for which the correction value was calculated; Calculating and applying the correction value. Minimizing the deviation by repeatedly calculating the index while varying the coordinate axis.

5. The motion analysis system according to claim 4.

15. the subject is an animal that is ambulatory; the movement is walking, the period reliability determination unit dynamically changes a weight in the weighted addition of the first index value and the second index value according to a time-dependent transition of the periodic behavior of the exercise movement of the subject; the cycle reliability determination unit adjusts a weight in the weighted addition of the first index value and the second index value so that, when the walking motion of the subject is in a stance phase, a weight for the first index value is greater than a weight for the second index value; The cycle reliability determination unit adjusts weights in the weighted addition of the first index value and the second index value such that, when the walking motion of the subject is in a swing phase, a weight for the second index value is greater than a weight for the first index value. The motion analysis system according to claim 10 .

Images