Gait analysis system and learning method thereof

Abstract

To solve problems that there is no mention with respect to a stride calculation algorithm and the algorithm has low accuracy, and a stride is difficult to be acquired by a camera included in a moving walking vehicle or movable body since the camera itself moves.SOLUTION: In a gait analysis system, a stride estimation algorithm uses as inputs, at least leg position information of a person to be measured and a movement amount acquired by a movement amount acquisition unit. A curve indicating a time transition of a distance between right and left legs as desired in a travel direction is used.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present invention relates to a gait analysis system that enables highly accurate estimation of stride length and a learning method thereof. [Background technology]

[0002] Patent Documents 1 to 3 disclose techniques for estimating a pedestrian's stride length by using a distance sensor, an acceleration sensor, or the like.

[0003] Patent Documents 4 and 5 disclose techniques for estimating the posture of a patient undergoing rehabilitation based on an image of the patient captured by a camera, and supporting the rehabilitation.

[0004] Non-Patent Document 1 discloses a technology for a walker-type gait analysis system that performs gait analysis by estimating the posture of a user through deep learning based on depth images captured by an RGB-D camera mounted on the walker. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2017-205213 A [Patent Document 2] JP 2019-219187 A [Patent Document 3] JP 2019-179456 A [Patent Document 4] JP 2015-089412 A [Patent Document 5] Patent No. 7042956 [Non-patent literature]

[0006] [Non-Patent Document 1] Palermo, M., Moccia, S., Migliorelli, L., Frontoni, E. and Santos, CP, "Real-time human pose estimation on a smart walker using convolutional neural networks", Expert Systems with Applications Vol.184 (2021), DOI:10.1016 / j.eswa.2021.115498. Summary of the Invention [Problem to be solved by the invention]

[0007] The distance sensors and acceleration sensors disclosed in Patent Documents 1 to 3 are expensive, and there is a demand for a cheaper method of estimating stride length and walking speed. Thus, simple techniques using deep learning are disclosed in Patent Documents 4 and 5. However, there is no mention of an algorithm for calculating stride length. Furthermore, the technique described in Non-Patent Document 1 does not take into account the amount of movement of the camera attached to the walker, and therefore cannot estimate stride length or walking speed. In view of the above-described conventional situation, the present disclosure has an object to estimate a human stride length and walking speed using a simple system. [Means for solving the problem]

[0008] The present disclosure solves the above-mentioned problems by providing a gait analysis system that estimates stride length and walking speed with high accuracy using an image acquisition unit such as an inexpensive web camera, a posture estimation algorithm, a stride length estimation algorithm, and a walking speed estimation algorithm.

[0009] A gait analysis system according to an embodiment of the present disclosure includes: an image acquisition unit for acquiring an image of a person to be measured who is walking; a calculation unit having a posture estimation algorithm for estimating skeletal information of the subject from the image and a stride estimation algorithm for estimating a stride; In a gait analysis system comprising: The stride estimation algorithm comprises: At least the leg position information of the subject is input, A curve showing the time progression of the distance between the left and right legs of any one of the legs in the direction of travel is used.

[0010] A gait analysis system according to an embodiment of the present disclosure includes: A walking aid to assist users in walking. An image acquisition unit provided on the walking frame for acquiring an image of a user; A movement amount acquisition unit that acquires a movement amount of the walking vehicle; A calculation unit having a posture estimation algorithm for estimating skeletal information of the user in the image and a stride estimation algorithm for estimating a stride; In a gait analysis system comprising: The stride estimation algorithm comprises: At least leg position information of the subject; The movement amount acquired by the movement amount acquisition unit; With the input, A curve showing the time progression of the distance between the left and right legs of any one of the legs in the direction of travel is used.

[0011] A gait analysis system according to an embodiment of the present disclosure includes: an image acquisition unit for acquiring an image of a person to be measured who is walking; a movement amount acquisition unit that acquires a movement amount of the image acquisition unit; A calculation unit having a posture estimation algorithm for estimating skeletal information of the user in the image and a stride estimation algorithm for estimating a stride; In a gait analysis system comprising: The stride estimation algorithm comprises: At least leg position information of the subject; The movement amount acquired by the movement amount acquisition unit; With the input, A curve showing the time progression of the distance between the left and right legs of any one of the legs in the direction of travel is used. Effect of the Invention

[0012] As described above, the gait analysis system of the present disclosure makes it possible to estimate a stride length with high accuracy at low cost. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is an overall configuration diagram of a system according to a first embodiment of the present disclosure. [Diagram 2] FIG. 2 is an external view of the first embodiment of the present disclosure. [Diagram 3] FIG. 3 is an example of the output of the pose estimation algorithm. [Figure 4] FIG. 4 is a diagram showing the definition of stride length. [Diagram 5] FIG. 5 shows an example of the transition of the estimated ankle position over time during normal walking and during unhealthy walking. [Figure 6] FIG. 6 is a diagram showing a flowchart of a method for estimating a stride length by directly utilizing a time transition in the moving direction of the ankle in the first embodiment of the present disclosure. [Figure 7] FIG. 7 is a diagram illustrating a curve showing the transition of the distance between the left and right ankles over time in the direction of travel in the first embodiment of the present disclosure. [Figure 8] FIG. 8 is a diagram showing a flowchart of a method for estimating a stride length using a curve showing a time transition of the distance between the left and right ankles in the direction of travel in the first embodiment of the present disclosure. [Figure 9] FIG. 9 is an overall configuration diagram of a system according to the second embodiment of the present disclosure. [Figure 10] FIG. 10 is an external view of the second embodiment of the present disclosure. [Figure 11] FIG. 11 is a diagram showing a method for estimating a curve showing the time transition of the distance between the left and right ankles in the traveling direction in the second embodiment of the present disclosure. [Figure 12] FIG. 12 is an external view when learning data is acquired in the second embodiment of the present disclosure. [Figure 13] FIG. 13 is an overall configuration diagram of a system according to the third embodiment of the present disclosure. [Figure 14]FIG. 14 is an external view of the third embodiment of the present disclosure. [Figure 15] FIG. 15 is a diagram illustrating a derived example in the third embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] (Background to this disclosure) In recent years, the importance of rehabilitation has increased due to the aging population, and walking rehabilitation is one example of this. Effective rehabilitation can be achieved by quantitatively analyzing walking (gait analysis). Among gait analysis, analyzing stride length is a difficult task because it is necessary to extract the timing when both feet land (stance phase) and calculate the distance between the left and right feet.

[0015] For example, according to Patent Document 1, stride length can be obtained by attaching multiple distance sensors to the lower limbs, but multiple sensors must be worn on the body. According to Patent Document 2, stride length can be obtained with one acceleration sensor, but it is known that calculation of travel distance by double integration of the acceleration sensor has a large error. According to Patent Document 3, stride length can be obtained by measuring the legs with a distance sensor attached to a moving object. Although it is not necessary to wear a sensor on the body, a moving object equipped with a distance sensor is required, which complicates the system and increases costs.

[0016] In recent years, advances in deep machine learning have led to the development of systems capable of gait analysis using simple methods such as those described in Patent Documents 4 and 5, which require only a camera and a computer or smartphone. These systems use posture estimation technology to estimate human skeletal information. However, simply using the skeletal information obtained by these posture estimations makes it difficult to extract the timing at which both feet land (stance phase) and to estimate the distance between the left and right feet, and it is not possible to obtain the stride length with high accuracy. Furthermore, these disclosures do not specifically state a method for estimating the stride length.

[0017] In addition, gait analysis when using a walking vehicle is also important. According to Non-Patent Document 1, whole-body gait analysis is performed using two cameras. However, since the camera mounted on the walking vehicle moves, information on the amount of movement such as stride length and walking speed cannot be obtained. In addition, using two cameras is not realistic in terms of mass production, as re-learning is required when the positional relationship between the cameras is changed.

[0018] The inventors discovered that by using information on the walking direction (travel direction) of a pedestrian in the skeletal information obtained from a posture estimation algorithm, it is possible to estimate the stride length with high accuracy. In addition, they discovered that even in images captured by a moving camera such as a walking vehicle or a handheld camera, it is possible to estimate the stride length with high accuracy by estimating a curve showing the time transition of the distance between the left and right legs of any one of the legs through multimodal learning that integrates the amount of movement and skeletal information.

[0019] Therefore, the present inventors arrived at the following invention.

[0020] A first invention is a gait analysis system comprising an image acquisition unit that acquires an image of a person being measured while walking, and a calculation unit having a posture estimation algorithm that estimates skeletal information of the person being measured from the image, and a stride estimation algorithm that estimates a stride, wherein the stride estimation algorithm receives as input time-series position information of the person being measured's legs in the direction of travel, and has the steps of extracting a section in which the speed in the direction of travel of specific parts of the left and right legs is equal to or less than a first threshold value, acquiring statistics for the section, and estimating the stride from information (such as the difference) before and after the statistics arranged in time series.

[0021] This makes it easier to extract the stride length even for disabled people who cannot lift their feet vertically. Also, since the direction of a human is greater in the forward direction than in the vertical direction, robustness against errors in the posture estimation algorithm is improved, making it easier to extract the stride length.

[0022] A second invention is a gait analysis system comprising an image acquisition unit that acquires an image of a person being measured while walking, and a calculation unit having a posture estimation algorithm that estimates skeletal information of the person being measured from the image, and a stride estimation algorithm that estimates a stride, wherein the stride estimation algorithm receives at least leg position information of the person being measured as input, and uses a curve that shows the time progression of the distance between the left and right of any leg in the direction of travel.

[0023] This makes it easy to detect stride length even in abnormal gait, such as a dragging gait caused by paralysis of the lower limbs of the subject, and is useful for gait analysis in rehabilitation. In addition, walking, which is a periodic movement, can be expressed as a curve showing the time transition of the periodic distance between the left and right legs of any given leg. The periodic expression provides excellent human readability and computer processability, such as determining the left and right legs by the code of the stride length curve.

[0024] A third invention is a gait analysis system comprising a walking frame for assisting a user while walking, an image acquisition unit attached to the walking frame for acquiring an image of the user, a movement amount acquisition unit for acquiring the amount of movement of the walking frame, and a calculation unit having a posture estimation algorithm for estimating skeletal information of the user from the image, and a stride estimation algorithm for estimating a stride, wherein the stride estimation algorithm receives as input at least leg position information of the person being measured and the movement amount acquired by the movement amount acquisition unit, and uses a curve showing the time progression of the distance between the left and right of any leg in the direction of travel.

[0025] This makes it possible to estimate the stride length of a user who uses a walking frame. By acquiring the amount of movement using the movement amount acquisition unit, the stride length can be estimated even in an environment where the image acquisition unit is mounted on the walking frame and moves.

[0026] A fourth aspect of the present invention is that the image acquisition unit and the movement amount acquisition unit are detachable from an existing walking frame.

[0027] This makes it possible to turn an existing walker into one that can perform gait analysis simply by attaching an inexpensive web camera, an encoder, and a small computer to it. Because the walker and other components can be assembled from ready-made products, the system can be offered at a lower cost than conventional gait analysis systems.

[0028] A fifth invention is a gait analysis system comprising an image acquisition unit that acquires an image of a person being measured who is walking, a movement amount acquisition unit that acquires the amount of movement of the image acquisition unit, and a calculation unit having a posture estimation algorithm that estimates skeletal information of the user in the image, and a stride estimation algorithm that estimates a stride, wherein the stride estimation algorithm receives as input at least leg position information of the person being measured and the amount of movement acquired by the movement amount acquisition unit, and uses a curve that shows the time progression of the distance between the left and right of any leg in the direction of travel.

[0029] This allows the stride length to be estimated even when the image is from a moving image acquisition unit. This allows the stride length to be estimated if the amount of movement can be acquired, and allows the stride length to be estimated by a moving body equipped with an image acquisition unit, a smartphone, or a tablet. In particular, the ability to estimate the stride length by a smartphone or tablet can provide a simple gait analysis means in clinical settings. A method for acquiring movement amount data on a terminal such as a smartphone or tablet, integrating the movement amount data with skeletal information, and performing motion analysis of a subject has already been disclosed, but by using a curve showing the time transition of the distance between the left and right of any leg in the direction of travel, the stride length can be estimated with higher accuracy.

[0030] A sixth invention is characterized in that the stride estimation algorithm estimates the stride from statistics of a curve showing the change over time in the distance between the left and right legs of any of the legs in the traveling direction when the differential value of the curve showing the change over time in the distance between the left and right legs of any of the legs is equal to or less than a second threshold value.

[0031] This makes it easy to extract the section in which both feet are in contact with the ground, and makes it easy to estimate the stride length.

[0032] The seventh invention is a gait analysis system comprising a walking frame for assisting a user walking, an image acquisition unit attached to the walking frame for acquiring an image of the user, a movement amount acquisition unit for acquiring the amount of movement of the walking frame, and a calculation unit that receives as input at least the image acquired from the image acquisition unit and the movement amount acquired from the movement amount acquisition unit, in which supervised learning is performed using as teacher data a curve showing the time progression of the distance between the left and right sides of any leg in the direction of travel.

[0033] This makes it possible to integrate the skeletal information of the walking subject and the amount of movement acquired by the movement amount acquisition unit, and estimate the stride length curve by multimodal machine learning. This makes it possible to estimate the stride length even when the image acquisition unit is mounted on a moving body such as a walking vehicle.

[0034] The eighth invention is a gait analysis system including an image acquisition unit that acquires an image of a person being measured while walking, a movement amount acquisition unit that acquires the amount of movement of the image acquisition unit, and a calculation unit that receives as input at least the image acquired from the image acquisition unit and the amount of movement acquired from the movement amount acquisition unit, in which supervised learning is performed using as teacher data a curve that shows the time progression of the distance between the left and right of any leg in the direction of travel.

[0035] This makes it possible to integrate the skeletal information of the walking subject with the amount of movement acquired by the movement amount acquisition unit, and estimate the stride length curve through multimodal machine learning. This makes it possible to estimate stride length even with an image acquisition unit mounted on a moving object. Gait analysis (stride length estimation) becomes possible with a single general-purpose device such as a smartphone or tablet terminal that has all of the image acquisition unit, movement amount acquisition unit, calculation unit, and analysis result display unit.

[0036] As described above, according to the gait analysis system of the present disclosure, stride length can be estimated inexpensively and with high accuracy.

[0037] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the embodiment. EXAMPLES

[0038] 1 and 2 are diagrams showing the configuration of the system, and Fig. 3 to Fig. 8 are diagrams explaining the processing of the system. Using these, the first embodiment will be explained.

[0039] (1. Overall configuration of the gait analysis system) First, the overall configuration of a gait analysis system 11 will be described with reference to FIGS.

[0040] Fig. 1 is a diagram showing the overall configuration of the system. Gait analysis system 11 is composed of image acquisition unit 12, calculation unit 13, and analysis result presentation unit 14. Calculation unit 13 has posture estimation algorithm 13a, stride estimation algorithm 13b, walking speed estimation algorithm 13c, and other gait analysis algorithm 13d. Image acquisition unit 12 acquires image data 15 of subject 21, calculation unit 13 analyzes the gait of subject 21, and analysis result presentation unit 14 presents gait analysis result 20. The analysis process will be described later.

[0041] FIG. 2 is an external view of the gait analysis system 11. The image acquisition unit 12 is arranged by a stand 22 so as to capture an image of the person being measured 21 walking. The image acquisition unit 12 may be any device capable of acquiring images, such as a general web camera, an RGB-D camera (depth camera), a smartphone, an industrial camera, a video camera, etc. The images also include videos that can be broken down into images for each frame. As for the stand 22, it is sufficient that the camera is arranged in an appropriate position, and stand 22 does not have to be used if the camera is mounted on a desk or held by hand. In this embodiment, the web camera is fixed by a tripod.

[0042] The calculation unit 13 is a processing device for processing an algorithm described later. If the calculation unit 13 can process the image acquired by the image acquisition unit 12, it may be separate from the image acquisition unit 12 or may be the same terminal as the image acquisition unit 12, such as a smartphone. If the calculation unit 13 is separate from the image acquisition unit 12, the image acquired by the image acquisition unit 12 may be transferred after being stored in an image storage device (not shown). For the transfer, a transfer means such as a method using a USB flash memory or wireless transfer via a local network is used. In addition, the image acquisition unit 12 and the calculation unit 13 may be directly connected by a communication means such as a USB cable. In this embodiment, the web camera and the calculation unit 13 are connected by a USB cable, and after the captured image data is stored, analysis is performed by a human instruction.

[0043] (2. Analysis Processing of Gait Analysis System) The posture estimation algorithm 13a for estimating the skeletal information 16 possessed by the calculation unit 13 is an algorithm for estimating the skeletal information 16 of the person being measured 21 based on an image of the person being measured 21 acquired by the image acquisition unit 12 as shown in FIG. 3. Using deep learning from the image, the positions of the center positions of the joints of the person appearing in the image, and the positions of the centers and end points of parts (centers of the eyes and nose) are estimated as key points 31. Although posture estimation is possible using only an image thanks to recent deep learning, depth information may also be used in addition to the image. The gait of the person being measured 21 is analyzed by posture estimation.

[0044] Generally, posture estimation algorithms include 2D posture estimation, which estimates two-dimensional keypoint positions on a captured image plane, and 3D posture estimation, which estimates depth information in addition to the image plane. In this embodiment, 3D posture estimation is used. In cases where an image can be captured from the side of a walking subject, information on the direction of travel can be obtained even with 2D posture estimation, and the present invention is applicable.

[0045] FIG. 4 shows the state when both feet are on the ground. The distance 43 between the heels of the left foot 41 and right foot 42 when both feet are on the ground is defined as the stride 17. In order to estimate the stride 17, it is necessary to extract the timing when both feet are on the ground. FIG. 5 shows the transition of the ankle position over time, estimated by the posture estimation algorithm 13a. The ankle was used because the posture estimation algorithm 13a used did not have a heel key point 31, and the key point 31 to be used can be any position on the foot, such as the heel or toes.

[0046] According to the time transition of the vertical position of the ankle in healthy walking in Figure 5 (a), the minimum section (when touching the ground) 51 required to extract the timing when the foot touches the ground and the section 52 where the foot is not touching the ground during the swing leg are easy to see, and it is easy to extract the timing when both feet are touching the ground. (In the figure, 51 and 52 are only shown for the right foot. The same is true for the left foot.) However, when looking at the time transition of the vertical position of the ankle in unhealthy walking in Figure 5 (b), the minimum section (when touching the ground) of the left foot is clear, but the minimum section (when touching the ground) 53 of the right foot is unclear, making it difficult to extract the timing when both heels are touching the ground and estimating the stride length 17.

[0047] Therefore, in this embodiment, the transition of the ankle position in the direction of travel over time is used as shown in Figures 5(c) and 5(d). The transition of the ankle position in the direction of travel over time makes it easy to understand the movement of the left and right feet (stance and swing phases) in both healthy and unhealthy walking, and makes it easy to extract the timing when both feet are on the ground, and to estimate the stride length 17.

[0048] In the progression of the position of the ankle in the direction of travel over time, the foot that is on the ground (the foot in the stance phase) does not move, and the foot that is not on the ground (the foot in the swing phase) moves in the direction of travel. Therefore, the section where there is no movement, i.e., the flat section 54 in the progression of the direction of travel over time, is the timing when the foot is on the ground, and the distance 55 before and after this section almost coincides with the stride length 17. The characteristics of this graph of the progression of the direction of travel over time do not change even in unhealthy walking. In other words, the stride length 17 can be easily estimated by using the progression of the ankle in the direction of travel over time.

[0049] Here, the process of estimating the stride length 17 will be described in Fig. 6. For the reasons mentioned above, position information in the forward direction is used instead of position information in the up and down direction which generally indicates the foot's contact with the ground.

[0050] First, the left foot is analyzed. In step ST61, the position 61 of the left ankle in the direction of travel is plotted in time series. In order to calculate the section in which the foot is on the ground, which is necessary for estimating the stride length 17, it is necessary to obtain an extraction section 62, which is a flat section in the direction of travel over time.

[0051] Therefore, in step ST62, the extraction section 62 is set as a section in which the differential value (speed) of the time transition in the traveling direction is equal to or less than a first threshold, and this section is extracted.

[0052] Next, in step ST63, the statistics of the extracted interval are obtained. In this embodiment, the average value is used as the statistics. However, any other means may be used to obtain the extracted interval, such as the median value or the average value multiplied by a coefficient.

[0053] Finally, in step ST64, the stride length 17 of each n-th step is estimated by taking the difference in the statistics of the sections arranged in time series as the stride length 17. In this embodiment, the difference in the statistics of the sections arranged in time series is taken as the stride length, but instead of the difference, statistical processing using the statistics of the sections arranged in time series or a machine learning method may be used.

[0054] The same analysis method is used for the right foot. It goes without saying that the order of processing the left and right feet may be reversed.

[0055] However, since the stride length 17 is a periodic movement, the analysis would be simpler if a periodic analysis could be performed.

[0056] Therefore, a curve showing the time transition of the left-right distance of any leg in the traveling direction shown in FIG. 7 is used. In this embodiment, a curve 74 showing the time transition of the left-right distance 73 in the traveling direction of a right ankle key point 72 relative to a left ankle key point 71 is used. The left-right distance 73 of this curve is always the distance in the traveling direction between the left and right ankles, regardless of the stance phase or swing phase (both feet are on the ground). In this embodiment, ankle key points 71 and 72 are used, but the key points used may be any position on the foot, such as the heel or toe. Hereinafter, the curve 74 showing the time transition of the left-right distance 73 of the ankles is defined as a stride curve 74.

[0057] The stride curve 74 is a periodic curve according to walking, and analysis is simplified. Around the time when both feet touch the ground, the amount of change in the stride curve 74 decreases, and the absolute value of the differential value 75 of the stride curve 74 falls within a certain threshold, making it possible to extract a both-foot-ground section 76. The threshold is a second threshold 77. Then, the stride 17 can be estimated by taking the average value of the stride curve 74 in that section as the estimated value of the stride 17. In addition, the sign of the stride curve 74 is used to determine whether the foot is left or right, and it is possible to determine the number of steps in the order in which the stride 17 was detected. The second threshold 77 in this embodiment is 100 mm / s. The second threshold 77 is a parameter that should be adjusted appropriately depending on the system configuration, but a range of about 20 mm / s to 300 mm / s is appropriate as a value for extracting a section in which the amount of change in the stride curve 74 decreases.

[0058] As shown in FIG. 5(e) and FIG. 5(f), the stride length curve 74 does not change its tendency depending on whether the person is walking in a healthy or unhealthy direction, and is useful for estimating the stride length 17 in the unhealthy direction.

[0059] A method for estimating the stride 17 using the stride curve 74 in this embodiment will be described in detail with reference to FIG.

[0060] First, in step ST81, the distance between the left and right ankles in the direction of travel (the stride length curve 74) is plotted in time series.

[0061] Next, in step ST82, the time series data of the distance between the left and right ankles in the direction of travel is differentiated with respect to time.

[0062] Next, in step ST83, a section where the time derivative value of the time-series data of the distance between the left and right ankles in the moving direction (derivative value 75 of the stride length curve 74) is equal to or smaller than a second threshold value 77 is extracted.

[0063] Next, in step ST84, the average value of the distance between the left and right ankles in the moving direction in the extracted section (the stride curve 74) is acquired as the stride 17.

[0064] Finally, in step ST85, the left or right foot is determined according to the sign of the acquired stride length 17, and the stride lengths 17 are determined in order from the first step onwards as the result of the gait analysis.

[0065] In this embodiment, the average value of the stride length curve 74 within the section is used, but other statistical values ​​within the section, such as the median value within the section or a value obtained by multiplying the average value by a coefficient, may also be used.

[0066] The walking speed estimation algorithm 13c estimates the walking speed 18 from the skeleton information 16. Also, a other gait analysis algorithm 13d may be provided to perform other desired gait analyses.

[0067] (3. Presentation of analysis results of gait analysis system) Each analysis result processed by the calculation unit 13 is presented to the subject 21, a physical therapist (not shown), etc. by the analysis result presentation unit 14. The presentation method may be displaying on a monitor (not shown) or listing on a printed matter. Also, based on the results of each analysis algorithm, a comprehensive index such as the degree of walking independence and the risk of falling may be presented.

[0068] The operation and effects of the gait analysis system 11 configured as above will be described below.

[0069] In estimating stride length using posture estimation algorithm 13a that estimates skeletal information 16, stride length 17 is estimated using position information in the direction of travel, not in the vertical direction (whether or not the feet are on the ground) of subject 21. The estimation includes step ST62 of extracting a section where the speed in the direction of travel of specific parts of the left and right legs is equal to or less than a first threshold, step ST63 of acquiring statistics for the section, and step ST64 of estimating stride length 17 from the distance before and after the statistics arranged in time series. This makes it possible to easily detect stride length 17 even in abnormal gait such as a dragging gait caused by paralysis of the lower limbs of subject 21, which is useful for gait analysis in rehabilitation.

[0070] Next, by using a curve (step length curve 74) showing the time transition of the distance 73 between the left and right legs of any one of the subjects 21, the step length 17 can be easily detected even in abnormal gait such as a dragging gait caused by paralysis of the lower limbs of the subject 21, which is useful for gait analysis in rehabilitation. In addition, walking, which is a periodic movement, can be expressed by the periodic step length curve 74, and the code of the step length curve 74 can be used to determine the left and right feet, etc., and the data is easy to read when viewed by humans and easy to process by computers. EXAMPLES

[0071] 9 and 10 are diagrams showing the system configuration, Fig. 11 is a diagram explaining the processing of the system, and Fig. 12 is a diagram explaining the appearance of the stride length curve when learning data is collected. Using these, Example 2 will be explained. Contents that overlap with Example 1 may be omitted.

[0072] (1. Overall configuration of the gait analysis system) The overall configuration of the gait analysis system 11 will be described with reference to the system configuration diagrams of FIG. 9 and FIG.

[0073] FIG. 9 is a diagram showing the overall configuration of the system. The gait analysis system 11 is composed of a walker 91, a calculation unit 13, and an analysis result presentation unit 14. The walker 91 includes an image acquisition unit 12 and a movement amount acquisition unit 92. The calculation unit 13 has a posture estimation algorithm 13a, a stride estimation algorithm 13b, a walking speed estimation algorithm 13c, and other gait analysis algorithms 13d. The image acquisition unit 12 acquires image data 15 of the user 101, the movement amount acquisition unit 92 acquires movement amount data 93 of the walker 91, the calculation unit 13 analyzes the walking of the user 101, and the analysis result presentation unit 14 presents the analysis results. The image data 15 and the movement amount data 93 have measured time information, and it is preferable that the two data can be analyzed in synchronization. The analysis process will be described later.

[0074] 10 is an external view of a gait analysis system 11 that uses a walking vehicle 91. An image acquisition unit 12 and a movement amount acquisition unit 92 are provided in the walking vehicle 91 that assists a user 101. The image acquisition unit 12 and the movement amount acquisition unit 92 may be provided detachably, or may be retrofittable to an existing walking vehicle 91.

[0075] The image acquisition unit 12 may be a device capable of acquiring images, such as a general web camera, an RGB-D camera (depth camera), a smartphone, an industrial camera, a video camera, etc. The images also include videos that can be decomposed into images for each frame.

[0076] The image acquisition unit 12 attached to the walker 91 and the user 101 are close to each other, so the subject distance is close. Therefore, with a normal camera, only a part of the body of the user 101 can be captured, and the gait of the entire body of the user 101 cannot be analyzed. Walking is a whole-body movement, and whole-body gait analysis is important. Therefore, it is advisable to use one wide-angle camera for the image acquisition unit 12. In particular, by using a wide-angle camera of 160 degrees or more, the entire body can be captured with one camera, and whole-body gait analysis becomes possible without the disadvantages of using multiple cameras.

[0077] The walking vehicle 91 is provided with a movement amount acquisition unit 92, which can acquire movement amount data 93. An encoder is provided on the running wheels of the walking vehicle 91 as the movement amount acquisition unit 92. The movement amount acquisition unit 92 may alternatively employ a method in which light is irradiated onto the floor surface and the reflection is detected by a sensor, a method in which a rotating body is grounded on the floor surface and the rotation is detected by a rotary encoder, a method in which the movement amount is obtained from self-position estimation, a method using pedestrian autonomous navigation positioning, or the like, as long as it is possible to acquire the movement amount.

[0078] The calculation unit 13 is a processing device for processing an algorithm described later. The calculation unit 13 may be separate from the walker 91 or may be the same unit mounted on the walker 91, as long as it can process the image data 15 acquired by the image acquisition unit 12 and the movement amount data 93 acquired by the movement amount acquisition unit 92. If it is separate from the walker 91, the image acquired by the image acquisition unit 12 and the movement amount data 93 acquired by the movement amount acquisition unit 92 may be transferred after being stored in a storage device (not shown). For the transfer, a transfer means such as a method using a USB flash memory or wireless transfer via a local network is used. In addition, the image acquisition unit 12, the movement amount acquisition unit 92, and the calculation unit 13 may be directly connected by a communication means such as a USB cable.

[0079] (2. Analysis Processing of Gait Analysis System) In this example, the stride 17 is estimated using the stride curve 74 as in the first embodiment. Regarding posture estimation, only 3D posture estimation can be used when measuring the user 101 from the front. In this example, the image acquisition unit 12 is provided on the walker 91, and therefore the image acquisition unit 12 moves. Therefore, it is not possible to estimate the stride curve 74 only by posture estimation from an image. The stride curve 74 has been explained in the first embodiment, so a description thereof will be omitted.

[0080] 11, skeletal information 16 of a walking user 101 and movement amount data 93 acquired by a movement amount acquisition unit 92 are integrated to estimate a stride length curve 74. In the machine learning method, position information 111 of ankle key points 71, 72, which are part of the skeletal information 16 of the user 101, and movement amount data 93 are input, and a model is constructed using supervised learning in which the true value of the stride length curve 74 measured by a measuring device is used as training data.

[0081] In this embodiment, a multi-layer perceptron (MLP) 112 was used. For input data, data of 81 frames including the estimated frame and 40 frames before and after were used to learn the transition of time-series data. The value of the stride length curve 74 in the center (41st frame) of the 81 frames was used as the output (estimated value).

[0082] To collect learning data in supervised learning, as shown in Fig. 12, retroreflective markers 121 attached to the user 101 or the walking frame 91 and a three-dimensional position measuring device 122 were used to collect data (true values) from the measuring device when the user 101 was walking, as well as image data 15 and movement amount data 93. The three-dimensional position measuring device 122 is a device that encloses a measurement range with multiple infrared cameras 123 and measures the positions of the retroreflective markers 121. The gait analysis system 11 may be provided with an infrared emitting body 124 for time synchronization with the three-dimensional position measuring device 122 when collecting learning data.

[0083] The position information 111 of the ankle key points 71, 72 does not have to be the ankle, but may be any key point 31 on the leg. The position information 111 may use both the up and down direction and the forward direction, or only one of them. In addition, multiple key points 31 may be input, not just one set of left and right key points 31. Furthermore, the data may be differentiated and converted into a speed. The movement amount data 93 may use a value converted into a speed based on the movement amount data 93.

[0084] In this embodiment, the MLP 112 is used as the machine learning method, but a recurrent neural network or a Transformer may also be used.

[0085] In this embodiment, a three-dimensional position measuring device 122 is used to obtain the true value of the stride length curve 74, which is the teacher data. Other means for obtaining the true value may be any means capable of measuring the left-right distance 73 between any legs in the direction of travel, such as a motion capture system in which acceleration sensors are attached to the legs or the whole body, a method using a 2D or 3D LIDAR, or a method of acquiring gait by arranging multiple sensors on the walking path.

[0086] The operation and effects of the gait analysis system 11 configured as above will be described below.

[0087] The skeletal information 16 of the walking user 101 estimated by the posture estimation algorithm 13a and the movement amount data 93 acquired by the movement amount acquisition unit 92 are integrated, and the stride length curve 74 is estimated by multimodal machine learning. In the conventional disclosure, it was difficult to estimate the stride length 17 and walking speed 18 from the skeletal information 16 of the image acquisition unit 12 mounted on a moving body such as a walker 91. Furthermore, a detailed estimation algorithm for the stride length 17 has not been disclosed. By using the stride length curve 74 of the present disclosure, the stride length 17 can be estimated even when the image acquisition unit 12 is mounted on a moving body such as a walker 91.

[0088] The image acquisition unit 12 and the movement amount acquisition unit 92 may be provided detachably, and can be retrofitted to an existing walker 91. This allows the walker 91 to be capable of gait analysis simply by attaching, for example, an inexpensive web camera, an encoder, and a small computer to the existing walker 91. Since the walker 91 can be configured using ready-made products, it can be provided at a lower cost than conventional gait analysis systems. EXAMPLES

[0089] 13 and 14 are diagrams showing the configuration of the system, and FIG. 15 is a diagram showing the derivation of the system. Using these, Example 3 will be described. Contents that overlap with Examples 1 and 2 may be omitted. Note that although subject 21 and user 101 are written differently depending on whether or not they use a walking frame 91, they are processed in the same way in the gait analysis (stride analysis) process.

[0090] (1. Overall configuration of the gait analysis system) The overall configuration of the gait analysis system 11 will be described with reference to the system configuration diagrams of FIG. 13 and FIG.

[0091] FIG. 13 is a diagram showing the overall configuration of the system. The gait analysis system 11 is composed of an image acquisition unit 12, a calculation unit 13, and an analysis result presentation unit 14. The image acquisition unit 12 includes a movement amount acquisition unit 92. The calculation unit 13 has a posture estimation algorithm 13a, a stride length estimation algorithm 13b, a walking speed estimation algorithm 13c, and other gait analysis algorithms 13d. The image acquisition unit 12 acquires image data 15 of the user 101, the calculation unit 13 analyzes the gait of the person being measured 21, and the analysis result presentation unit 14 presents the analysis results. The image acquisition unit 12, the movement amount acquisition unit 92, the calculation unit 13, and the analysis result presentation unit 14 may be configured in the same terminal as long as they have the functions, and examples of such terminals include a smartphone and a tablet.

[0092] 14, the image acquisition unit 12 captures an image of a person being measured 21 walking. The image acquisition unit 12 may be any device capable of acquiring images, such as a general web camera, an RGB-D camera (depth camera), a smartphone, an industrial camera, a video camera, etc. The images also include moving images that can be decomposed into images for each frame.

[0093] The image acquiring unit 12 includes a movement amount acquiring unit 92 that acquires movement amount data 93. In the embodiment of Fig. 14, the image acquiring unit 12 is mounted on a moving body 141 that includes the movement amount acquiring unit 92. In this case, the moving body 141 is moved by a measurer 142 such as a physical therapist so that the person 21 can be measured.

[0094] Another configuration of the image acquisition unit 12 and the movement amount acquisition unit 92 is shown in a derived example in FIG.

[0095] 15(a), a method of moving a smartphone or the like is used, in which a terminal 151 such as a smartphone or tablet is used, the image acquisition unit 12 is a camera or the like included in the terminal 151, and self-position estimation or pedestrian autonomous navigation positioning can be used for the movement amount acquisition unit 92. A person measuring 142 such as a physical therapist holding the terminal 151 measures the person being measured 21, and image data 15 and movement amount data 93 can be acquired. This allows gait analysis to be performed.

[0096] 15(b), the image acquisition unit 12 is mounted on a moving body 141 equipped with a movement amount acquisition unit 92. The moving body 141 moves automatically by a driving means 152 according to the relative distance to the subject 21, and gait analysis can be performed.

[0097] 15(c), in which the subject pushes a moving object, the image acquisition unit 12 is mounted on a moving object 141 equipped with a movement amount acquisition unit 92. The subject 21 pushes the moving object 141 using a bar 153, thereby enabling gait analysis.

[0098] (2. Analysis Processing of Gait Analysis System) In this embodiment, the stride 17 is estimated using the stride curve 74 in the same manner as in the second embodiment. The stride curve 74 has been described in the first embodiment, so a description thereof will be omitted. As in the second embodiment, the image acquisition unit 12 moves in this embodiment. Therefore, it is not possible to estimate the stride curve 74 only using the skeleton information 16 from the posture estimation algorithm 13a.

[0099] Therefore, skeletal information 16 of the walking subject 21 and movement amount data 93 acquired by the movement amount acquisition unit 92 are integrated to estimate the stride length curve 74. The stride length curve 74 is estimated using a machine learning method in which position information 111 of the ankle key points 71, 72 of the subject 21 and the movement amount data 93 are input, and the true value of the stride length curve 74 measured by the measuring device is used as training data. The rest of the process is the same as in Example 2, so it will be omitted.

[0100] The operation and effects of the gait analysis system 11 configured as above will be described below.

[0101] Skeleton information 16 of the walking subject 21 and movement amount data 93 acquired by the movement amount acquisition unit 92 are integrated, and a stride length curve 74 is estimated by multimodal machine learning. This makes it possible to estimate the stride length 17 even if the image data 15 is from a moving image acquisition unit 12. This makes it possible to estimate the stride length 17 if the movement amount data 93 can be acquired, and makes it possible to estimate the stride length 17 on a moving body 141 equipped with the image acquisition unit 12 or on a terminal 151 such as a smartphone or tablet.

[0102] A method has already been disclosed in which a terminal 151 such as a smartphone or tablet acquires movement amount data 93 using a self-position estimation function or the like, and integrates the movement amount data 93 with skeletal information 16 to perform a motion analysis of the subject 21. However, by using the stride length curve 74 of the present invention and processing thereof, the stride length 17 can be estimated with high accuracy.

[0103] Although the gait analysis system 11 according to the present invention has been described above by way of the above embodiment, the present invention is not limited thereto. Needless to say, modifications are possible within the scope of achieving the object of the present invention. In other words, the embodiment disclosed herein should be considered to be illustrative in all respects and not restrictive. In other words, the scope of the present invention is indicated by the claims, not by the above description, and is intended to include all modifications within the meaning and scope equivalent to the claims. [Industrial Applicability]

[0104] The present disclosure is applicable to gait analysis in rehabilitation, particularly assessment of stride length. [Explanation of symbols]

[0105] 11 Gait Analysis System 12 Image acquisition section 13 Arithmetic section 13a Pose Estimation Algorithm 13b Stride length estimation algorithm 13c Walking speed estimation algorithm 13d Other gait analysis algorithms 14 Analysis result presentation section 15 Image data 16 Skeletal information 17 Stride 18 Walking Speed 19 Left / right movement, joint angles, etc. 20 Gait analysis results 21 Subject 22 Stand 31 Key Points 41 Left Foot 42 Right Foot 43 Distance between left and right heels 51 Minimum interval during normal walking (at contact) 52 Section where feet are not on the ground 53 Minimum distance in non-healthy walking (at contact) 54 Flat section of time progression 55 Distance before and after the section 61 Position information of the left ankle in the direction of travel 62 Extraction Section 71 Left ankle key point 72 Right ankle key point 73 Distance between left and right 74 Stride Curve 75 Differential value of stride length curve 76 Between the heel contact areas 77 Second Threshold 91 Walking car 92 Movement amount acquisition section 93 Movement Data 101 User 111 Ankle key point location information 112 Multi-layer perceptron (MLP) 121 Retroreflective marker 122 3D position measuring machine 123 Infrared Camera 124 Infrared emitter 141 Mobile 142 Measurer 151 Devices such as smartphones and tablets 152 Driving means 153 Bar

Claims

1. an image acquisition unit for acquiring an image of a person to be measured who is walking; a calculation unit having a posture estimation algorithm for estimating skeletal information of the subject from the image and a stride estimation algorithm for estimating a stride; In a gait analysis system comprising: The stride estimation algorithm comprises: time-series position information of the leg of the subject in the moving direction is input; extracting a section in which the speed of a specific part of the left and right legs in a traveling direction is equal to or less than a first threshold; obtaining statistics for said intervals; A step of estimating a stride length from information before and after the time-series statistics; A gait analysis system comprising:

2. an image acquisition unit for acquiring an image of a person to be measured who is walking; a calculation unit having a posture estimation algorithm for estimating skeletal information of the subject from the image and a stride estimation algorithm for estimating a stride; In a gait analysis system comprising: The stride estimation algorithm comprises: At least the leg position information of the subject is input, A gait analysis system that uses a curve that shows the time change in the distance between the left and right legs of any given leg in the direction of travel.

3. A walking frame for assisting a user in walking; An image acquisition unit provided on the walking frame for acquiring an image of a user; A movement amount acquisition unit that acquires a movement amount of the walking vehicle; A calculation unit having a posture estimation algorithm for estimating skeletal information of the user in the image and a stride estimation algorithm for estimating a stride; In a gait analysis system comprising: The stride estimation algorithm comprises: At least leg position information of the subject; The movement amount acquired by the movement amount acquisition unit; With the input, A gait analysis system that uses a curve that shows the time change in the distance between the left and right legs of any given leg in the direction of travel.

4. The image acquisition unit and the movement amount acquisition unit are 4. The gait analysis system according to claim 3, which is detachable from an existing walking vehicle.

5. an image acquisition unit for acquiring an image of a person to be measured who is walking; a movement amount acquisition unit that acquires a movement amount of the image acquisition unit; A calculation unit having a posture estimation algorithm for estimating skeletal information of the user in the image and a stride estimation algorithm for estimating a stride; In a gait analysis system comprising: The stride estimation algorithm comprises: At least leg position information of the subject; The movement amount acquired by the movement amount acquisition unit; With the input, A gait analysis system that uses a curve that shows the time change in the distance between the left and right legs of any given leg in the direction of travel.

6. The stride estimation algorithm comprises: When a differential value of a curve showing a time transition of a distance between the left and right legs of any one of the legs in the traveling direction is equal to or less than a second threshold value, 6. A gait analysis system according to claim 2, wherein the stride is estimated from statistics of a curve showing a time transition of the distance between the left and right legs of the arbitrary one of the legs.

7. A walking aid to assist users in walking. An image acquisition unit provided on the walking frame for acquiring an image of a user; A movement amount acquisition unit that acquires a movement amount of the walking vehicle; a calculation unit that receives as input at least the image acquired from the image acquisition unit and the movement amount acquired from the movement amount acquisition unit; In the gait analysis system consisting of A learning method that uses supervised learning, using as training data a curve showing the time progression of the distance between the left and right legs of any given leg in the direction of travel.

8. an image acquisition unit for acquiring an image of a person to be measured who is walking; a movement amount acquisition unit that acquires a movement amount of the image acquisition unit; a calculation unit that receives as input at least the image acquired from the image acquisition unit and the movement amount acquired from the movement amount acquisition unit; In the gait analysis system consisting of A learning method that uses supervised learning, using as training data a curve showing the time progression of the distance between the left and right legs of any given leg in the direction of travel.

Images