A multi-angle combined gait feature extraction method and system based on bone coordinate data

Abstract

The application provides a multi-angle combined gait feature extraction method and system based on skeleton coordinate data. The steps of the method comprise the following: using a skeleton coordinate extraction device to extract human skeleton coordinate points; performing denoising processing based on the human skeleton coordinate points to reduce original noise; performing feature extraction on the human skeleton coordinate point sequence after denoising processing, and using multi-angle to extract combined features. The application aims at gait movement, extracts gait key features through multiple angles, and thus improves the classification task ability of gait movement.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer application, and particularly relates to a multi-angle combined gait feature extraction method and system based on bone coordinate data. BACKGROUND

[0002] Gait is a kind of motion form widely used in daily life, and it is one of the basic motions of human body, which can reflect the health status of human body.Gait motion can be modeled in a variety of ways, such as vision using color cameras (Reference Nieto-Hidalgo M, Ferrández-Pastor F J, Valdivieso-Sarabia R J, et al. A vision based proposal for classification of normal and abnormal gait using RGB camera [J]. Journal of biomedical informatics, 2016, 63: 82-89.), using IMUs (Reference Seel T, Raisch J, Schauer T. IMU-based joint angle measurement for gait analysis [J]. Sensors, 2014, 14(4): 6891-6909.), vision using depth cameras (Reference Stone E, Skubic M. Evaluation of an inexpensive depth camera for in-home gait assessment [J]. Journal of Ambient Intelligence and Smart Environments, 2011, 3(4): 349-361.), using optical devices (Reference Mirek E, Rudzińska M, Szczudlik A. The assessment of gait disorders in patients with Parkinson's disease using the three-dimensional motion analysis system Vicon [J]. Neurologia i neurochirurgia polska, 2007, 41(2): 128-133.), using wireless signals (Reference Hsu C Y, Liu Y, Kabelac Z, et al. Extracting gait velocity and stride length from surrounding radio signals [C] / / Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 2017: 2116-2126.).

[0003] The final modeling of gait motion obtains user's bone data coordinate points, and based on the bone data coordinate points, there are many works, which can assist in the diagnosis of Parkinson's disease (reference literature Rocha A P, Choupina H, Fernandes J M, et al. Parkinson's disease assessment based on gait analysis using an innovative RGB-D camera system [C] / / 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 2014: 3126-3129.), which can be used for human identity recognition (reference literature Kwon B, Lee S. Human skeleton data augmentation for person identification over deep neural network [J]. Applied Sciences, 2020, 10(14): 4849.), and which can be used for gait parameter calculation (reference literature Gabel M, Gilad-Bachrach R, Renshaw E, et al. Full body gait analysis with Kinect [C] / / 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 2012: 1964-1967.).

[0004] However, in order to make the algorithm based on the skeleton point of gait motion more accurate on various tasks such as prediction and classification, it is necessary to extract the gait features of the skeleton point. Therefore, a good gait feature extraction method is very important, and a good gait feature extraction method can make the classification tasks of gait motion more accurate, and has important significance for gait motion. SUMMARY

[0005] The purpose of the present application is to provide a multi-angle combined gait feature extraction technology based on bone coordinate data, which is aimed at gait motion, extracts gait key features through multiple angles, so as to realize the improvement of the classification task ability of gait motion.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] A multi-angle combined gait feature extraction method based on skeleton coordinate data, the steps of which comprise:

[0008] extracting human skeleton coordinate points using a skeleton coordinate extraction device;

[0009] performing denoising processing based on the human skeleton coordinate points;

[0010] combining the denoised human skeleton coordinate point sequence using multi-angle for combined feature extraction.

[0011] Further, the human skeleton coordinate points are extracted using a skeleton coordinate extraction device, which means that a visual device, a sensor device or an optical signal device is used to collect skeleton coordinate points during human gait movement.

[0012] Further, the denoising processing based on the human skeleton coordinate points means that the original coordinate data is processed using mean filtering, Gaussian filtering and other denoising methods to reduce noise.

[0013] Further, the combined feature extraction using multi-angle means that features are extracted from the time angle, the space angle, the motion angle and the histogram angle, and then the features are combined.

[0014] Further, the time angle feature extraction means joint speed feature: for human movement, joint speed feature is one of the most basic features, since the joints are constantly active during movement, joint speed can intuitively reflect the movement of the joint. Therefore, the joint speed during gait is used as the time angle feature, including the three-dimensional speed of nine joint nodes: waist, left thigh, right thigh, left knee, right knee, left ankle, right ankle, left toe and right toe There are 27 speed features, and for each joint node, the i-th frame speed feature is:

[0015]

[0016] Where x i+2 , y i+2 , z i+2 represent the three-axis coordinates of the last two frames of the i-th frame, x i-2 , y i-2 , z i-2 represent the three-axis coordinates of the first two frames of the i-th frame, and Δt represents the time of five frames. The speed feature of each joint node is calculated according to the formula.

[0017] Furthermore, the extraction of spatial features refers to the spatial distance between the device and the person: since gait movement is directed towards the data acquisition device, it is subject to certain limitations in terms of movement distance and range. Simultaneously, gait movement exhibits certain spatial regularities, thus its spatial features can be extracted. This invention uses the straight-line distance between the person and the data acquisition device as a spatial feature:

[0018] d i =||Pelvis.x i Pelvis.y i Pelvis.z i ||2

[0019] Among them, Pelvis.x i Pelvis.y i Pelvis.z i These represent the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the waist, respectively.

[0020] Furthermore, feature extraction of movement angles refers to the combined features of gait movement angles. These features can capture the characteristics of the human body during movement, enabling better classification of specific gait cycles. These features include:

[0021] (1) The distance between the knees, that is, the distance between the left and right knees, can be used to detect gait cycle movements:

[0022]

[0023] Among them, Dis_Knee i This indicates the distance between the knees, KneeLeft.x i KneeLeft.y i KneeLeft.z i KneeRight.x represents the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the left knee, respectively. i KneeRight.y i KneeRight.z i These represent the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the right knee, respectively.

[0024] (2) The distance between the waist and the toes can be used to detect gait cycle movements:

[0025]

[0026]

[0027] Among them, Dis_LeftFoot i Dis_RightFoot represents the distance from the left toe to the waist. i This indicates the distance from the right toe to the waist, FootLeft.x i FootLeft.z i This represents the x-axis coordinate and z-axis coordinate of the left toe, FootRight. i FootRight.z i This represents the x-axis coordinate and z-axis coordinate of the right toe.

[0028] (3) Hip rotation angle: The hip rotation angle can be used as a feature to detect whether a body rotation or turning motion has occurred.

[0029] θ i =acos(v1·v2 / ||v1||||v2||)

[0030] Where v1 = (HipLeft.x i -HipRight.x i HipLeft.z i -HipRight.z i ), where v2 is the X-axis. HipLeft.x i HipLeft.z i HipRight.x represents the x-axis coordinate and z-axis coordinate of the left thigh. i HipRight.z i This represents the x-axis coordinate and z-axis coordinate of the right thigh.

[0031] Furthermore, histogram-based feature extraction aims to mitigate the impact of outliers and enhance feature robustness. This invention proposes a local distance histogram method to construct binned features. The histogram feature design is as follows:

[0032] 1) Divide the walking distance into bins at 0.05-meter intervals; where a bin refers to an interval used for counting.

[0033] 2) Count the number of frames in which the ankle coordinates of the left and right feet fall within each bucket within 20 frames, and use this as a histogram;

[0034] 3) Find the mode of the histogram count and use it as the histogram feature value;

[0035] 4) Return the histogram feature values ​​for the left and right feet.

[0036] Histogram features can minimize the impact of outliers generated by the acquisition device on the feature vector, thereby increasing the robustness of the feature.

[0037] A gait feature extraction system based on skeletal coordinate data using angle combination employing the above method includes:

[0038] The skeletal data acquisition module is used to extract human skeletal coordinate points using a skeletal coordinate extraction device.

[0039] The noise reduction module is used to perform noise reduction based on human skeleton coordinate points;

[0040] The feature extraction module is used to extract features from the denoised human skeleton coordinate point sequence using a combination of multiple angles.

[0041] This invention addresses the use of human skeletal point coordinate acquisition devices to extract gait motion features. These features can be used to classify gait motion states and perform other classification tasks based on gait motion, thereby improving classification accuracy.

[0042] Compared with existing technologies, the beneficial effects of this invention are: this invention extracts gait motion features from multiple perspectives, taking into account both the general features of spatiotemporal motion and the specific features of human motion; in particular, we innovatively propose a distance-based histogram feature, which not only effectively solves the data jump problem but also makes the classification model more robust; our gait feature extraction technology combines features from multiple perspectives into a feature vector, which can make gait motion-based classification more accurate and robust. Attached Figure Description

[0043] Figure 1 This is a flowchart of a gait feature extraction technique based on skeletal coordinate data with multiple angle combinations.

[0044] Figure 2 This is a schematic diagram of gait motion acquisition according to the present invention.

[0045] Figure 3 This is a schematic diagram of the coordinate axes of human skeletal points.

[0046] Figure 4 This is a schematic diagram of time characteristics.

[0047] Figure 5 This is a schematic diagram of spatial features.

[0048] Figure 6 This is a schematic diagram of motion combination characteristics.

[0049] Figure 7 This is a schematic diagram of histogram features. Detailed Implementation

[0050] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings.

[0051] This embodiment provides a multi-angle combination method for gait feature extraction based on skeletal coordinate data, such as... Figure 1 As shown, the steps are as follows:

[0052] 1) Turn on the human skeleton point acquisition device;

[0053] 2) Wait for the user to perform gait movements, and then have the user perform gait movements within the range of the data acquisition device, such as... Figure 2 As shown;

[0054] 3) Obtain the sequence of skeletal coordinate points during the user's gait movement after the user's gait movement ends;

[0055] 4) Denoise the skeletal coordinate point sequence.

[0056] 5) Extract gait features from Google coordinates from multiple angles of the denoised skeletal coordinate point sequence.

[0057] 6) Combine the feature extraction results from each angle to form a combined feature vector, which can be used as input for other classification models.

[0058] In step 5), the multi-angle gait feature extraction mainly includes:

[0059] ① Temporal characteristics, such as Figure 4 The image shows joint velocity characteristics: using joint velocities during gait as time-dependent features, including the velocities in the three-dimensional directions of nine joints: waist, left thigh, right thigh, left knee, right knee, left ankle, right ankle, left toe, and right toe. There are a total of 27 velocity features. For each keypoint, its velocity feature in the i-th frame is:

[0060]

[0061] Where, x i+2 y i+2 , z i+2 This represents the three-axis coordinates of the two frames following the i-th frame, x. i-2 y i-2 , z i-2 Let Δt represent the three-axis coordinates of the two frames preceding the i-th frame, and Δt represent the time span of five frames. The velocity characteristics of each joint are calculated using this formula.

[0062] ② Spatial characteristics, such as Figure 5The figure shows the spatial distance between the device and the person: since gait movement is directed towards the data acquisition device, there are certain limitations on the movement distance and range. At the same time, gait movement has certain spatial regularities, therefore, spatial features can be extracted. This invention uses the straight-line distance between the person and the data acquisition device as a spatial feature:

[0063] d i =||Pelvis.x i Pelvis.y i Pelvis.z i ||2

[0064] Among them, Pelvis.x i Pelvis.y i Pelvis.z i These represent the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the waist, respectively.

[0065] ③ Motion characteristics, such as Figure 6 As shown: These refer to the combined features of gait movement angles. These features can capture the characteristics of the human body during movement, enabling better classification of specific gait cycles. These features include:

[0066] (i) The distance between the knees, i.e., the distance between the left and right knees, can be used to detect gait cycle movements:

[0067]

[0068] Among them, Dis_Knee i This indicates the distance between the knees, KneeLeft.x i KneeLeft.y i KneeLeft.z i KneeRight.x represents the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the left knee, respectively. i KneeRight.y i KneeRight.z i These represent the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the right knee, respectively.

[0069] (ii) The distance between the waist (Pelvis) and the toes (Foot) can be used to detect gait cycle movements:

[0070]

[0071]

[0072] Among them, Dis_LeftFoot i Dis_RightFoot represents the distance from the left toe to the waist. i This indicates the distance from the right toe to the waist, FootLeft.x i FootLeft.z i This represents the x-axis coordinate and z-axis coordinate of the left toe, FootRight. i FootRight.z i This represents the x-axis coordinate and z-axis coordinate of the right toe.

[0073] (iii) Hip rotation angle: The hip rotation angle can be used as a feature to detect whether a body rotation or turning motion has occurred.

[0074] θ i =acos(v1·v2 / ||v1||||v2||)

[0075] Where v1 = (HipLeft.x i -HipRight.x i HipLeft.z i -HipRight.z i ), where v2 is the X-axis. HipLeft.x i HipLeft.z i HipRight.x represents the x-axis coordinate and z-axis coordinate of the left thigh. i HipRight.z i This represents the x-axis coordinate and z-axis coordinate of the right thigh. A diagram of the coordinate axes is shown below. Figure 3 As shown.

[0076] ④ Histogram features, such as Figure 7 As shown: The design method for histogram features is as follows:

[0077] 1) Divide the walking distance into bins (intervals, used for counting) at intervals of 0.05 meters.

[0078] 2) Count the number of frames in which the ankle coordinates of the left and right feet fall within each bucket within 20 frames, and use this as a histogram.

[0079] 3) Find the mode of the histogram count and use it as the histogram feature value.

[0080] 4) Return the histogram feature values ​​for the left and right feet.

[0081] Based on the same inventive concept, another embodiment of the present invention provides a gait feature extraction system based on skeletal coordinate data using angle combinations employing the above-described method, comprising:

[0082] The skeletal data acquisition module is used to extract human skeletal coordinate points using a skeletal coordinate extraction device.

[0083] The noise reduction module is used to perform noise reduction based on human skeleton coordinate points;

[0084] The feature extraction module is used to extract features from the denoised human skeleton coordinate point sequence using a combination of multiple angles.

[0085] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smartphone, etc.) including a memory and a processor, wherein the memory stores a computer program configured to be executed by the processor, and the computer program includes instructions for performing the steps of the method of the present invention.

[0086] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk), which stores a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0087] The method of the present invention has been described in detail above through formal expressions and embodiments, but the specific implementation of the present invention is not limited thereto. Those skilled in the art can make various obvious changes and modifications without departing from the spirit and principles of the method described in the present invention. The scope of protection of the present invention should be determined by the claims.

Claims

1. A method for extracting gait features based on skeletal coordinate data using a multi-angle combination, characterized in that, Includes the following steps: Human skeleton coordinate points are extracted using a skeletal coordinate extraction device; Denoising is performed based on human skeletal coordinate points; The denoised human skeleton coordinate point sequence is used to extract features by combining multiple angles; The use of multi-angle combined feature extraction refers to extracting features from the time angle, spatial angle, motion angle, and histogram angle to obtain time features, spatial features, motion features, and histogram features, and then combining the features; The histogram features are obtained by statistically analyzing local distances during gait, including the following steps: 1) Divide the walking distance into bins at 0.05-meter intervals; 2) Count the number of frames in which the ankle coordinates of the left and right feet fall within each bucket within 20 frames, and use this as a histogram; 3) Find the mode of the histogram count and use it as the histogram feature value; 4) Return the histogram feature values ​​for the left and right feet.

2. The method according to claim 1, characterized in that, The time characteristics include the joint velocities during the movement of human skeletal points, including the velocities in three dimensions of nine joint points: waist, left thigh, right thigh, left knee, right knee, left ankle, right ankle, left toe, and right toe.

3. The method according to claim 1, characterized in that, The spatial characteristics include the spatial distance between the equipment and the person.

4. The method according to claim 1, characterized in that, The movement characteristics include the distance between the knees, the distance between the waist and the toes, and the angle of hip rotation.

5. The method according to claim 4, characterized in that, The distance between the knees is the three-dimensional distance between the left and right knees; the distance between the waist and the toes is the distance between the waist and the toes in two dimensions, where the two dimensions refer to the x-axis and the z-axis; the angle of hip rotation is the rotation angle between the line connecting the left and right hips and the x-axis.

6. A gait feature extraction system based on skeletal coordinate data, employing an angle combination method according to any one of claims 1 to 5, characterized in that, include: The skeletal data acquisition module is used to extract human skeletal coordinate points using a skeletal coordinate extraction device. The noise reduction module is used to perform noise reduction based on human skeleton coordinate points; The feature extraction module is used to extract features from the denoised human skeleton coordinate point sequence using a combination of multiple angles.

7. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method described in any one of claims 1 to 5.

Images