Gait feature based frailty classification method for the elderly

Abstract

The application discloses a kind of based on gait feature's old person debilitation grade research method. Including the following steps: first, using deep learning network to extract human key feature points, avoid the problem such as traditional method feature point identification inaccuracy.Second, effectively combine the deep and shallow features in the contour segmentation network, so as to more finely segmented human contour.On this basis, feature extraction is combined with attention mechanism, highlight useful information to improve recognition rate.The method disclosed in the application links gait features with the debilitation level of the elderly while improving the human contour extraction method, and optimizes the feature extraction module. After extracting the features, classification experiments are conducted to verify the correlation between gait features and the debilitation of the elderly. The effective combination of gait features and the debilitation of the elderly is realized, which has a wide application prospect in expanding the effective protection population of comprehensive intervention for the elderly, improving the quality of medical services and the precise use of medical resources.

Description

Technical Field

[0001] This invention relates to a method for studying the frailty level of the elderly based on gait characteristics, belonging to the fields of computer vision and image information segmentation and classification, and geriatric medical assessment. Background Technology

[0002] Gait recognition, as the only biometric authentication technology capable of remote identification, is attracting increasing attention from society and research institutions. It has broad application prospects in social security, identity authentication, and video surveillance. Unlike other biometric authentication technologies, gait is an external, dynamic manifestation of a person and is closely related to time and space. Furthermore, compared to other biometric authentication technologies based on static features, gait recognition has significant advantages such as non-contact, non-invasiveness, and difficulty in concealment. Specifically, gait can be detected remotely (>5m) and monitored at low resolution, exhibiting strong robustness. Secondly, gait recognition can be performed without the observer's knowledge, requiring no cooperation from the observer. Thirdly, while other authentication technologies, such as facial recognition, can be concealed by covering the face, gait is much more difficult to hide.

[0003] Currently, researchers have proposed numerous algorithms in the field of gait recognition, which can be mainly divided into two types: model-based methods and appearance-based methods. Model-based methods involve modeling the human body, approximating it with a model, extracting parameters from the model as features, and finally performing classification and recognition. Commonly used models include elliptical models, skeleton models, and segmented models. Elliptical models use eight ellipses to represent different parts of the body, using features such as the angles and lengths between them; skeleton models use straight line segments to form the human skeleton, using their lengths and angles as features; and segmented models are formed by thickening the line segments on top of skeleton models. Model-based methods can effectively describe the relationships and changes of various body parts during human movement, and theoretically, their recognition performance can reach a high level when the model is accurate. However, their main drawbacks are the difficulty of modeling, the high computational complexity, and the need for parameters to be selected based on human experience.

[0004] Unlike model-based methods, appearance-based methods extract features from aspects such as changes in the shape of the human body contour before recognition. Since the acquired features are often high-dimensional, the processing needs to consider specific dimensionality reduction or feature extraction. Currently, there is no unified dimensionality reduction or feature extraction technique to accomplish this task. Summary of the Invention

[0005] This invention proposes a method for studying the frailty level of the elderly based on gait features. The purpose is to extract gait features by combining the gait profile and posture key points of the elderly, and to verify that the gait features of the elderly are strongly correlated with their frailty status after using deep learning.

[0006] The present invention achieves the above objectives through the following technical solutions:

[0007] (1) Self-built gait dataset SCU-Gait.

[0008] (2) Use the AlphaPose network to extract key feature points of the human body, calculate the pose features, and verify the correlation between pose features and attenuation level through two models: SVM and MLP.

[0009] (3) Based on the basic network Pose2Seg, the segmentation module is improved by introducing a dense connection module to replace it and optimize the segmentation network.

[0010] (4) Improve the feature extraction network by introducing an attention module to optimize global features. Combine key points and contour maps to extract features and use the features to classify attenuation levels. Attached Figure Description

[0011] Figure 1 This invention is a visualization diagram based on key human body features.

[0012] Figure 2 This is a diagram of the segmentation network improved based on the dense connection module in this invention.

[0013] Figure 3 This is a diagram of the feature extraction network structure based on the attention module of this invention.

[0014] Figure 4 This is a structural diagram of the dual-channel feature extraction and classification method of the present invention. Detailed Implementation

[0015] The present invention will be further described below with reference to the accompanying drawings:

[0016] The method and results for building the SCU-Gait dataset are as follows:

[0017] Gait video was captured using five cameras, with the angles of two adjacent cameras differing by a certain distance. To ensure the stability of the gait videos, a tripod was used to stabilize the camera. Furthermore, the camera's position during filming had specific requirements: the gait videos should include the subject's complete body parts, and the distance between the camera and the subject should not be too far or too close. There were age requirements for the subjects, as the frailty grading study primarily targets individuals over 65 years of age; therefore, the subjects in this dataset were no younger than 65. Secondly, subjects should have the ability to walk and move independently, ensuring they could independently complete the gait video recording. Next, to ensure subjects could complete the Fried phenotype assessment, their hearing and vision needed to be normal, and they should not have Parkinson's disease, myasthenia gravis, or neuromuscular disorders. Finally, it was necessary to ensure that subjects were informed about the study and participated voluntarily. After multiple batches of data collection, a total of 144 elderly samples have been processed so far. Each subject walked along a doctor-specified route six times in their most natural state, and five different perspectives were recorded each time. Therefore, each subject contained a total of [number missing] videos. A series of video sequences were created, with all participants' gait videos saved in MP4 format. Each gait video was 4-6 seconds long, and the participants walked a distance exceeding 3 meters. Each gait video contained at least two complete gait cycles. The frame rate for each gait video was 25 frames per second, and the resolution of each frame was [resolution missing]. Because some gait videos showed significant blurring or even corruption, the gait videos were filtered out. Currently, the SCU-Gait dataset contains a total of 4200 gait videos.

[0018] In addition to recording gait videos, it was also necessary to obtain the frailty level for each sample. Therefore, our partner developed a questionnaire based on the Fried frailty phenotype criteria. By analyzing the survey results, we obtained the frailty level for each participant. Among them, there were 97 samples in the normal category, 27 samples in the pre-frailty category, and only 22 samples in the frailty category.

[0019] Considering that human gait may change at any time, a frame-by-frame acquisition method was used to process gait videos. Because some gait videos did not contain human instances in the first 1-2 seconds, and some gait videos had angle changes caused by the subject turning at the end, the obtained gait images were screened a second time. Currently, a total of 430,000 gait images have been obtained, with an average of 111 gait images per gait video. A multi-view gait dataset SCU-Gait with corresponding frailty levels of the elderly has been successfully constructed.

[0020] The method for extracting human key points, calculating pose features, and verifying the correlation using SVM and MLP models is as follows:

[0021] AlphaPose is used to extract key human feature points. This network employs a top-down approach to detect the coordinates of key human feature points. Specifically, it first performs object detection and localization, and then detects key human feature points based on the detection bounding boxes. AlphaPose was chosen because even if the bounding boxes located during object detection have some errors, this network can still detect the coordinates of key human feature points relatively well.

[0022] In the SCU-Gait dataset, the coordinates of the hip and knee of human instances in all gait images were fully extracted, while the coordinates of other parts may be missing. Therefore, in this experiment, we consider using the difference in thigh joint angle change as a pose feature to classify the frailty level of elderly individuals. Compared with several other perspectives, the perspective is... The angles extracted from the gait images are the most accurate, so only the viewpoint is used. Experiments were conducted using the difference in thigh angle variation as a pose feature. To calculate the pose feature, the coordinates of the hip and left and right knees of the human instance in each gait sequence were first extracted from the JSON file corresponding to each sample. Then, Equation 2-5 was used to calculate the thigh angle value corresponding to each frame of the human instance's gait image. To effectively learn the pose feature of each sample, the angle sequence of each sample needs to be compressed into a feature in some way. This invention uses the difference between the maximum and minimum values ​​of the left and right thigh angles in a gait sequence of each sample as the pose feature, so the dimension of the pose feature for each sample is 2.

[0023] (1)

[0024] To demonstrate the correlation between the postural characteristics of the elderly and their frailty level, the difference in the thigh joint angle of human instances was used as the postural characteristic of the samples. Experiments were conducted using SVM and MLP to classify the frailty level of the elderly. The final classification accuracy is shown in Table 1.

[0025] Table 1. Classification results of posture features

[0026]

[0027] After observing Table 1, it can be found that there is a certain correlation between the difference in the angle of human body instances and the frailty level of the elderly.

[0028] The optimized segmentation network is constructed as follows:

[0029] To better address the issue of imprecise edge segmentation in the Pose2Seg model, this invention introduces a densely connected module. This module is optimized to perform the segmentation task and replaces the segmentation module in the base model. Furthermore, to enable this segmentation network to better segment gait images in the self-built dataset SCU-Gait, features from SCU-Gait are incorporated when creating the pose template. The improved network is named DPose2Seg.

[0030] A complete segmentation process in the DPose2Seg model is as follows: First, the image to be segmented is standardized. Then, the image to be segmented and its corresponding human key feature point coordinates are used as input to the feature extraction module to detect human instances. The feature extraction module used in the DPose2Seg network is the same as that in the Pose2Seg network, both being pre-trained FPN networks. The detection results are then affine aligned, and skeleton features are extracted using the affine-aligned key feature point coordinates. The affine-aligned detection results and skeleton features are concatenated and used as input to the segmentation module. The segmentation module in the DPose2Seg network is a fully convolutional densely connected module. The output of the segmentation module undergoes an inverse affine transformation to obtain the final segmentation result of the model.

[0031] Table 2 shows the segmentation results of each segmentation model on the COCO dataset. Mask R-CNN is a classic algorithm in instance segmentation research. Cascade R-CNN improves the object detection results by cascading multiple R-CNN structures and setting multiple detectors with different thresholds, thereby improving the instance segmentation results.

[0032] Table 2 COCO validation set results

[0033]

[0034] As can be seen from Table 2, compared with other networks, the instance segmentation method DPose2Seg designed in this invention has improved to a certain extent in all three evaluation metrics. Among them, the most important evaluation metric, mAP, is improved by 2% compared with the second place, while AP (median) and AP (large) are also improved compared with the second place.

[0035] The method for constructing feature maps and classification networks is as follows:

[0036] The experiments in this invention use the GaitSet network as the base network, and optimize its feature extraction module and loss function to design a new gait recognition network, Att-GaitSet. Att-GaitSet also uses a dimension of... The gait sequence images are used as network input. The frame-level feature extraction module in Att-GaitSet consists of residual units and max pooling operations. The input features need to be processed by three consecutive optimized frame-level feature extraction modules, and then the global features are optimized by an attention module. Next, the maximum value of each frame is extracted and concatenated as the sequence-level features of the samples. Finally, the extracted sequence-level features are processed by the HPM module to obtain the network output, and the parameters in Att-GaitSet are optimized using a multi-loss function fusion method.

[0037] The introduced attention mechanism first uses the global features of the input to learn different weights corresponding to each pixel, then uses the learned weights to optimize the frame-level features, and finally extracts the maximum value of the image features of each frame and concatenates them as the sequence-level features in the Att-GaitSet network.

[0038] Specifically, the original input features are first processed by three different statistical functions. , and The result is then concatenated with the original input and passed through a... The convolutional layers obtain weights corresponding to different pixels. These weights are then multiplied by the original input features to obtain the optimized frame-level features. Finally, the optimized frame-level features are processed by a statistical function. The maximum value of each gait image frame is obtained, as shown in Equation 4-4. Finally, the maximum values ​​of each gait sequence are concatenated to obtain the sequence-level features corresponding to each sample.

[0039] (2)

[0040] in Represents the original frame-level features, [ Indicates a cascading operation. Represent a The convolution operation.

[0041] A dual-channel deep convolutional neural network is designed based on the Att-GaitSet network. The inputs to the two convolutional neural networks are a sequence of gait contour maps and their corresponding pose image sequences, respectively, with both input images having a size of [insert size here]. First, the gait contour sequence image and pose sequence image of each sample are input into the corresponding neural network pathway. A frame-level feature extraction module, consisting of residual units and max pooling operations, extracts the depth features of each frame. Then, an attention module optimizes the global features, extracting the maximum value of each frame as the sequence-level feature for each sample. Finally, the sequence-level features from the two different pathways are fused and passed through an HPM module, where multiple loss functions are used to optimize the network model parameters.

[0042] Finally, the extracted features were used for classification experiments using Vgg16 and AlexNet networks, respectively.

[0043] Tables 3 and 4 respectively show the accuracy, precision, and recall of two different classification models for classifying frailty levels in the elderly using gait features extracted by a self-designed convolutional autoencoder. Observing Table 3, it can be seen that the final model achieves a relatively high accuracy, exceeding 90%, when using gait features to classify frailty levels in the elderly. This demonstrates that there are gait differences between frail elderly individuals and those in a normal state, making gait feature classification a feasible approach for elderly frailty levels.

[0044] Table 3. Accuracy of classification using gait features

[0045]

[0046] Table 4 shows the precision and recall for each of the normal, pre-weakness, and weak categories. Since the normal category has the most samples, both models are most effective at identifying normal samples, achieving good results in both precision and recall, with a relatively balanced performance. Both classification models show a decrease in precision and recall for the pre-weakness and weakness categories, especially for the pre-weakness data. This is because the pre-weakness stage is a transitional state; the gait characteristics of samples in this stage are less different from those in the normal and weakness categories, making it easier for the classification model to misclassify them as either normal or weak.

[0047] Table 4 shows the precision and recall rates for different categories using gait features for classification.

[0048]

[0049] The results in Tables 3 and 4 demonstrate a correlation between frailty status and gait characteristics in the elderly, suggesting further in-depth exploration is warranted. Combining gait characteristics with commonly used frailty assessment methods is also a feasible approach for judging the frailty status of the elderly. Since the classification model trained using multidimensional gait characteristics exhibits high accuracy and recall for the normal class, it can be used to detect elderly individuals in the normal class. If the classification model detects a sample as belonging to the abnormal class, a secondary assessment of the subject is necessary, incorporating other frailty assessment criteria. This does not mean the classification model trained in this paper is useless, as the proportion of elderly individuals in a normal state is the highest in reality. Therefore, combining gait characteristics with other methods for assessing frailty levels in the elderly can, to some extent, alleviate the burden on physicians.

Claims

1. A method for classifying frailty levels in the elderly based on gait characteristics, characterized in that... Includes the following steps: (1) Five cameras were used to shoot gait videos, and the angles of two adjacent cameras differed. After multiple batches of data collection, 144 processed elderly samples were obtained. (2) After extracting the key feature points of the human body using the AlphaPose network, the coordinates of the key feature points of each human body instance are saved in the corresponding JSON file. The thigh angle value corresponding to each frame of the gait image of the human body instance is calculated by using the extracted coordinates. Then, the difference between the maximum and minimum values ​​of the left and right thigh angles in a gait sequence of each sample is calculated, and the difference is used as the posture feature. The calculated posture features are then classified using the SVM model and the MLP multilayer perceptron, respectively. The accuracy of the posture feature classification is obtained through binary and tri-classification experiments, which proves the correlation between human posture features and the frailty level of the elderly. (3) The dense connection module was modified into a fully convolutional network to replace the segmentation module in Pose2Seg. Based on this, DPose2Seg was designed. Specifically, the segmentation module extracts features in a dense connection manner, and an upsampling operation is added at the end of the dense connection module so that the resolution of the features can be restored to the size of the original input to complete the segmentation task. Adding an upsampling path to every layer in the densely connected module would lead to excessive model complexity and computational overhead. Therefore, upsampling is only performed on the feature map of the last convolutional layer in each densely connected block. In addition, to mitigate the information loss caused by pooling operations in the densely connected module, skip connections are introduced between the upsampling stage and the corresponding downsampling stage of the module. The purpose of this is to combine deep and shallow features in the network to recover features more precisely and effectively. (4) To address the shortcomings of the GaitSet network in extracting sequence-level features without considering global feature information, an attention module is proposed to optimize the sequence-level feature extraction module in the basic network. First, different weights corresponding to each pixel are learned using the input global features. Then, the learned weights are used to optimize the frame-level features. Finally, the maximum value of each frame image feature is extracted and concatenated as the sequence-level features in the Att-GaitSet network. Specifically, the original input features are first processed by three different statistical functions. , and The result is concatenated with the original input and then processed through a... The convolutional layers obtain weights corresponding to different pixels. These weights are then multiplied by the original input features to obtain the optimized frame-level features. Finally, the optimized frame-level features are processed by a statistical function. The maximum value of each gait image frame is obtained; finally, the maximum values ​​of each gait sequence are concatenated to obtain the sequence-level features corresponding to each sample. in Represents the original frame-level features, [ Indicates a cascading operation. Represent a The convolution operation.

Images