A method for recognizing gait signals of Parkinson's patients

Abstract

The present invention discloses a method for identifying gait signals of Parkinson's patients, comprising: obtaining spatial features of gait signal data using a capsule network CapsNet; obtaining the characteristics of the intrinsic time correlation of gait signal data through a long short-term memory network LSTM; respectively inputting the spatial features and the characteristics of the intrinsic time correlation of the gait signal data into a softmax function, performing weighted summation on the obtained results, and obtaining a final gait classification. Combining the two network structures, an auxiliary diagnosis model of Parkinson's disease with strong generalization ability can be trained, which can well identify abnormal gaits of Parkinson's patients, such as freezing gait, difficulty in turning, unstable posture, and panic gait, etc., thereby improving the accuracy of abnormal gait identification of Parkinson's patients and reducing misdiagnosis caused by subjective reasons of doctors.

Description

Technical Field

[0001] The present invention relates to the field of intelligent medical technology, and in particular to a method for recognizing gait signals of Parkinson's patients. Background Art

[0002] Parkinson's disease (PD) is considered to be the second most common age-related neurodegenerative disease after Alzheimer's disease. It is a degenerative disease of the nervous system in middle-aged and elderly people with a variety of clinical symptoms. The most common motor symptom is gait disorder. Gait freezing in gait disorder is mainly manifested as short-term difficulty in walking and taking small steps, which is more obvious when standing up or sitting down. This symptom is more common in the middle and late stages of the disease, and as the patient ages, gait freezing will become more and more serious, leading to falls, inability to walk, etc., interfering with the patient's daily activities and seriously reducing the patient's quality of life. At present, its diagnosis and efficacy evaluation mostly rely on the clinical experience and subjective judgment of medical staff, lacking objective and quantitative evaluation criteria. Due to the subjectivity of the diagnostic results, it is easy to cause misdiagnosis and affect the patient's treatment time.

[0003] At present, in the study of auxiliary diagnosis algorithms for Parkinson's disease based on gait analysis, most researchers use different gait data collection methods to extract the gait characteristics of the subjects, train classifiers or deep learning models, and finally input the subject's data into the trained classifier or model to diagnose whether the subject has PD. For example, Reginya et al. used a smartphone with an accelerometer and gyroscope to detect Parkinson's disease (PD) patients, analyzed the amplitude and spectral parameters of the acceleration signal and rotation speed to find out the differences between PD patients and non-PD patients. Guo Rui et al. extracted the human skeleton sequence from the video, constructed the spatiotemporal graph of joints and bones, and then evaluated whether the subject was a PD patient through a two-stream spatiotemporal graph convolutional network. Because the data collected by the above method has low precision, insufficient data type, and may be easily affected by external factors such as the external environment, it is not conducive to the training of the classifier or model, resulting in low accuracy of the final classification result. Summary of the invention

[0004] The purpose of the present invention is to provide a method for identifying gait signals of Parkinson's patients, which greatly improves the accuracy of PD diagnosis through the spatial characteristics of gait signals, saves medical staff's consultation time, and provides favorable assistance for the diagnosis of Parkinson's disease.

[0005] To achieve the above-mentioned purpose, the present application proposes a method for recognizing gait signals of Parkinson's patients, including: inputting the pre-processed gait data into the CapsNet Channel network and the LSTM Channel network respectively for joint training. For the CapsNet Channel network, the pre-processed gait data is sent to the convolution layer to extract the gait features, and then the gait features are sent to the CBAM attention module to generate a gait feature map, enhance the useful features of the gait data, and reduce the useless features. The gait feature map is sent to the primary capsule layer, and the gait feature map is vectorized to obtain the underlying gait feature U; through the dynamic routing algorithm, communicate with the intermediate capsule layer, denoise the gait feature information through the intermediate capsule layer, and compress the vector neurons; finally, iterate the dynamic routing algorithm again, and output the final result by connecting the primary capsule layer, the intermediate capsule layer and the digital capsule layer; for the LSTM Channel network, the gait data is converted into a one-dimensional vector as input, and the useless gait data information is first discarded through the forget gate to obtain the forget gate f t , and then the input gate determines which new gait information is retained in the cell state, and the input gate i is obtained. t , candidate memory cells and memory unit C t Finally, the output gate determines the information h of the LSTM network output data t The channel is composed of two layers of LSTM structures stacked linearly, and a dropout structure is added to prevent overfitting.

[0006] Furthermore, the gait data is preprocessed in the following manner: by oversampling the gait data, the problem of unbalanced data samples is solved; and a Butterworth bandpass filter is used to filter out noise and interference in the gait data, thereby correcting the delay in the time series data.

[0007] Furthermore, the CBAM attention module includes a channel attention module and a spatial attention module. The gait features extracted by the convolution layer are first sent to the channel attention module to obtain the feature F, and then the feature F is sent to the spatial attention module to generate a gait feature map. This enhances the performance of gait features in space and channels, highlights the important features of gait signals, and reduces other useless information.

[0008] Compared with the prior art, the above technical solution adopted by the present invention has the following advantages:

[0009] 1. The present invention solves the problems of unbalanced gait data samples, noise and interference, and is ready for training a Parkinson's disease auxiliary diagnosis system with strong generalization ability.

[0010] 2. Effectively extract the spatiotemporal features of gait, enhance the important features of gait signals, and reduce other useless information.

[0011] 3. The capsule network CapsNet was introduced, and the CapsNet-LSTM dual-channel hybrid network model was proposed to improve the accuracy of gait classification for Parkinson's disease. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is the channel attention module diagram;

[0013] Figure 2 This is a diagram of the spatial attention model;

[0014] Figure 3 This is a dual-channel hybrid model diagram of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application, that is, the embodiments described are only part of the embodiments of the present application, not all of the embodiments.

[0016] Example 1

[0017] This embodiment provides a method for identifying gait signals of Parkinson's patients, which collects gait data based on a sensing pressure sensor, pre-processes the collected data, and solves problems such as unbalanced gait data samples, missing data, and noise and interference in the gait data. Then a self-attention mechanism is embedded to cleverly extract key information from the gait data and ignore irrelevant information, thereby enhancing the important features of the gait signal. The Capsule Network (Capsule Network, CapsNet) is introduced to replace the traditional convolutional neural network. CapsNet has good feature representation capabilities and can maintain good performance when there are few gait data samples or unbalanced samples. For time series data such as gait signals, the capsule network can well reflect the spatial characteristics of gait signals, which can greatly improve the accuracy of PD diagnosis, save medical staff's consultation time, and provide favorable assistance for the diagnosis of Parkinson's disease. Its specific implementation method is:

[0018] Use capsule network CapsNet to obtain the spatial features of gait signal data;

[0019] Specifically, the capsule network CapsNet of the present invention includes a convolutional layer, a CBAM attention module, a primary capsule layer, an intermediate capsule layer, and a digital capsule layer. Compared with the original capsule network, the capsule network adds an intermediate capsule layer, reduces the dimension of the capsule first and then increases the dimension, improves the utilization rate of the capsule, and further classifies the gait features. The feature extraction and processing process of the capsule network is as follows:

[0020] S11: inputting the preprocessed gait data into a convolution layer, and extracting gait features through the convolution layer, wherein the convolution kernel size is 9x9 and activated by a ReLU function;

[0021] S12: sending the gait feature to the CBAM attention module to generate a gait feature map, and sending the gait feature map to the primary capsule layer;

[0022] Specifically, the CBAM attention module includes a channel attention module and a spatial attention module, such as Figure 1 As shown in FIG, the channel attention module focuses on meaningful gait features, inputs a HxWxC gait feature E, uses AvgPool and MaxPool to use different information respectively, obtains a 1x1xC channel description, and then sends it to a shared two-layer neural network. The two features are added and then passed through an activation function to obtain the weight coefficient. Finally, the weight coefficient is multiplied by the original gait feature E to obtain the scaled feature F; as shown in FIG. Figure 2 As shown in the figure, after the feature F is sent to the spatial attention module, the channel-based MaxPool and AvgPool operations are performed respectively. After the results are spliced ​​together, they pass through a convolution layer and are reduced to one channel. The spatial attention feature map is generated through the sigmoid function. Finally, the spatial attention feature map is multiplied with the input feature F to obtain the final gait feature map, which enhances the performance of gait features in space and channels.

[0023] S13: The feature map extracted by the primary capsule layer is converted into a vector capsule, and the underlying gait feature U is obtained by using the ReLU activation function, and then the underlying gait feature U is combined with the weight matrix W ij Multiply to get high-level features Right now Iterate the dynamic routing algorithm once, communicate with the intermediate capsule layer, and obtain the capsule after dimension upgrade; the intermediate capsule layer performs denoising on the gait feature information, and uses a squash function to compress the vector neurons to 0 to 1 to ensure that the direction of the vector remains unchanged;

[0024] S14: Continue to iterate the dynamic routing algorithm, the primary capsule layer and the intermediate capsule layer score and predict the high-level features through posture relationships, and selectively activate the digital capsule layer, which performs spatial feature prediction of gait signal data.

[0025] The intrinsic time correlation characteristics of gait signal data are obtained through the long short-term memory network LSTM;

[0026] Specifically, the present invention utilizes the autoregressive network architecture characteristics of the LSTM network, which has good dynamic learning of the intrinsic time correlation of time series data, so that it has excellent performance in learning the temporal dependency of long-sequence gait data. It is composed of two LSTM structures stacked linearly, and each layer has the same number of memory units. In order to prevent overfitting, a dropout structure is added to the LSTM network with an inactivation ratio of 0.5. Since the gait cycle of most participants is 1s, and the gait data is sampled every 10ms, the number of memory units in each layer is set to 100. Through the memory unit C t The information that has passed through the forget gate and the input gate is retained, so that LSTM can effectively transmit historical information with a long time interval, thereby obtaining the characteristics of the intrinsic time correlation of gait data. Its working process is as follows:

[0027] S21: Input the preprocessed gait data into the LSTM network and use X t The gait data sample at time t represents the gait data. When it passes through the cell unit of the LSTM network, the useless gait data information is discarded by the forget gate, and its output is:

[0028] f t =σ(W f ·[h t-1 , x t ]+b f )

[0029] Among them, W f is the weight matrix of the forget gate, h t-1 Indicates the output value of the LSTM network at the previous moment, b f is the bias term of the forget gate, and σ is the sigmoid function.

[0030] S22: The input gate determines which new gait information is retained in the cell state, that is, the output value h of the LSTM network at the previous moment t-1 And the gait data x t The updated gait information is determined by the operation of the input gate, specifically:

[0031] i t =σ(W i ·[h t-1 ,xt ]+b i )

[0032] Among them, W i refers to the weight, b i Refers to the bias value; then use h t-1 and x t Get new candidate memory units through a tanh layer

[0033]

[0034] Among them, W c refers to another weight, b c Refers to the bias term; then by the memory unit C t Indicates the cell update status of the LSTM network:

[0035]

[0036] Among them, C t-1 represents the state of the previous layer, f t represents the forget gate, i t Represents the input gate.

[0037] S23: The judgment condition O is obtained through a sigmoid layer of an output gate t , and then pass through a tanh layer to get a vector between -1 and 1, and finally use this vector and the judgment condition O t Multiply to get the gait data information h output by the LSTM network t for:

[0038] O t =σ(W o [h t-1 ,x t ]+b o )

[0039] h t =O t tanh(C t ).

[0040] The spatial features and the intrinsic time correlation characteristics of the gait signal data are respectively input into the softmax function, and the obtained results are weighted summed to obtain the final gait classification.

[0041] The preprocessed gait data is fed into two networks respectively to maximize the capture of valuable gait spatiotemporal features, such as Figure 3As shown, a dual-channel hybrid model can be designed. For time-series data such as gait signals, CapsNet can be used to effectively learn the spatiotemporal characteristics of gait data, and LSTM has a strong processing capability advantage for time-series data. Capsule networks and long short-term memory networks can be integrated to train a Parkinson's disease auxiliary diagnosis model with strong generalization ability, thereby improving the auxiliary diagnosis efficiency of Parkinson's disease.

[0042] The experimental hardware configuration of the present invention is CPU (Inter Core i9-10920x), 32GB memory, GPU (NVIDIARTX6000) video memory 24GB, operating system Window10, CUDA, Python3.6 and deep learning framework Pytorch. The basic information of the subjects of the gait dataset used is shown in Table 1.

[0043] Table 1 Basic information of subjects

[0044]

[0045] The above dataset consists of 3 PD gait sub-datasets contributed by 3 researchers (Ga, Ju and Si), which include gait information of 93 patients with idiopathic Parkinson's disease and 73 healthy controls (average age 66.3 years, 55% male). The samples were oversampled to solve the problem of data sample imbalance, and a Butterworth bandpass filter was used to remove noise and interference contained in the data.

[0046] It should be noted that: the present invention is aimed at the existing gait recognition methods, most of which use convolution blocks to obtain global features when extracting features, thereby ignoring the weighting of some important information and gait features that can reflect local information. In order to solve the above problems, the present invention introduces the CBAM attention mechanism, which combines the spatial and channel attention mechanism modules. The results output by the convolution layer are first passed through a channel attention module to obtain the weighted results, and then passed through a spatial attention module to finally weight the results. Channel attention and spatial attention respectively focus on the feature content and spatial position of the features in the gait data information, which greatly improves the feature extraction effect of the network. The channel attention module and the spatial attention module can be expressed as:

[0047]

[0048] Where F represents the input feature, W0 and W1 represent the two-layer parameters in the multi-layer perceptron model, and σ represents the sigmoid function.

[0049]

[0050] Where F represents the input feature, f 7×7 It indicates that the convolution kernel size of the network is 7x7.

[0051] The CapsNet-LSTM dual-channel hybrid model proposed in the present invention can maximize the extraction of temporal information of gait signals. CapsNet can maintain good performance when there are few data samples or unbalanced data samples, and can well represent the spatial characteristics of time series data; LSTM has excellent performance in learning the temporal dependency of long sequence data. By combining the two network structures, an auxiliary diagnosis model for Parkinson's disease with strong generalization ability can be trained, which can well identify the abnormal gait of Parkinson's patients, such as freezing gait, turning difficulty, unstable posture and panic gait, etc., thereby improving the accuracy of abnormal gait recognition of Parkinson's patients and reducing misdiagnosis caused by subjective reasons of doctors.

[0052] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims

1. A method for identifying gait signals of Parkinson's patients, characterized in that: include: The capsule network CapsNet is used to obtain the spatial features of gait signal data, specifically: S11: inputting the preprocessed gait data into a convolutional layer, and extracting gait features through the convolutional layer; S12: Send the gait feature to the CBAM attention module to generate a gait feature map, and send the gait feature map to the primary capsule layer; the CBAM attention module includes a channel attention module and a spatial attention module. The gait feature extracted by the convolution layer is first sent to the channel attention module to obtain the feature F, and then the feature F is sent to the spatial attention module to generate a gait feature map; the channel attention module inputs a HxWxC gait feature E, uses AvgPool and MaxPool to use different information respectively, obtains a 1x1xC channel description, and then sends A shared two-layer neural network is input, the two features obtained are added and then passed through an activation function to obtain a weight coefficient, and finally the weight coefficient is multiplied by the original gait feature E to obtain the scaled feature F; after the feature F is sent to the spatial attention module, the channel-based MaxPool and AvgPool operations are performed respectively, and the results are spliced ​​together, and then passed through a convolution layer to reduce the dimension to 1 channel, and a spatial attention feature map is generated through a sigmoid function, and finally the spatial attention feature map is multiplied with the input feature F to obtain the final generated gait feature map; S13: The feature map extracted by the primary capsule layer is converted into a vector capsule, and the underlying gait feature U is obtained by using the ReLU activation function, and then the underlying gait feature U is combined with the weight matrix W ij Multiply to get high-level features Right now Iterate the dynamic routing algorithm once, communicate with the intermediate capsule layer, and obtain the capsule after dimension upgrade; the intermediate capsule layer performs denoising on the gait feature information, and uses a squeezing function to compress the vector neurons to 0 to 1 to ensure that the direction of the vector remains unchanged; S14: Continue to iterate the dynamic routing algorithm, the primary capsule layer and the intermediate capsule layer score and predict the high-level features through posture relationships, and selectively activate the digital capsule layer, which performs spatial feature prediction of gait signal data. The capsule network CapsNet includes a convolutional layer, a CBAM attention module, a primary capsule layer, a mid-level capsule layer and a digital capsule layer; The intrinsic time correlation characteristics of gait signal data are obtained through the long short-term memory network LSTM; The spatial features and the intrinsic time correlation characteristics of the gait signal data are respectively input into the softmax function, and the obtained results are weighted summed to obtain the final gait classification; 2. A method for identifying gait signals of Parkinson's disease patients according to claim 1, characterized in that: The characteristics of the intrinsic time correlation of gait signal data are obtained through the long short-term memory network LSTM, specifically: S21: Input the preprocessed gait data into the LSTM network and use X t The gait data sample at time t represents the gait data. When it passes through the cell unit of the LSTM network, the useless gait data information is discarded by the forget gate; S22: The input gate determines which new gait information is retained in the cell state, that is, the output value h of the LSTM network at the previous moment t-1 And the gait data x t Determine and update gait information through the operation of the input gate; S23: The judgment condition O is obtained through a sigmoid layer of an output gate t , and then pass through a tanh layer to get a vector between -1 and 1, and finally use this vector and the judgment condition O t Multiply to get the gait data information h output by the LSTM network t .

3. A method for identifying gait signals of Parkinson's disease patients according to claim 2, characterized in that: Useless gait data information is discarded by the forget gate, and its output is: f t =σ(W f ·[h t-1 ,x t ]+b f ) Among them, W f is the weight matrix of the forget gate, h t-1 Indicates the output value of the LSTM network at the previous moment, b f is the bias term of the forget gate, and σ is the sigmoid function.

4. A method for identifying gait signals of Parkinson's disease patients according to claim 1, characterized in that: The output value h of the LSTM network at the previous moment t-1 And the gait data x t The updated gait information is determined by the operation of the input gate, specifically: i t =σ(W i ·[h t-1 ,x t ]+b i ) Among them, W i refers to the weight, b i Refers to the bias value; then use h t-1 and x t Get new candidate memory units through a tanh layer Among them, W c refers to another weight, b c Refers to the bias term; then by the memory unit C t Indicates the cell update status of the LSTM network: Among them, C t-1 represents the state of the previous layer, f t represents the forget gate, i t Represents the input gate.

5. A method for identifying gait signals of Parkinson's disease patients according to claim 1, characterized in that: The gait data information h output by the LSTM network t for: The t =σ(W o [h t-1 ,x t ]+b o ) h t =O f ·tanh(C t )。

Images