Abnormal gait judgment method and system based on multi-feature fusion

Abstract

The invention relates to an abnormal gait judgment method and system based on multi-feature fusion, and belongs to the technical field of electronic information. According to the method, a 3D skeleton sequence is acquired by means of a depth camera such as Kinect, weights are set for articulation points based on the change degree of Z-axis coordinates of the articulation points, weighted sum data of all the articulation points are calculated for each frame, gait cycles are divided based on the weighted sum data, and feature vectors are spliced according to the divided gait cycles to obtain a feature matrix used for model input. Compared with a traditional method, the method has the advantages that data collection does not depend on wearable equipment, collection is more convenient, meanwhile, model recognition is more accurate, the early diagnosis precision of the Parkinson's disease is expected to be improved, technical support is provided for early diagnosis and treatment of the disease, and application of the gait recognition technology in the medical field is promoted.

Description

TECHNICAL FIELD

[0001] The application relates to an abnormal gait judgment method and system based on multi-feature fusion, which is helpful for early diagnosis of Parkinson's disease and belongs to the electronic information technical field. BACKGROUND

[0002] Neurological diseases are often accompanied by motor symptoms such as abnormal gait, so the evaluation of abnormal gait of patients helps the early diagnosis of these diseases, and the traditional clinical evaluation which depends on rich experience is highly subjective and low in efficiency, and cannot meet the demand for early and accurate diagnosis of related diseases.

[0003] Gait abnormality detection excluding subjective factors is a key technology for disease diagnosis and treatment and rehabilitation, and is of great significance to health protection. Early research often collects data through inertial sensors and analyzes them in combination with neural networks, and domestic innovative models are also constantly emerging to optimize gait identification ability.

[0004] However, the existing research has obvious deficiencies: first, it relies on wearable devices to collect data, which is inconvenient to use and has low patient compliance, limiting the application scenarios; second, the algorithm is difficult to capture early subtle gait changes such as Parkinson's disease, leading to difficulties in identifying abnormal gait with early subtle gait changes, further leading to difficulties in improving the accuracy of early diagnosis of related diseases. SUMMARY

[0005] The purpose of the present application is to provide an abnormal gait judgment method and system based on multi-feature fusion to solve the problems of inconvenient gait data collection and difficulty in identifying subtle gait changes in the prior art.

[0006] To achieve the above-mentioned purpose, the scheme of the present application includes:

[0007] An abnormal gait judgment method based on multi-feature fusion of the present application includes the following steps:

[0008] 1) Collecting walking skeleton original data containing joint coordinates during walking of a subject;

[0009] 2) Calculating the standard deviation of the Z coordinates of each joint in all frames, taking the ratio of the standard deviation of any joint to the sum of the standard deviations of all joints as the weight of the joint, and calculating the weighted sum of the Z coordinates of each joint in any frame as the weighted sum data of the frame;

[0010] 3) Taking the local minimum value of the weighted sum data in the frame sequence as the boundary to divide the frame sequence into gait cycles;

[0011] 4) combining the feature vectors between two adjacent nodes in the same frame and the feature vectors between the same nodes in adjacent frames to obtain a combined feature vector, arranging all the combined feature vectors in a gait cycle to obtain a feature matrix of the cycle, and arranging the feature matrices of each gait cycle of a subject to obtain a feature matrix set;

[0012] 5) training a fully connected neural network by taking the feature matrix set of multiple subjects as a training set to obtain a classification model;

[0013] 6) using the classification model to evaluate whether the gait of the measured object is an abnormal gait.

[0014] Further, in step 1), the subjects include Parkinson patients of different grades and healthy people.

[0015] Further, in step 2), before calculating the standard deviation of the Z coordinates of each joint in all frames, an interpolation method based on a linear two-point equation is used to smooth the normalized original data.

[0016] Further, in step 2), the Z coordinate of a joint in a frame is the product of the Z coordinate of the corresponding joint in the frame and the confidence of the corresponding joint in the frame.

[0017] Further, in step 3), the gait cycle set G is:

[0018] G={[j1,j2],[j2,j3],…,[j L-1 ,j L ]}

[0019] where I min ={j1,j2,…,j L} is the local minimum sequence of the weighted sum data of each frame of the subject sorted by frame timestamp.

[0020] Further, in step 4), the feature vectors between two adjacent nodes in the same frame and the feature vectors between the same nodes in adjacent frames are spliced up and down to obtain a combined feature vector.

[0021] Further, in step 4), the transpose of all the combined feature vectors in a gait cycle is spliced up and down to obtain a feature matrix of the cycle.

[0022] Further, in step 5), the fully connected neural network includes a 3D convolution layer for feature extraction, a max-pooling layer for performing a pooling operation on the features extracted by the convolution layer to reduce the dimension of the data, an attention mechanism layer for performing a weighting process on the features after dimension reduction to enhance the attention to the key classification features, and a fully connected layer for feature transformation and classification decision on the output of the attention mechanism layer and the flattened features.

[0023] Further, in step 1), the walking skeleton original data is collected by using an Azure Kinect body sensor.

[0024] The application also provides an abnormal gait judgment system based on multi-feature fusion.

[0025] The application has the following beneficial effects:

[0026] The application provides an abnormal gait judgment method and system based on multi-feature fusion. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a flowchart of the abnormal gait judgment method based on multi-feature fusion of the application;

[0028] Fig. 2(a) is a training and verification loss result graph of the classification model recognition result of the application;

[0029] Fig. 2(b) is a training and verification accuracy result graph of the classification model recognition result of the application;

[0030] Fig. 2(c) is a receiver operating characteristic example result graph of the classification model recognition result of the application;

[0031] Fig. 2(d) is a precision, recall and F1 score result graph of the classification model recognition result of the application;

[0032] Fig. 3(a) is a training and verification loss result graph of the traditional model recognition result of the prior art;

[0033] Fig. 3(b) is a training and verification accuracy result graph of the traditional model recognition result of the prior art;

[0034] Fig. 3(c) is a receiver operating characteristic example result graph of the traditional model recognition result of the prior art;

[0035] Fig. 3(d) is a precision, recall and F1 score result graph of the traditional model recognition result of the prior art. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the present application more clear, specific, complete and detailed, the present application will be described below in conjunction with the drawings and examples.

[0037] In order to solve the problems of strong subjectivity, low detection efficiency and difficulty in capturing subtle gait changes of the existing early Parkinson's disease diagnosis method based on gait feature recognition, the present application provides an abnormal gait feature recognition method based on skeleton data, which can be applied to early Parkinson's disease diagnosis.

[0038] The present application takes early Parkinson's disease recognition as the research object, extracts human skeleton joint coordinate data collected by a Kinect camera, establishes a convolutional neural network model based on 3D skeleton coordinates to capture subtle gait change features of early Parkinson's disease, designs a self-attention skeleton convolution architecture, constructs a segmented information processing module, and establishes a local and global information collaborative analysis mechanism, so as to accurately recognize abnormal gait features and provide an efficient and objective technical solution for early diagnosis of Parkinson's disease.

[0039] Method implementation:

[0040] Figure 1 As shown in the figure, it is a flowchart of an abnormal gait judgment method based on multi-feature fusion of the present application, which uses a trained classification model to classify the input walking data to obtain the classification results of patients (i.e. evaluate abnormal gait) and normal people (i.e. evaluate normal gait). The establishment of the classification model includes the following steps:

[0041] Step 1: Collect the walking skeleton raw data of the subjects by using, for example, a Kinect (Microsoft's motion sensing camera), and record 32 joint coordinate points (for example, 32, which is not limited by the present application);

[0042] Step 2: Read the raw data, eliminate invalid frames, smooth and normalize the raw data, calculate the weighted sum of the joint points, and remove outliers;

[0043] Step 3: Smooth the weighted sum data, find the local minimum value point, and divide the gait cycle in combination with the threshold value and time interval;

[0044] Step 4: For a gait cycle, calculate the position change vector of adjacent joint points in the same frame, the position change vector of the same joint point in adjacent frames, splice them into a 6-dimensional feature vector, and then form a feature matrix with the feature vectors of a gait cycle; form a feature matrix set with the feature matrices of each gait cycle of a subject;

[0045] Step 5: Construct a fully connected neural network, take the feature matrix set of each subject as a data set, divide the data set, train the fully connected network with an optimizer and a loss function to obtain a classification model;

[0046] Step 6, evaluate the resulting classification model with the test set, calculate the precision, recall and other indicators.

[0047] When the classification model is used, the same method (steps 1-4) is used to obtain the feature matrix set formed by the feature matrix of each gait cycle from the walking skeleton original data of the measured object. The feature matrix set is input into the trained classification model, and the abnormal gait evaluation result that can assist in the diagnosis of Parkinson's disease is output.

[0048] As a specific embodiment, step 1 is to prepare data for model training, the main purpose is to collect three-dimensional skeleton node coordinate data of subjects during gait walking, subjects include different grades of Parkinson's disease patients (people with different degrees of abnormal gait) and healthy people.

[0049] In this embodiment, an Azure Kinect body camera is used to record the gait process of the subject, and each frame of data is obtained, wherein each frame of data contains 32 joint nodes of the subject, denoted as f={C0, C1,..., C31}, wherein Ci=(Xi, Yi, Zi, A) represents the coordinates of the ith joint node in three-dimensional space, Xi, Yi and Zi are the coordinates of the joint node on the X, Y and Z axes respectively, and A is the confidence degree represented by this data. The collected data will be used for subsequent gait feature analysis, and can be further used for pathological research of Parkinson's disease.

[0050] As a specific embodiment, the specific implementation steps of step 2 are as follows:

[0051] Step 2.1, select an interpolation method to smooth and normalize the original data, and fill in the missing data by interpolation. Based on the straight line two-point equation, its expression is represented as:

[0052]

[0053] Two known data points (x0, y0) and (x1, y1), and want to estimate the y value at x (x0 < x < x1). Substitute the timestamp of the missing frame into the equation to get the interpolated coordinates (x, y):

[0054]

[0055] Step 2.2, extract the joint node coordinates and confidence according to the interpolated frame data, represented as (num_frames, 32, 3), wherein num_frames is the number of valid frames after screening, 32 is the number of joint nodes, and 3 represents 3 coordinate dimensions of each joint node;

[0056] Step 2.3. Calculate the weighted sum data of each frame for the division of gait cycle, based on the extracted joint Z coordinates (coordinates in vertical direction, representative of the human body walking) and the confidence data.

[0057] First, calculate the adjusted Z coordinates adjusted_z ij , which are the product of the Z coordinates of a joint in a frame (i_framej_joint) and the corresponding confidence confidence ij for the joint in each frame, and then calculate the standard deviation std_dev j between frames for the adjusted Z coordinates of the joint. The standard deviation represents the degree of variation of the Z coordinates of the corresponding joint in each frame; a larger standard deviation indicates that the Z coordinates are more dispersed in each frame, i.e., the joint has a larger movement amplitude during walking; a smaller standard deviation indicates that the Z coordinates are less dispersed in each frame, i.e., the joint has a smaller movement amplitude during walking.

[0058] The expressions are formula (3) and formula (4), respectively:

[0059] adjusted_z ij = z ij × confidence ij (3)

[0060] std_dev j = σ (adjusted_z 1j , adjusted_z 2j , …, adjusted_z num_frames,j ) (4)

[0061] wherein z ij is the Z coordinate of the jth joint in the ith frame data, confidence ij is the confidence of the jth joint in the ith frame data (i = 1, 2, …, num_frames; j = 1, 2, …, 32), and σ represents a function for calculating the standard deviation.

[0062] For any joint, calculate the normalized weight of the joint j by formula (5), and set the weight value of the standard deviation of 0 to a very small value, e.g., 1e-8. This step sets the weight of the joint with a small standard deviation, i.e., a small movement amplitude during walking, to be small; and sets the weight of the joint with a large standard deviation, i.e., a large movement amplitude during walking, to be large.

[0063]

[0064] Finally, obtain the weighted sum of the ith frame data (denoted as Si in step 3.1). The expression is formula (6):

[0065]

[0066] As a specific embodiment, the specific implementation steps of step 3 are as follows:

[0067] Step 3.1, find the local minimum of the extracted weighted sum data, and let the smoothed weighted sum data sequence be S = [S1, S2, …, SN], where N is the length of the sequence, that is, the number of frames. The local minimum expression is formula (7):

[0068] I min = {i ∈ (2, 3, …, N-1) | S i-1 > S i ∧ S i+1 > S i} (7)

[0069] Step 3.2, divide the gait cycle by the local minimum sequence I min = {j1, j2, …, j L} obtained by step 3.1, where L = |I min |, the number of local minima, and the number of gait cycles is L-1. Let the gait cycle set be G, where each gait cycle g n can be expressed as: g n = [j n , j n+1 ]. That is, G = {[j1, j2], [j2, j3], …, [j L-1 , j L ]}.

[0070] As a specific embodiment, the specific implementation steps of step 4 are as follows:

[0071] Step 4.1, introduce the feature vectors of the same frame adjacent two nodes and the feature vectors of the same node adjacent frames. For the same frame f(j n ≤ f < j n ) in the nth gait cycle g n+1 = [j n , j n+1 ], the feature vectors of the adjacent two nodes k and k+1 can be calculated by formula (8), where K is the total number of joints, which is 32 in this embodiment:

[0072]

[0073] In the nth gait cycle g n = [j n , j n+1 ], for the same node k, the feature vectors of the adjacent two frames f and f+1(jn ≤f<j n+1 The eigenvector between -1) can be calculated by formula (9):

[0074]

[0075] Step 4.2, the eigenvector of the same frame adjacent two nodes and the eigenvector of the same node adjacent frame are combined, and the upper and lower splicing is performed in this embodiment, so that a combined eigenvector v f,k,n of 6 dimensions (6 rows and 1 column) is obtained, that is, formula (10); other splicing methods can also be used in other embodiments;

[0076]

[0077] Step 4.3, for the nth gait cycle g n = [j n , j n+1 ], all the combined eigenvectors v f,k,n are arranged in order, which constitutes the corresponding eigenvector matrix M n of the gait cycle. In this embodiment, the transpose (1 row and 6 columns) of the combined eigenvector v f,k,n is spliced up and down to obtain the eigenvector matrix M n of the nth gait cycle, and other splicing methods can also be used to obtain the eigenvector matrix. The number of rows of the matrix M n is (j n+1 -j n -1)×(K-1), and the number of columns is 6, that is, formula (11).

[0078]

[0079] Wherein, the superscript T represents the transpose of the vector, and the column vector is converted into a row vector for matrix arrangement.

[0080] For each gait cycle corresponding to the eigenvector matrix M n , n = 1, 2, …, L-1, the eigenvector matrix set of the whole gait cycle set G of the subject can be represented as {M1, M2, …, M L-1}, and these eigenvector matrices can be used as input data of the model for subsequent training and analysis.

[0081] In step 5, the three-dimensional bone node coordinate data of each subject collected in step 1 during gait walking is processed through steps 2, 3, and 4 to obtain the feature matrix set of the entire gait cycle set G of each subject, and the feature matrix set of the entire gait cycle set G of each subject constitutes a data set, which is divided into a training set and a test set; according to the prior art, a fully connected neural network is constructed, an optimizer and a loss function are selected, the training set is used to train the fully connected network, and a classification model is obtained.

[0082] The specific processing process of the classification model is as follows:

[0083] The 3D convolution layer is used to extract features from the input feature matrix, that is, the features are extracted through the convolution operation shown in formula (12):

[0084]

[0085] Where a, b, and c are convolution kernel size parameters, Kernel is the convolution kernel, and ConvOutput is the output after convolution operation, which realizes the preliminary extraction of the space-time features in the feature matrix.

[0086] The maximum pooling operation is performed on the features after convolution to reduce the data dimension and retain key features, and the maximum pooling process is recorded as formula (13):

[0087]

[0088] Where s m ,s n ,s p is the size parameter of the pooling window, and PoolOutput is the output result after pooling.

[0089] The attention mechanism layer is used to weight the features after pooling to enhance the attention to key classification features, establish a feature weight model, and the specific formula (14) is as follows:

[0090]

[0091] Where Score i is the feature score, N is the number of features, AttentionWeight is the calculated attention weight, and AttendedFeatures is the feature after attention weighting.

[0092] The features after attention processing are flattened and input into the fully connected layer, the feature transformation and classification decision are performed through the multi-layer fully connected network, and the classification model formula (15) is established:

[0093] y = sigma(W2 ReLU(W1 Flatten(AttendedFeatures) + b1) + b2) (15)

[0094] wherein W1, W2 are weight matrices, b1, b2 are bias vectors, Flatten is a flattening operation, ReLU is a ReLU activation function, sigma is a Sigmoid activation function, and y is the final output classification probability.

[0095] In step 6, the gait cycle data of the subject (including normal people and patients) in the test set after the step-by-step skeleton node information is divided in the gait cycle is shown in Table 1.

[0096] Table 1

[0097]

[0098] Then, the feature vector is constructed, and the feature parameters of the present case are shown in Table 2:

[0099] Table 2

[0100]

[0101] The results of the present application based on the weighted average gait cycle division (Weighted average) and the existing technology traditional gait cycle division (Traditional gait cycle) using the same model (3D cnn+transformer) are shown in Table 3:

[0102] Table 3

[0103]

[0104] From the data in Table 3, in terms of training accuracy (Training accuracy), the traditional gait cycle hypothesis method reaches 0.96, while the model improved by the present application improves the training accuracy to 0.99. Compared with the traditional gait cycle, the improvement is 3%, which fully reflects the great advantage of the present application method in data learning ability;

[0105] In terms of verify accuracy (Verify accuracy), the method of the present application reaches 0.94. Among them, the verify accuracy of the traditional gait cycle is only 0.89, and the improved method of the present application has a gap of 5 percentage points compared with the traditional method. The gap reflects that the classification model of the present application has stronger stability in the cross-sample generalization scene, can better adapt to the diversity of data distribution in actual application, and provides more reliable technical support for clinical diagnosis of related diseases;

[0106] On the precision (Precession) index, the precision of the method of the application is 0.88, slightly lower than 0.89 of the traditional method. But combined with the improvement of the verification accuracy, it shows that the classification model of the application has more advantages in the overall classification reliability, especially in the complex sample environment, it is not easy to be affected by individual misjudgment to affect the overall performance.

[0107] On the recall (Recall) aspect, based on the improved gait cycle classification model, the recall rate reaches 0.87, which is higher than 0.84 of the traditional method, which means that it can reduce the misdiagnosis rate by 3.6% in identifying the abnormal gait of Parkinson's disease patients, and has important practical value for early clinical screening.

[0108] On the F1 score (F1 Sorce) index which comprehensively reflects the balance performance of the model, the F1 scores of both sides are 0.87, but the classification model of the application has double advantages in training accuracy (+3.1%) and verification accuracy (+5%), which highlights its comprehensive competitiveness in training stability and generalization ability. This shows that although it is flat with the traditional method in a single index, the model realizes the coordinated improvement of multi-dimensional performance by optimizing the gait cycle hypothesis and network structure, and is more suitable for practical application scenarios with higher reliability requirements.

[0109] The training and verification loss of the recognition result of the classification model of the application can be referred to Figure 2(a), the training and verification accuracy can be referred to Figure 2(b), the receiver operating characteristic example can be referred to Figure 2(c), and the precision, recall rate and F1 score can be referred to Figure 2(d).

[0110] The training and verification loss of the recognition result of the traditional model in the prior art can be referred to Figure 3(a), the training and verification accuracy can be referred to Figure 3(b), the receiver operating characteristic example can be referred to Figure 3(c), and the precision, recall rate and F1 score can be referred to Figure 3(d).

[0111] The abnormal gait recognition method based on the 3D skeleton coordinate convolution network is proposed. The method acquires 3D skeleton sequence by means of Kinect, designs a self-attention convolution architecture to capture early features, and the training optimization of the model can also use the PD-Walk data set in addition to the subjects. Compared with the traditional method, the method of the application does not rely on wearable devices for data collection, is more convenient, and the model is more accurate, which is expected to improve the early diagnosis accuracy of Parkinson's disease, provide technical support for early diagnosis and treatment of the disease, and promote the application of gait recognition technology in the medical field.

[0112] System embodiment:

[0113] The abnormal gait judgment system based on multi-feature fusion of the application can be a computer storing the classification model trained in the method embodiment.

[0114] The walking skeleton original data of the measured object is collected in the manner of step 1 of the method embodiment, including 32 joint coordinates; the data is processed in the manner of step 2 of the method embodiment, the gait cycle is divided in the manner of step 3 of the method embodiment, the feature matrix set of each gait cycle of the measured object is obtained in the manner of step 4 of the method embodiment, the feature matrix set is input into the trained classification model in the computer, and the abnormal gait evaluation result that can assist in the diagnosis of Parkinson's disease is output.

Claims

1. A method for determining an abnormal gait based on multi-feature fusion, characterized in that, The method comprises the following steps: 1) collecting walking skeleton original data containing coordinates of joint points during walking of a subject; 2) calculating the standard deviation of Z coordinates of each joint point in all frames, taking the ratio of the standard deviation of any joint point to the sum of standard deviations of all joint points as the weight of the joint point, and calculating the weighted sum of Z coordinates of each joint point in any frame as the weighted sum data of the frame; 3) dividing the frame sequence into gait cycles according to the local minimum value of the weighted sum data in the frame sequence as the boundary; 4) combining the feature vectors between adjacent two nodes in the same frame and the feature vectors between the same nodes in adjacent frames to obtain a combined feature vector, and arranging all the combined feature vectors of a gait cycle to obtain a feature matrix of the cycle, the feature matrices of each gait cycle of a subject constitute a feature matrix set; 5) training a fully connected neural network using the feature matrix sets of multiple subjects as a training set to obtain a classification model; 6) using the classification model to evaluate whether the gait of the measured object is an abnormal gait. 2.The method according to claim 1, characterized in that, In step 1), the subjects include Parkinson's patients of different grades and healthy people. 3.The method according to claim 1, characterized in that, In step 2), before calculating the standard deviation of Z coordinates of each joint point in all frames, a two-point equation interpolation method based on a straight line is used to smooth the normalized original data. 4.The method according to claim 1, characterized in that, In step 2), the Z coordinate of a joint point in a frame is the product of the Z coordinate of the corresponding joint point in the frame and the confidence of the corresponding joint point in the frame. 5.The method of claim 1, wherein, In step 3), the gait cycle set G is: G = {[j1, j2], [j2, j3],..., [j L-1 , L ]} wherein the sequence of local minima I of the weighted sum data of the frames of the subject ordered by frame timestamp min = {j1, j2,..., j L}. 6.The method of claim 1, wherein, In step 4), the feature vectors between adjacent two nodes in the same frame and the feature vectors between the same nodes in adjacent frames are spliced up and down to obtain a combined feature vector. 7.The method of claim 1, wherein, In step 4), the transpose of all the combined feature vectors of a gait cycle is spliced up and down to obtain a feature matrix of the cycle. 8.The method of claim 1, wherein, In step 5), the fully connected neural network comprises a 3D convolution layer for feature extraction, a max-pooling layer for performing a pooling operation on the features extracted by the convolution layer to reduce the data dimension, an attention mechanism layer for performing weighted processing on the features after dimension reduction to enhance the attention to key classification features, and a fully connected layer for feature transformation and classification decision on the output of the attention mechanism layer and the flattened features. 9.The method of claim 1, wherein, In step 1), the walking skeleton original data is collected using an Azure Kinect body sensor.

10. A multi-feature fusion-based abnormal gait judgment system, characterized in that, The classification model in the abnormal gait judgment method based on multi-feature fusion according to any one of claims 1-9. The classification model in the abnormal gait judgment method based on multi-feature fusion according to any one of claims 1-9.

Images