Accurate grading method and system based on video data

Abstract

The invention provides a precise grading method and system based on video data. The method comprises the following steps: S1, acquiring abnormal posture video data of normal postures and different nervous system severity levels; s2, decomposing the acquired video data into frames, and enhancing each frame of image; s3, gait key point detection is carried out on each frame of image after preprocessing, and key point positions are extracted; s4, main features of the gait key points are extracted; s5, extracting gesture features; and S6, training a deep neural network model for nervous system grading according to the gesture feature vector and the gait feature vector. According to the invention, the accuracy and reliability of disease detection are obviously improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a precise grading method and system based on video data. Background Art

[0002] Neurological diseases, such as Parkinson's disease, Alzheimer's disease, and multiple sclerosis, are serious health problems that affect millions of people worldwide. Traditional diagnostic methods for neurological diseases rely mainly on the experience and subjective judgment of clinicians, and often involve complex and expensive medical examinations such as magnetic resonance imaging and computed tomography. These methods are not only time-consuming and labor-intensive, but also have certain limitations for early diagnosis and dynamic monitoring of disease progression.

[0003] In recent years, with the rapid development of computer vision and deep learning technology, motion analysis methods based on video data have gradually become a research hotspot. Using the gait and gesture features in video data, neurological diseases can be non-invasively detected and evaluated, thus providing an efficient, economical and convenient means for early diagnosis and monitoring of diseases.

[0004] Existing gait analysis methods usually only focus on the spatiotemporal characteristics of gait, such as step length, step width, and step speed, while ignoring subtle changes during movement. Gesture recognition methods are mostly used in the field of human-computer interaction and are rarely used in medical diagnosis. At the same time, relying solely on gait or gesture features to detect neurological diseases may not provide enough information, so it is necessary to combine gait and gesture features to comprehensively analyze the patient's motor ability. Summary of the invention

[0005] In view of the defects in the prior art, an object of the present invention is to provide a precise grading method and system based on video data.

[0006] According to the present invention, a precise grading method based on video data includes:

[0007] Step S1: Acquire video data of normal posture and abnormal posture with different neurological severity levels;

[0008] Step S2: decomposing the acquired video data into frames and enhancing each frame image;

[0009] Step S3: gait key point detection is performed on each frame of the preprocessed image to extract the key point position;

[0010] Step S4: extracting main features of gait key points;

[0011] Step S5: extracting gesture features;

[0012] Step S6: training a deep neural network model for neural system classification based on the gesture feature vector and the gait feature vector.

[0013] Preferably, in step S1:

[0014] The total number of subjects is N, and the number of normal subjects is N n , the number of subjects is N d ; Divide the subjects into N l levels, and the number of subjects in each level is

[0015] A high-definition camera was used to capture the side view of each subject under uniform lighting conditions. Each subject took a gait video with a length of N. t minute.

[0016] Preferably, in step S2:

[0017] At the time interval of t1, the gait segment video of each subject is divided into Frame image;

[0018] For all frame images, the equalizeHist method in OpenCV is used to adjust the contrast of each frame image T to obtain the contrast-enhanced image Te. The hard threshold method is used to binarize Te, and the pixels greater than the threshold I are the foreground:

[0019]

[0020] Among them, Te t is the image after contrast enhancement in the tth frame, F t It is the binary image of frame t after removing the background.

[0021] Preferably, in step S3:

[0022] For gait, a total of Mp posture feature points are set, and for gestures, a total of Mh posture feature points are set. OpenPose is used to detect feature points for each frame of image to obtain the position coordinate information of each posture in each frame of image; the coordinate information P of the gait key point is:

[0023] P={p1,p2,...,p Mp}

[0024] p1={(x1,y1),(x2,y2),...,(x Nf ,y Nf )}

[0025] Among them, p1 is the coordinate position information set of the first key point in all time frames, (x1, y1) is the coordinate information of the key point of the first frame image, x1 is the horizontal coordinate of the key point of the first frame, and y1 is the vertical coordinate of the key point of the first frame; p Mp is the key point position information set of the last frame image, x Nf is the horizontal coordinate of the last key single position in the image, y Nf It is the vertical coordinate of the last key single position in the image.

[0026] Preferably, in step S4:

[0027] Step S4.1: Extract static features for each key point, including the average position and its variance, as follows:

[0028]

[0029] in, and They represent the average horizontal and vertical positions of the key points, t is the key point index, x t is the horizontal coordinate of the tth key point, y t is the ordinate of the t-th key point, and Respectively represent the variance of the horizontal and vertical positions of the key points;

[0030] Step S4.2: Extract dynamic features for each key point, including the velocity, acceleration, trajectory length and motion range of each key point, as follows:

[0031] The velocity v of the key point t :

[0032]

[0033] Where t1 is the time interval between frames;

[0034] Calculate the acceleration a of each key point t :

[0035] a t =(v t -v t-1 ) / t1

[0036] Calculate the instantaneous motion range R of each feature point i :

[0037] R i =max(x t )-min(x t )+max(y t )-min(y t )

[0038] Calculate the movement direction θ of the key points of each frame t :

[0039]

[0040] Among them, θ t represents the angle of motion direction in frame t;

[0041] The rate of change of motion direction describes the change of the motion direction of the key point:

[0042] Δθ t =θ t -θ t-1

[0043] Among them, Δθ t represents the rate of change of motion direction in frame t;

[0044] Angular velocity indicates the speed at which a key point rotates around a center point and is suitable for analyzing rotational motion:

[0045]

[0046] Among them, ω t represents the angular velocity at the tth frame, Δθ t Indicates the rate of change of motion direction;

[0047] Motion uniformity measures how the velocity of a keypoint changes between the current frame and the previous frame:

[0048] U t =|v t -v t-1 |

[0049] Among them, U t represents the motion uniformity at frame t;

[0050] Step S4.3: Gait feature vector construction:

[0051] Pull the features of all frames of each key point into a column to form a feature vector, then the feature vector of each key point is:

[0052] f i =[f i,1,1 ,f i,1,2 ,…,f i,1,11 ,f i,2,1 ,f i,2,2 ,…,f i,2,11 ,…,f i,Nf,1 ,f i,Nf,2 ,…,f i,Nf,11 ]

[0053] Among them, fi,Nf,11 is the 11th feature of the Nf-th frame image of the i-th key point;

[0054] Concatenate the feature vectors of all key points to form the final input vector F:

[0055] F=[f1,f2,…,f K ]

[0056] Among them, K is the total number of key points in each frame of gait image.

[0057] Preferably, in step S5:

[0058] Step S5.1: Obtaining multiple reference gesture data:

[0059] Display K1 fixed gestures on the mobile phone screen and record the key point coordinates of each benchmark gesture. The key point coordinates of each gesture are represented by G k , where (k=1,2,…,K1), where K1 is the number of benchmark gestures; each gesture contains N key point coordinates, expressed as

[0060] G k =[(x k1 ,y k1 ),(x k2 ,y k2 ),…,(x kN ,y kN )]

[0061] Among them, each point (x ki ,y ki ) represents the coordinates of the i-th key point of the reference gesture k;

[0062] Step S5.2: Get gesture data:

[0063] Make the same multiple fixed gestures as above under the camera, use OpenPose to detect the key points of the gesture image, and record the key point coordinates P of each gesture k , each gesture contains N points, expressed as:

[0064] P k =[(u k,1 ,v k,1 ),(u k,2 ,v k,2 ),…,(u k,N ,v k,N )]

[0065] Among them, each point (u k,i ,v k,i ) represents the coordinates of the i-th key point of the patient's gesture k;

[0066] Step S5.3: Gesture feature extraction:

[0067] Step S5.3.1: Calculate each reference gesture path G using Euclidean distance k The corresponding patient gesture path P k The matching degree between k The calculation formula is:

[0068]

[0069] Among them, G ki =(u ki ,v ki ) represents the i-th key point of the reference gesture k, P ki =(u ki ,v ki ) represents the i-th key point of the patient's gesture k;

[0070] Step S5.3.2 calculates the response time:

[0071] Record the time T when the patient starts to make each gesture start,k ; When D k (G k ,P k )<ε, record the time T to complete the gesture end,k Response time T k The calculation formula is:

[0072] T k =T end,k -T start,k

[0073] Step S5.3.2: Calculate the tremor index:

[0074] Calculate the high-frequency changes in the gesture path, the tremor index L k The calculation formula is:

[0075]

[0076] Where ΔP k,i Represents the displacement change of the i-th point, and the calculation formula is:

[0077]

[0078] Among them, u k,i is the horizontal coordinate of the i-th key point of the k-th gesture, v k,i is the ordinate of the i-th key point of the k-th gesture;

[0079] Step S5.4: Feature vector construction For each patient gesture k, construct a gesture feature vector Hk , which includes the gesture matching degree D k , response time T k , Tremor Index L k It is expressed as:

[0080] H k =[D k ,T k ,L k ]

[0081] Concatenate the feature vectors of all gestures to form the final comprehensive feature vector It is expressed as:

[0082]

[0083] Preferably, in step S6:

[0084] Step S6.1: Fusion of gesture feature vector and gait feature vector:

[0085]

[0086] Among them, V is the total feature vector of a subject;

[0087] And level V:

[0088]

[0089] The number of its elements is:

[0090] Nn=33×Nf×K×K1

[0091] The gait feature vector F is standardized:

[0092]

[0093] Among them, μ is the feature mean, σ is the standard deviation of the feature, and Z is the standardized feature value;

[0094] Step S6.2: Data dimensionality reduction:

[0095] Step S6.2.1: Combine the data of N subjects to obtain the data matrix D

[0096] D=[Z1,...,Z N ] T

[0097] Step S6.2.2: Generate a random projection matrix R using Rademacher distribution, whose size is Nn×r, r<<Nn, where the element R ij The assignment method is as follows:

[0098]

[0099] Among them, P ij is the element in the i-th row and j-th column of the matrix P. The matrix P is a random matrix in which each element is generated using a random number generator and the value range is [0,1];

[0100] Step S6.2.3: Multiply the data matrix D and the projection matrix R to obtain the projected data matrix Y:

[0101] Y=D×R

[0102] Among them, the size of Y is N×r;

[0103] Step S6.3: Construct training set and test set:

[0104] The reduced-dimensional data Y is divided into a training set and a test set. The training set is used to train the model, and the test set is used to evaluate the model performance.

[0105] Step S6.4: Input the normalized features into a multilayer perceptron network. The specific design is as follows: This network includes an input layer with r neurons, three hidden layers with M1, M3, M3 neurons and an output layer with 5 neurons. The specific training process is as follows:

[0106] Step S6.4.1: Input the data into the first hidden layer, with the input being the feature vector V ' , the output is the hidden layer feature l1, and the first hidden layer is set to contain M1 neurons:

[0107] l1=ReLU(W1Y+b1)

[0108] Among them, the size of W1 is M1×Nn, the size of l1 is M1, b1 is the bias term, its size is M1, and ReLU is the activation function;

[0109] Step S6.4.2: Input the output data of step S6.4.1 into the second hidden layer, the input is the feature vector l1, the output is the hidden layer feature l2, and the second hidden layer is set to contain M2 neurons;

[0110] l2=ReLU(W2l1+b2)

[0111] Among them, the size of W2 is M2×M1, the size of l2 is M2, and b2 is the bias term, whose size is M2;

[0112] Step S6.4.3: Input the output data in step S6.4.2 into the third hidden layer, with the input being the feature vector l2 and the output being the hidden layer feature l3; assume that the third hidden layer contains M3 neurons:

[0113] l3=ReLU(W3l2+b3)

[0114] Among them, the size of W2 is M3×M2, the size of l3 is M3, and b3 is the bias term, whose size is M3;

[0115] Step S6.4.4: Input the output data in step S6.4.3 into the output layer, and the input is the feature vector l3,

[0116] y=Softmax(W o l3+b0)

[0117] Among them, the size of W2 is 5×M3, the size of y is 5, and the size of b o is the bias term, its size is 5, and Softmax is the activation function;

[0118] Step S6.4.5: Use the cross entropy loss function to measure the difference between the predicted probability distribution y and the actual label; the formula of the cross entropy loss function is as follows:

[0119]

[0120] in, represents the loss value, N represents the number of samples, y true,i,c represents the cth class true label of the ith sample, y i,c represents the predicted probability of the cth category of the i-th sample;

[0121] Step S6.4.6: Use the back propagation algorithm to update the weights W1, W2, W3, W o and bias b1, b2, b3, b o The above process is repeated m1 times to obtain the final network model parameters W1, W2, W3, W o and bias b1, b2, b3, b o ; m1 times are set manually;

[0122] Step S6.5: Input the various indicators of the personnel to be graded into the network to obtain the severity level of the personnel to be graded

[0123]

[0124] p new It is the characteristics of the personnel to be classified after data dimension reduction.

[0125] According to the present invention, a precise grading system based on video data is provided, comprising:

[0126] Module M1: Acquire video data of normal posture and abnormal posture with different neurological severity levels;

[0127] Module M2: decompose the acquired video data into frames and enhance each frame image;

[0128] Module M3: Detect gait key points on each frame of preprocessed image and extract the key point positions;

[0129] Module M4: Extract the main features of gait key points;

[0130] Module M5: Extract gesture features;

[0131] Module M6: Training a deep neural network model for neural system classification based on the gesture feature vector and the gait feature vector.

[0132] Preferably, in the module M1:

[0133] The total number of subjects is N, and the number of normal subjects is N n , the number of subjects is N d ; Divide the subjects into N l levels, and the number of subjects in each level is

[0134] A high-definition camera was used to capture the side view of each subject under uniform lighting conditions. Each subject took a gait video with a length of N. t minute;

[0135] In the module M2:

[0136] At the time interval of t1, the gait segment video of each subject is divided into Frame image;

[0137] For all frame images, the equalizeHist method in OpenCV is used to adjust the contrast of each frame image T to obtain the contrast-enhanced image Te. The hard threshold method is used to binarize Te, and the pixels greater than the threshold I are the foreground:

[0138]

[0139] Among them, Te t is the image after contrast enhancement in the tth frame, F t is the binary image of frame t after removing the background;

[0140] In the module M3:

[0141] For gait, a total of Mp posture feature points are set, and for gestures, a total of Mh posture feature points are set. OpenPose is used to detect feature points for each frame of image to obtain the position coordinate information of each posture in each frame of image; the coordinate information P of the gait key point is:

[0142] P={p1,p2,...,p Mp}

[0143] p1={(x1,y1),(x2,y2),...,(x Nf ,y Nf )}

[0144] Among them, p1 is the coordinate position information set of the first key point in all time frames, (x1, y1) is the coordinate information of the key point of the first frame image, x1 is the horizontal coordinate of the key point of the first frame, and y1 is the vertical coordinate of the key point of the first frame; p Mp is the key point position information set of the last frame image, x Nf is the horizontal coordinate of the last key single position in the image, y Nf is the ordinate of the last key single position in the image;

[0145] In the module M4:

[0146] Module M4.1: Extract static features for each key point, including the average position and its variance, as follows:

[0147]

[0148] in, and They represent the average horizontal and vertical positions of the key points, t is the key point index, x t is the horizontal coordinate of the tth key point, y t is the ordinate of the t-th key point, and Respectively represent the variance of the horizontal and vertical positions of the key points;

[0149] Module M4.2: Extract dynamic features for each key point, including the velocity, acceleration, trajectory length and motion range of each key point, as follows:

[0150] The velocity v of the key point t :

[0151]

[0152] Where t1 is the time interval between frames;

[0153] Calculate the acceleration a of each key point t :

[0154] a t =(v t -v t-1 ) / t1

[0155] Calculate the instantaneous motion range R of each feature point i :

[0156] R i =max(x t )-min(x t )+max(y t )-min(y t )

[0157] Calculate the movement direction θ of the key points of each frame t :

[0158]

[0159] Among them, θ t represents the angle of motion direction in frame t;

[0160] The rate of change of motion direction describes the change of the motion direction of the key point:

[0161] Δθ t =θ t -θ t-1

[0162] Among them, Δθ t represents the rate of change of motion direction in frame t;

[0163] Angular velocity indicates the speed at which a key point rotates around a center point and is suitable for analyzing rotational motion:

[0164]

[0165] Among them, ω t represents the angular velocity at the tth frame, Δθ t Indicates the rate of change of motion direction;

[0166] Motion uniformity measures how the velocity of a keypoint changes between the current frame and the previous frame:

[0167] U t =|v t -v t-1 |

[0168] Among them, U t represents the motion uniformity at frame t;

[0169] Module M4.3 gait feature vector construction:

[0170] Pull the features of all frames of each key point into a column to form a feature vector, then the feature vector of each key point is:

[0171] f i =[f i,1,1 ,f i,1,2 ,…,f i,1,11 ,f i,2,1 ,f i,2,2 ,…,f i,2,11 ,…,f i,Nf,1 ,f i,Nf,2 ,…,f i,Nf,11 ]

[0172] Among them, f i,Nf,11 is the 11th feature of the Nf-th frame image of the i-th key point;

[0173] Concatenate the feature vectors of all key points to form the final input vector F:

[0174] F=[f1,f2,…,f K ]

[0175] Among them, K is the total number of key points in each frame of gait image.

[0176] Preferably, in the module M5:

[0177] Module M5.1: Obtaining multiple reference gesture data:

[0178] Display K1 fixed gestures on the mobile phone screen and record the key point coordinates of each benchmark gesture. The key point coordinates of each gesture are represented by G k , where (k=1,2,…,K1), where K1 is the number of benchmark gestures; each gesture contains N key point coordinates, expressed as

[0179] G k =[(x k1 ,y k1 ),(x k2 ,y k2 ),…,(x kN ,y kN )]

[0180] Among them, each point (x ki ,y ki ) represents the coordinates of the i-th key point of the reference gesture k;

[0181] Module M5.2: Get gesture data:

[0182] Make the same multiple fixed gestures as above under the camera, use OpenPose to detect the key points of the gesture image, and record the key point coordinates P of each gesturek , each gesture contains N points, expressed as:

[0183] P k =[(u k,1 ,v k,1 ),(u k,2 ,v k,2 ),…,(u k,N ,v k,N )]

[0184] Among them, each point (u k,i ,v k,i ) represents the coordinates of the i-th key point of the patient's gesture k;

[0185] Module M5.3: Gesture feature extraction:

[0186] Module M5.3.1: Calculate each reference gesture path G using Euclidean distance k The corresponding patient gesture path P k The matching degree between k The calculation formula is:

[0187]

[0188] Among them, G ki =(u ki ,v ki ) represents the i-th key point of the reference gesture k, P ki =(u ki ,v ki ) represents the i-th key point of the patient's gesture k;

[0189] Module M5.3.2 Calculate response time:

[0190] Record the time T when the patient starts to make each gesture start,k When D k (G k ,P k )<ε, record the time T to complete the gesture end,k ; Response time T k The calculation formula is:

[0191] T k =T end,k -T start,k

[0192] Module M5.3.2: Calculate Tremor Index:

[0193] Calculate the high-frequency changes in the gesture path, the tremor index L k The calculation formula is:

[0194]

[0195] Where ΔP k,i Represents the displacement change of the i-th point, and the calculation formula is:

[0196]

[0197] Among them, u k,i is the horizontal coordinate of the i-th key point of the k-th gesture, v k,i is the ordinate of the i-th key point of the k-th gesture;

[0198] Module M5.4: Feature vector construction For each patient gesture k, construct the gesture feature vector H k , which includes the gesture matching degree D k , response time T k , Tremor Index L k It is expressed as:

[0199] H k =[D k ,T k ,L k ]

[0200] Concatenate the feature vectors of all gestures to form the final comprehensive feature vector It is expressed as:

[0201]

[0202] In the module M6:

[0203] Module M6.1: Fusion of gesture feature vector and gait feature vector:

[0204]

[0205] Among them, V is the total feature vector of a subject;

[0206] And level V:

[0207]

[0208] The number of its elements is:

[0209] Nn=33×Nf×K×K1

[0210] The gait feature vector F is standardized:

[0211]

[0212] Among them, μ is the feature mean, σ is the standard deviation of the feature, and Z is the standardized feature value;

[0213] Module M6.2: Data Dimensionality Reduction:

[0214] Module M6.2.1: Combine the data of N subjects to obtain the data matrix D

[0215] D=[Z1,...,Z N ] T

[0216] Module M6.2.2: Generate a random projection matrix R using Rademacher distribution, whose size is Nn×r, r<<Nn, where the element R ij The assignment method is as follows:

[0217]

[0218] Among them, P ij is the element in the i-th row and j-th column of the matrix P. The matrix P is a random matrix in which each element is generated using a random number generator and the value range is [0,1];

[0219] Module M6.2.3: Multiply the data matrix D and the projection matrix R to obtain the projected data matrix Y:

[0220] Y=D×R

[0221] Among them, the size of Y is N×r;

[0222] Module M6.3: Construct training and test sets:

[0223] The reduced-dimensional data Y is divided into a training set and a test set. The training set is used to train the model, and the test set is used to evaluate the model performance.

[0224] Module M6.4: Input the normalized features into a multilayer perceptron network. The specific design is as follows: This network contains an input layer with r neurons, three hidden layers with M1, M3, M3 neurons and an output layer with 5 neurons. The specific training process is as follows:

[0225] Module M6.4.1: Input data into the first hidden layer, with the input being the feature vector V ' , the output is the hidden layer feature l1, and the first hidden layer is set to contain M1 neurons:

[0226] l1=ReLU(W1Y+b1)

[0227] Among them, the size of W1 is M1×Nn, the size of l1 is M1, b1 is the bias term, its size is M1, and ReLU is the activation function;

[0228] Module M6.4.2: Input the output data of module M6.4.1 into the second hidden layer, with the input being the feature vector l1 and the output being the hidden layer feature l2. The second hidden layer is set to contain M2 neurons;

[0229] l2=ReLU(W2l1+b2)

[0230] Among them, the size of W2 is M2×M1, the size of l2 is M2, and b2 is the bias term, whose size is M2;

[0231] Module M6.4.3: Input the output data in module M6.4.2 into the third hidden layer, with the input being the feature vector l2 and the output being the hidden layer feature l3; assume that the third hidden layer contains M3 neurons:

[0232] l3=ReLU(W3l2+b3)

[0233] Among them, the size of W2 is M3×M2, the size of l3 is M3, and b3 is the bias term, whose size is M3;

[0234] Module M6.4.4: Input the output data in module M6.4.3 into the output layer, with the input being feature vector l3,

[0235] y=Softmax(W o l3+b0)

[0236] Among them, the size of W2 is 5×M3, the size of y is 5, and the size of b o is the bias term, its size is 5, and Softmax is the activation function;

[0237] Module M6.4.5: Use the cross entropy loss function to measure the difference between the predicted probability distribution y and the actual label; the formula of the cross entropy loss function is as follows:

[0238]

[0239] in, represents the loss value, N represents the number of samples, y true,i,c represents the cth class true label of the ith sample, y i,c represents the predicted probability of the cth category of the ith sample;

[0240] Module M6.4.6: Use the back-propagation algorithm to update weights W1, W2, W3, W o and bias b1, b2, b3, b o The above process is repeated m1 times to obtain the final network model parameters W1, W2, W3, W o and bias b1, b2, b3, b o ; m1 times are set manually;

[0241] Module M6.5: Input the various indicators of the personnel to be graded into the network to obtain the severity level of the personnel to be graded

[0242]

[0243] p new It is the characteristics of the personnel to be classified after data dimension reduction.

[0244] Compared with the prior art, the present invention has the following beneficial effects:

[0245] 1. The present invention combines gait and gesture video data, and uses computer vision and deep learning technology to comprehensively analyze the patient's motion characteristics and extract static and dynamic features. These features are input into the deep learning neural network model for classification and grading, which significantly improves the accuracy and reliability of disease detection.

[0246] 2. The present invention uses video data for analysis without invasive operation, and provides an efficient, economical and convenient detection method compared with traditional medical examination methods (such as MRI and CT). The high level of automation of the system reduces manual operation and subjective judgment errors, improves clinical work efficiency, and reduces the burden on patients.

[0247] 3. The present invention can realize early detection and dynamic monitoring of neurological diseases. By continuously analyzing the patient's gait and gesture video data, the system can monitor the disease progression in real time, help clinicians adjust the treatment plan in time, and improve the overall treatment effect. It is particularly suitable for neurological diseases such as Parkinson's disease, Alzheimer's disease and multiple sclerosis.

[0248] 4. The present invention is not only applicable to the detection of neurological diseases such as Parkinson's disease, Alzheimer's disease and multiple sclerosis, but can also be extended to the evaluation of other neurological diseases and injuries. The system acquires and analyzes patient health data in real time, provides high-quality data support for clinical research, promotes the development of personalized medicine, and improves the quality of life and survival time of patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0249] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0250] Figure 1 It is a schematic diagram of the hand gesture diagram;

[0251] Figure 2 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0252] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0253] Embodiment 1:

[0254] The present invention relates to the field of computer vision and nervous system disease detection, and specifically, to a method, medium and device for detecting nervous system diseases based on gait and gesture video data. The method extracts gait and gesture features from the video, and uses deep learning technology to analyze and classify the extracted features, thereby effectively detecting nervous system diseases such as Parkinson's disease, Alzheimer's disease and multiple sclerosis. The present invention aims to provide a method for accurately grading nervous system diseases such as neurotoxicity, Alzheimer's disease, Parkinson's disease and multiple sclerosis based on video data. By combining gait and gesture video data, using computer vision and deep learning technology, the patient's movement characteristics are analyzed from multiple angles to improve the accuracy and reliability of disease detection.

[0255] According to the present invention, a precise classification method based on video data is provided. Figure 2 As shown, including:

[0256] Step S1: Acquire video data of normal posture and abnormal posture with different neurological severity levels;

[0257] Specifically, in step S1:

[0258] The total number of subjects is N, and the number of normal subjects is N n , the number of subjects is N d ; Divide the subjects into N l levels, and the number of subjects in each level is

[0259] A high-definition camera was used to capture the side view of each subject under uniform lighting conditions. Each subject took a gait video with a length of N. t minute.

[0260] Step S2: decomposing the acquired video data into frames and enhancing each frame image;

[0261] Specifically, in step S2:

[0262] At the time interval of t1, the gait segment video of each subject is divided into Frame image;

[0263] For all frame images, the equalizeHist method in OpenCV is used to adjust the contrast of each frame image T to obtain the contrast-enhanced image Te. The hard threshold method is used to binarize Te, and the pixels greater than the threshold I are the foreground:

[0264]

[0265] Among them, Te t is the image after contrast enhancement in the tth frame, F t It is the binary image of frame t after removing the background.

[0266] Step S3: gait key point detection is performed on each frame of the preprocessed image to extract the key point position;

[0267] Specifically, in step S3:

[0268] For gait, a total of Mp posture feature points are set, and for gestures, a total of Mh posture feature points are set. OpenPose is used to detect feature points for each frame of image to obtain the position coordinate information of each posture in each frame of image; the coordinate information P of the gait key point is:

[0269] P={p1,p2,...,p Mp}

[0270] p1={(x1,y1),(x2,y2),...,(x Nf ,t Nf )}

[0271] Among them, p1 is the coordinate position information set of the first key point in all time frames, (x1, y1) is the coordinate information of the key point of the first frame image, x1 is the horizontal coordinate of the key point of the first frame, and y1 is the vertical coordinate of the key point of the first frame; p Mp is the key point position information set of the last frame image, x Nf is the horizontal coordinate of the last key single position in the image, y Nf It is the vertical coordinate of the last key single position in the image.

[0272] Step S4: extracting main features of gait key points;

[0273] Specifically, in step S4:

[0274] Step S4.1: Extract static features for each key point, including the average position and its variance, as follows:

[0275]

[0276]

[0277] in, and They represent the average horizontal and vertical positions of the key points, t is the key point index, x t is the horizontal coordinate of the tth key point, y t is the ordinate of the t-th key point, and Respectively represent the variance of the horizontal and vertical positions of the key points;

[0278] Step S4.2: Extract dynamic features for each key point, including the velocity, acceleration, trajectory length and motion range of each key point, as follows:

[0279] The velocity v of the key point t :

[0280]

[0281] Where t1 is the time interval between frames;

[0282] Calculate the acceleration a of each key point t :

[0283] a t =(v t -v t-1 ) / t1

[0284] Calculate the instantaneous motion range R of each feature point i :

[0285] R i =max(x t )-min(x t )+max(y t )-min(y t )

[0286] Calculate the movement direction θ of the key points of each frame t :

[0287]

[0288] Among them, θ t represents the angle of motion direction in frame t;

[0289] The rate of change of motion direction describes the change of the motion direction of the key point:

[0290] Δθ t =θ t -θ t-1

[0291] Among them, Δθ t represents the rate of change of motion direction in frame t;

[0292] Angular velocity indicates the speed at which a key point rotates around a center point and is suitable for analyzing rotational motion:

[0293]

[0294] Among them, ω t represents the angular velocity at the tth frame, Δθ t Indicates the rate of change of motion direction;

[0295] Motion uniformity measures how the velocity of a keypoint changes between the current frame and the previous frame:

[0296] U t =|v t -v t-1 |

[0297] Among them, U t represents the motion uniformity at frame t;

[0298] Step S4.3: Gait feature vector construction:

[0299] Pull the features of all frames of each key point into a column to form a feature vector, then the feature vector of each key point is:

[0300] f=[f i,1,1 ,f i,1,2 ,…,f i,1,11 ,f i,2,1 ,f i,2,2 ,…,f i,2,11 ,…,f i,Nf,1 ,f i,Nf,2 ,…,f i,Nf,11 ]

[0301] Among them, f i,Nf,11 is the 11th feature of the Nf-th frame image of the i-th key point;

[0302] Concatenate the feature vectors of all key points to form the final input vector F:

[0303] F=[f1,f2,…,f K ]

[0304] Among them, K is the total number of key points in each frame of gait image.

[0305] Step S5: extracting gesture features;

[0306] Specifically, in step S5:

[0307] Step S5.1: Obtaining multiple reference gesture data:

[0308] Display K1 fixed gestures on the mobile phone screen and record the key point coordinates of each benchmark gesture. The key point coordinates of each gesture are represented by G k , where (k=1,2,…,K1), where K1 is the number of benchmark gestures; each gesture contains N key point coordinates, expressed as

[0309] G k =[(x k1 ,y k1 ),(x k2 ,y k2 ),…,(x kN ,y kN )]

[0310] Among them, each point (x ki ,y ki ) represents the coordinates of the i-th key point of the reference gesture k;

[0311] Step S5.2: Get gesture data:

[0312] Make the same multiple fixed gestures as above under the camera, use OpenPose to detect the key points of the gesture image, and record the key point coordinates P of each gesture k , each gesture contains N points, expressed as:

[0313] P k =[(u k,1 ,v k,1 ),(u k,2 ,v k,2 ),…,(u k,N ,v k,N )]

[0314] Among them, each point (u k,i ,v k,i ) represents the coordinates of the i-th key point of the patient's gesture k;

[0315] Step S5.3: Gesture feature extraction:

[0316] Step S5.3.1: Calculate each reference gesture path G using Euclidean distance k The corresponding patient gesture path P k The matching degree between k The calculation formula is:

[0317]

[0318] Among them, G ki =(u ki ,v ki ) represents the i-th key point of the reference gesture k, Pki =(u ki ,v ki ) represents the i-th key point of the patient's gesture k;

[0319] Step S5.3.2 calculates the response time:

[0320] Record the time T when the patient starts to make each gesture start,k When D k (G k ,P k )<ε, record the time T to complete the gesture end,k ; Response time T k The calculation formula is:

[0321] T k =T end,k -T start,k

[0322] Step S5.3.2: Calculate the tremor index:

[0323] Calculate the high-frequency changes in the gesture path, the tremor index L k The calculation formula is:

[0324]

[0325] Where ΔP k,i Represents the displacement change of the i-th point, and the calculation formula is:

[0326]

[0327] Among them, u k,i is the horizontal coordinate of the i-th key point of the k-th gesture, v k,i is the ordinate of the i-th key point of the k-th gesture;

[0328] Step S5.4: Feature vector construction For each patient gesture k, construct a gesture feature vector H k , which includes the gesture matching degree D k , response time T k , Tremor Index L k It is expressed as:

[0329] H k =[D k ,T k ,L k ]

[0330] Concatenate the feature vectors of all gestures to form the final comprehensive feature vector It is expressed as:

[0331]

[0332] Step S6: training a deep neural network model for neural system classification based on the gesture feature vector and the gait feature vector.

[0333] Specifically, in step S6:

[0334] Step S6.1: Fusion of gesture feature vector and gait feature vector:

[0335]

[0336] Among them, V is the total feature vector of a subject;

[0337] And level V:

[0338]

[0339] The number of its elements is:

[0340] Nn=33×Nf×K×K1

[0341] The gait feature vector F is standardized:

[0342]

[0343] Among them, μ is the feature mean, σ is the standard deviation of the feature, and Z is the standardized feature value;

[0344] Step S6.2: Data dimensionality reduction:

[0345] Step S6.2.1: Combine the data of N subjects to obtain the data matrix D

[0346] D=[Z1,...,Z N ] T

[0347] Step S6.2.2: Generate a random projection matrix R using Rademacher distribution, whose size is Nn×r, r<<Nn, where the element R ij The assignment method is as follows:

[0348]

[0349] Among them, P ij is the element in the i-th row and j-th column of the matrix P. The matrix P is a random matrix in which each element is generated using a random number generator and the value range is [0,1];

[0350] Step S6.2.3: Multiply the data matrix D and the projection matrix R to obtain the projected data matrix Y:

[0351] Y=D×R

[0352] Among them, the size of Y is N×r;

[0353] Step S6.3: Construct training set and test set:

[0354] The reduced-dimensional data Y is divided into a training set and a test set. The training set is used to train the model, and the test set is used to evaluate the model performance.

[0355] Step S6.4: Input the normalized features into a multilayer perceptron network. The specific design is as follows: This network includes an input layer with r neurons, three hidden layers with M1, M3, M3 neurons and an output layer with 5 neurons. The specific training process is as follows:

[0356] Step S6.4.1: Input the data into the first hidden layer, with the input being the feature vector V ′ , the output is the hidden layer feature l1, and the first hidden layer is set to contain M1 neurons:

[0357] l1=ReLU(W1Y+b1)

[0358] Among them, the size of W1 is M1×Nn, the size of l1 is M1, b1 is the bias term, its size is M1, and ReLU is the activation function;

[0359] Step S6.4.2: Input the output data of step S6.4.1 into the second hidden layer, the input is the feature vector l1, the output is the hidden layer feature l2, and the second hidden layer is set to contain M2 neurons;

[0360] l2=ReLU(W2l1+b2)

[0361] Among them, the size of W2 is M2×M1, the size of l2 is M2, and b2 is the bias term, whose size is M2;

[0362] Step S6.4.3: Input the output data in step S6.4.2 into the third hidden layer, with the input being the feature vector l2 and the output being the hidden layer feature l3; assume that the third hidden layer contains M3 neurons:

[0363] l3=ReLU(W3l2+b3)

[0364] Among them, the size of W2 is M3×M2, the size of l3 is M3, and b3 is the bias term, whose size is M3;

[0365] Step S6.4.4: Input the output data in step S6.4.3 into the output layer, and the input is the feature vector l3,

[0366] y=Softmax(Wo l3+b0)

[0367] Among them, the size of W2 is 5×M i , y has a size of 5, b o is the bias term, its size is 5, and Softmax is the activation function;

[0368] Step S6.4.5: Use the cross entropy loss function to measure the difference between the predicted probability distribution y and the actual label; the formula of the cross entropy loss function is as follows:

[0369]

[0370] in, represents the loss value, N represents the number of samples, y true,i,c represents the cth class true label of the ith sample, y i,c represents the predicted probability of the cth category of the i-th sample;

[0371] Step S6.4.6: Use the back propagation algorithm to update the weights W1, W2, W3, W o and bias b1, b2, b3, b o The above process is repeated m1 times to obtain the final network model parameters W1, W2, W3, W o and bias b1, b2, b3, b o ; m1 times are set manually;

[0372] Step S6.5: Input the various indicators of the personnel to be graded into the network to obtain the severity level of the personnel to be graded

[0373]

[0374] p new It is the characteristics of the personnel to be classified after data dimension reduction.

[0375] Embodiment 2:

[0376] Embodiment 2 is a preferred example of Embodiment 1, and is used to illustrate the present invention in more detail.

[0377] The present invention also provides a precise grading system based on video data, which can be implemented by executing the process steps of the precise grading method based on video data, that is, those skilled in the art can understand the precise grading method based on video data as a preferred implementation of the precise grading system based on video data.

[0378] According to the present invention, a precise grading system based on video data is provided, comprising:

[0379] Module M1: Acquire video data of normal posture and abnormal posture with different neurological severity levels;

[0380] Specifically, in the module M1:

[0381] The total number of subjects is N, and the number of normal subjects is N n , the number of subjects is N d ; Divide the subjects into N l levels, and the number of subjects in each level is

[0382] A high-definition camera was used to capture the side view of each subject under uniform lighting conditions. Each subject took a gait video with a length of N. t minute;

[0383] Module M2: decompose the acquired video data into frames and enhance each frame image;

[0384] In the module M2:

[0385] At the time interval of t1, the gait segment video of each subject is divided into Frame image;

[0386] For all frame images, the equalizeHist method in OpenCV is used to adjust the contrast of each frame image T to obtain the contrast-enhanced image Te. The hard threshold method is used to binarize Te, and the pixels greater than the threshold I are the foreground:

[0387]

[0388] Among them, Te t is the image after contrast enhancement in the tth frame, F t is the binary image of frame t after removing the background;

[0389] Module M3: Detect gait key points on each frame of preprocessed image and extract the key point positions;

[0390] In the module M3:

[0391] For gait, a total of Mp posture feature points are set, and for gestures, a total of Mh posture feature points are set. OpenPose is used to detect feature points for each frame of image to obtain the position coordinate information of each posture in each frame of image; the coordinate information P of the gait key point is:

[0392] P={p1,p2,...,p Mp}

[0393] p1={(x1,y1),(x2,y2),...,(x Nf ,y Nf )}

[0394] Among them, p1 is the coordinate position information set of the first key point in all time frames, (x1, y1) is the coordinate information of the key point of the first frame image, x1 is the horizontal coordinate of the key point of the first frame, and y1 is the vertical coordinate of the key point of the first frame; p Mp is the key point position information set of the last frame image, x Nf is the horizontal coordinate of the last key single position in the image, y Nf is the ordinate of the last key single position in the image;

[0395] Module M4: Extract the main features of gait key points;

[0396] In the module M4:

[0397] Module M4.1: Extract static features for each key point, including the average position and its variance, as follows:

[0398]

[0399] in, and They represent the average horizontal and vertical positions of the key points, t is the key point index, x t is the horizontal coordinate of the tth key point, y t is the ordinate of the t-th key point, and Respectively represent the variance of the horizontal and vertical positions of the key points;

[0400] Module M4.2: Extract dynamic features for each key point, including the velocity, acceleration, trajectory length and motion range of each key point, as follows:

[0401] The velocity v of the key point t :

[0402]

[0403] Where t1 is the time interval between frames;

[0404] Calculate the acceleration a of each key point t :

[0405] a t =(v t -v t-1 ) / t1

[0406] Calculate the instantaneous motion range R of each feature point i :

[0407] R i =max(x t )-min(x t )+max(y t )-min(y t )

[0408] Calculate the movement direction θ of the key points of each frame t :

[0409]

[0410] Among them, θ t represents the angle of motion direction in frame t;

[0411] The rate of change of motion direction describes the change of the motion direction of the key point:

[0412] Δθ t =θ t -θ t-1

[0413] Among them, Δθ t represents the rate of change of motion direction in frame t;

[0414] Angular velocity indicates the speed at which a key point rotates around a center point and is suitable for analyzing rotational motion:

[0415]

[0416] Among them, ω t represents the angular velocity at the tth frame, Δθ t Indicates the rate of change of motion direction;

[0417] Motion uniformity measures how the velocity of a keypoint changes between the current frame and the previous frame:

[0418] U t =|v t -v t-1 |

[0419] Among them, U t represents the motion uniformity at frame t;

[0420] Module M4.3 gait feature vector construction:

[0421] Pull the features of all frames of each key point into a column to form a feature vector, then the feature vector of each key point is:

[0422] f i =[f i,1,1 ,f i,1,2 ,…,f i,1,11 ,fi,2,1 ,f i,2,2 ,…,f i,2,11 ,…,f i,Nf,1 ,f i,Nf,2 ,…,f i,Nf,11 ]

[0423] Among them, f i,Nf,11 is the 11th feature of the Nf-th frame image of the i-th key point;

[0424] Concatenate the feature vectors of all key points to form the final input vector F:

[0425] F=[f1,f2,…,f K ]

[0426] Among them, K is the total number of key points in each frame of gait image.

[0427] Module M5: Extract gesture features;

[0428] Specifically, in the module M5:

[0429] Module M5.1: Obtaining various reference gesture data:

[0430] Display K1 fixed gestures on the mobile phone screen and record the key point coordinates of each benchmark gesture. The key point coordinates of each gesture are represented by G k , where (k=1,2,…,K1), where K1 is the number of benchmark gestures; each gesture contains N key point coordinates, expressed as

[0431] G k =[(x k1 ,y k1 ),(x k2 ,y k2 ),…,(x kN ,y kN )]

[0432] Among them, each point (x ki ,y ki ) represents the coordinates of the i-th key point of the reference gesture k;

[0433] Module M5.2: Get gesture data:

[0434] Make the same multiple fixed gestures as above under the camera, use OpenPose to detect the key points of the gesture image, and record the key point coordinates P of each gesture k , each gesture contains N points, expressed as:

[0435] P k =[(u k,1 ,vk,1 ),(u k,2 ,v k,2 ),…,(u k,N ,v k,N )]

[0436] Among them, each point (u k,i ,v k,i ) represents the coordinates of the i-th key point of the patient's gesture k;

[0437] Module M5.3: Gesture feature extraction:

[0438] Module M5.3.1: Calculate each reference gesture path G using Euclidean distance k The corresponding patient gesture path P k The matching degree between k The calculation formula is:

[0439]

[0440] Among them, G ki =(u ki ,v ki ) represents the i-th key point of the reference gesture k, P ki =(u ki ,v ki ) represents the i-th key point of the patient's gesture k;

[0441] Module M5.3.2 Calculate response time:

[0442] Record the time T when the patient starts to make each gesture start,k When D k (G k ,P k )<ε, record the time T to complete the gesture end,k ; Response time T k The calculation formula is:

[0443] T k =T end,k -T start,k

[0444] Module M5.3.2: Calculate Tremor Index:

[0445] Calculate the high-frequency changes in the gesture path, the tremor index L k The calculation formula is:

[0446]

[0447] Where ΔP k,i Represents the displacement change of the i-th point, and the calculation formula is:

[0448]

[0449] Among them, u k,i is the horizontal coordinate of the i-th key point of the k-th gesture, v k,i is the ordinate of the i-th key point of the k-th gesture;

[0450] Module M5.4: Feature vector construction For each patient gesture k, construct the gesture feature vector H k , which includes the gesture matching degree D k , response time T k , Tremor Index L k It is expressed as:

[0451] H k =[D k ,T k ,L k ]

[0452] Concatenate the feature vectors of all gestures to form the final comprehensive feature vector It is expressed as:

[0453]

[0454] Module M6: Training a deep neural network model for neural system classification based on the gesture feature vector and the gait feature vector.

[0455] In the module M6:

[0456] Module M6.1: Fusion of gesture feature vector and gait feature vector:

[0457]

[0458] Among them, V is the total feature vector of a subject;

[0459] And level V:

[0460]

[0461] The number of its elements is:

[0462] Nn=33×Nf×K×K1

[0463] The gait feature vector F is standardized:

[0464]

[0465] Among them, μ is the feature mean, σ is the standard deviation of the feature, and Z is the standardized feature value;

[0466] Module M6.2: Data Dimensionality Reduction:

[0467] Module M6.2.1: Combine the data of N subjects to obtain the data matrix D

[0468] D=[Z1,...,Z N ] T

[0469] Module M6.2.2: Generate a random projection matrix R using Rademacher distribution, whose size is Nn×r, r<<Nn, where the element R ij The assignment method is as follows:

[0470]

[0471] Among them, P ij is the element in the i-th row and j-th column of the matrix P. The matrix P is a random matrix in which each element is generated using a random number generator and the value range is [0,1];

[0472] Module M6.2.3: Multiply the data matrix D and the projection matrix R to obtain the projected data matrix Y:

[0473] Y=D×R

[0474] Among them, the size of Y is N×r;

[0475] Module M6.3: Construct training and test sets:

[0476] The reduced-dimensional data Y is divided into a training set and a test set. The training set is used to train the model, and the test set is used to evaluate the model performance.

[0477] Module M6.4: Input the normalized features into a multilayer perceptron network. The specific design is as follows: This network contains an input layer with r neurons, three hidden layers with M1, M3, M3 neurons and an output layer with 5 neurons. The specific training process is as follows:

[0478] Module M6.4.1: Input data into the first hidden layer, with the input being the feature vector V ′ , the output is the hidden layer feature l1, and the first hidden layer is set to contain M1 neurons:

[0479] l1=ReLU(W1Y+b1)

[0480] Among them, the size of W1 is M1×Nn, the size of l1 is M1, b1 is the bias term, its size is M1, and ReLU is the activation function;

[0481] Module M6.4.2: Input the output data of module M6.4.1 into the second hidden layer, with the input being the feature vector l1 and the output being the hidden layer feature l2. The second hidden layer is set to contain M2 neurons;

[0482] l2=ReLU(W2l1+b2)

[0483] Among them, the size of W2 is M2×M1, the size of l2 is M2, and b2 is the bias term, whose size is M2;

[0484] Module M6.4.3: Input the output data in module M6.4.2 into the third hidden layer, with the input being the feature vector l2 and the output being the hidden layer feature l3; assume that the third hidden layer contains M3 neurons:

[0485] l3=ReLU(W3l2+b3)

[0486] Among them, the size of W2 is M3×M2, the size of l3 is M3, and b3 is the bias term, whose size is M3;

[0487] Module M6.4.4: Input the output data in module M6.4.3 into the output layer, with the input being feature vector l3,

[0488] y=Softmax(W o l3+b0)

[0489] Among them, the size of W2 is 5×M3, the size of y is 5, and the size of b o is the bias term, its size is 5, and Softmax is the activation function;

[0490] Module M6.4.5: Use the cross entropy loss function to measure the difference between the predicted probability distribution y and the actual label; the formula of the cross entropy loss function is as follows:

[0491]

[0492] in, represents the loss value, N represents the number of samples, y true,i,c represents the cth class true label of the ith sample, y i,c represents the predicted probability of the cth category of the i-th sample;

[0493] Module M6.4.6: Use the back-propagation algorithm to update weights W1, W2, W3, W o and bias b1, b2, b3, b o The above process is repeated m1 times to obtain the final network model parameters W1, W2, W3, W o and bias b1, b2, b3, b o ; m1 times are set manually;

[0494] Module M6.5: Input the various indicators of the personnel to be graded into the network to obtain the severity level of the personnel to be graded

[0495]

[0496] p new It is the characteristics of the personnel to be classified after data dimension reduction.

[0497] Embodiment 3:

[0498] Embodiment 3 is a preferred example of Embodiment 1, and is used to illustrate the present invention in more detail.

[0499] The present invention provides a method for extracting features by combining gait video and gesture video data for detecting neurological diseases such as Parkinson's disease. The algorithm involves key point detection, feature extraction, feature vector construction and disease classification in gait and gesture videos.

[0500] A method for grading neurological diseases based on computer vision provided by the present invention comprises:

[0501] Step S1: Obtaining video data of normal posture and abnormal posture of different neurological disease levels

[0502] Step S2: Decompose the acquired video data into frames, and enhance each frame image to improve the image quality and the accuracy of feature extraction.

[0503] Step S3: Perform gait key point detection on each frame of the preprocessed image and extract the key point positions.

[0504] Step S4: extract the main features of the gait key points.

[0505] Step S5: Gesture feature extraction.

[0506] Step S6: Training a deep neural network model for neurological disease classification.

[0507] Preferably, in step S1:

[0508] The total number of subjects is N, and the number of normal subjects is N n , the number of subjects with neurological diseases is N d The subjects with neurological diseases were divided into N l levels, and the number of subjects in each level is

[0509] A high-definition camera was used to capture the side view of each subject under uniform lighting conditions. Each subject took a gait video with a length of N. tminute.

[0510] Preferably, in step S2:

[0511] At the time interval of t1, the gait segment video of each subject is divided into Frame image.

[0512] For all frame images, first use the equalizeHist method in OpenCV to adjust the contrast of each frame image T to obtain the contrast-enhanced image Te, and then use the hard threshold method to binarize Te. Pixels greater than the threshold I are foreground:

[0513]

[0514] Among them, Te t is the image after contrast enhancement in the tth frame, F t It is the binary image of frame t after removing the background.

[0515] Preferably, in step S3:

[0516] For gait, a total of Mp posture feature points are set, and for gestures, a total of Mh posture feature points are set. Use OpenPose to detect feature points for each frame of image and obtain the position coordinate information of each posture in each frame of image. The coordinate information of gait key points is:

[0517] P={p1,p2,...,p Mp}

[0518] p1={(x1,y1),(x2,y2),...,(x Nf ,y Nf )}

[0519] Where p1 is the coordinate position information set of the first key point in all time frames, (x1, y1) is the coordinate information of the key point of the first frame image, x1 is the horizontal coordinate of the key point of the first frame, and y1 is the vertical coordinate of the key point of the first frame.

[0520] The coordinate information of the gesture key points is:

[0521] H={h1,h2,...,h Nh}

[0522] h1={(x1,y1),(x2,y2),...,(x Nf ,y Nf )}

[0523] Where h1 is the coordinate position information set of the first key point of the finger in all time frames, (x1, y1) is the coordinate information of the key point of the first frame image, x1 is the horizontal coordinate of the key point of the first frame, and y1 is the vertical coordinate of the key point of the first frame.

[0524] Preferably, in step S4:

[0525] Step S4.1 extracts static features for each key point, including the average position and its variance, as follows:

[0526]

[0527]

[0528] in and Respectively represent the average horizontal and vertical positions of key points, and Respectively represent the variance of the horizontal and vertical positions of the key points.

[0529] Step S4.2 extracts dynamic features for each key point, including the velocity, acceleration, trajectory length and motion range of each key point, as follows.

[0530] Speed ​​of key points:

[0531]

[0532] Calculate the acceleration of each key point using the following formula:

[0533] a t =(v t -v t-1 ) / t1

[0534] Calculate the instantaneous motion range of each feature point using the following formula:

[0535] R i =max(x t )-min(x t )+max(y t )-min(y t )

[0536] Calculate the movement direction of the key points of each frame, the formula is as follows.

[0537]

[0538] Among them, θ t Indicates the angle of motion direction at the tth frame.

[0539] The rate of change of motion direction describes the change of the motion direction of the key point.

[0540] Δθ t =θ t -θ t-1

[0541] Among them, Δθ t Represents the rate of change of motion direction at the tth frame.

[0542] Angular velocity indicates the speed at which a key point rotates around a center point and is suitable for analyzing rotational motion.

[0543]

[0544] Among them, ω t represents the angular velocity at the tth frame, Δθ t Indicates the rate of change of motion direction

[0545] Motion uniformity measures how the velocity of a keypoint changes between the current frame and the previous frame.

[0546] U t =|v t -v t-1 |

[0547] Among them, U t represents the motion uniformity at the tth frame.

[0548] Step S4.3 Gait feature vector construction

[0549] Pull the features of all frames of each key point into a column to form a feature vector, then the feature vector of each key point is:

[0550] f i =[f i,1,1 ,f i,1,2 ,…,f i,1,11 ,f i,2,1 ,f i,2,2 ,…,f i,2,11 ,…,f i,Nf,1 ,f i,Nf,2 ,…,f i,Nf,11 ]

[0551] Concatenate the feature vectors of all key points to form the final input vector:

[0552] F=[f1,f2,…,f K ]

[0553] Preferably, in step S5:

[0554] Step S5.1: Obtaining multiple reference gesture data

[0555] Display K1 fixed gestures on the mobile phone screen and record the key point coordinates of each benchmark gesture, such as Figure 1 As shown by the larger dots, the key point coordinates of each gesture are represented by G k , where (k=1,2,…,K1), each gesture contains N key point coordinates, expressed as

[0556] G k =[(x k1 ,y k1 ),(x k2 ,y k2 ),…,(x kN ,y kN )]

[0557] Among them, each point (x ki ,y ki ) represents the coordinates of the i-th key point of the reference gesture k.

[0558] Step S5.2: Obtaining patient gesture data

[0559] Ask the patient to make the same multiple fixed gestures mentioned above under the camera, and use OpenPose to detect the key points of the gesture images, such as Figure 1 As shown by the blue dots, the key point coordinates P of each gesture are recorded. k , each gesture contains N points, expressed as:

[0560] P k =[(u k1 ,v k1 ),(u k2 ,v k2 ),…,(u kN ,v kN )]

[0561] Among them, each point (u ki ,v ki ) represents the coordinates of the i-th key point of the patient's gesture k.

[0562] Step S5.3 Gesture feature extraction

[0563] Step S5.3.1 Calculate each reference gesture path G using the Euclidean distance k The corresponding patient gesture path P k The matching degree between k The calculation formula is:

[0564]

[0565] Among them, G ki =(u ki ,v ki) represents the i-th key point of the reference gesture k, P ki =(u ki ,v ki ) represents the i-th key point of patient gesture k.

[0566] Step S5.3.2 Calculate response time

[0567] Record the time T when the patient starts to make each gesture start,k When D k (G k ,P k )<ε, record the time T to complete the gesture end,k Response time T k The calculation formula is:

[0568] T k =T end,k -T start,k

[0569] Step S5.3.2 Calculate the tremor index:

[0570] Calculate the high-frequency changes in the gesture path, the tremor index L k The calculation formula is:

[0571]

[0572] Where ΔP k,i Represents the displacement change of the i-th point, and the calculation formula is:

[0573]

[0574] Step S5.4: Feature vector construction For each patient gesture k, construct a gesture feature vector H k , which includes the gesture matching degree D k , response time T k , Tremor Index L k It is expressed as:

[0575] H k =[D k ,T k ,L k ]

[0576] Concatenate the feature vectors of all gestures to form the final comprehensive feature vector It is expressed as:

[0577]

[0578] Preferably, in step S6:

[0579] Step S6.1: Fusion of gesture feature vector and gait feature vector

[0580]

[0581] Among them, V is the total feature vector of a subject.

[0582] And level V:

[0583]

[0584] The number of its elements is Nn = 33 × Nf × K × K1

[0585] The gait feature vector F is normalized

[0586]

[0587] μ is the feature mean, σ is the standard deviation of the feature, and Z is the standardized feature value.

[0588] Step S6.2 Data dimensionality reduction:

[0589] Step S6.2.1 Combine the data of N subjects to obtain the data matrix D

[0590] D=[Z1,...,Z N ] T

[0591] Step S6.2.2 generates a random projection matrix R using Rademacher distribution, whose size is Nn×r, r<<Nn, where the element R ij The assignment method is as follows:

[0592]

[0593] Where P ij is the element in the i-th row and j-th column of the matrix P. The matrix P is a random matrix in which each element is generated using a random number generator and the value range is [0,1].

[0594] Step S6.2.3 multiplies the data matrix D and the projection matrix R to obtain the projected data matrix Y:

[0595] Y=D×R

[0596] The size of Y is N×r

[0597] Step S6.3 builds training and test sets:

[0598] The reduced data Y is divided into a training set and a test set. The training set is used to train the model, and the test set is used to evaluate the model performance. The data division ratio can be set to 80% for training and 20% for testing.

[0599] Step S6.4: Input the above normalized features into a multilayer perceptron network, which is specifically designed as follows: This network includes an input layer with r neurons, three hidden layers with M1, M3, M3 neurons and an output layer with 5 neurons. The specific training process is as follows:

[0600] Step S6.4.1 Input data into the first hidden layer, the input is the feature vector V′, and the output is the hidden layer feature l1. The first hidden layer is assumed to contain M1 neurons.

[0601] l1=ReLU(W1Y+b1)

[0602] Among them, the size of W1 is M1×Nn, the size of l1 is M1, b1 is the bias term, its size is M1, and ReLU is the activation function.

[0603] Step S6.4.2: Input the output data in step S5.5.1 into the second hidden layer, the input is the feature vector l1, and the output is the hidden layer feature l2. The second hidden layer is assumed to contain M2 neurons.

[0604] l2=ReLU(W2l1+b2)

[0605] Among them, the size of W2 is M2×M1, the size of l2 is M2, and b2 is the bias term, whose size is M2.

[0606] Step S6.4.3: Input the output data in step S5.5.2 into the third hidden layer, the input is the feature vector l2, and the output is the hidden layer feature l3. The third hidden layer is assumed to contain M3 neurons.

[0607] l3=ReLU(W3l2+b3)

[0608] Among them, the size of W2 is M3×M2, the size of l3 is M3, and b3 is the bias term, whose size is M3.

[0609] Step S6.4.4 inputs the output data in step S5.5.3 into the output layer, and the input is the feature vector l3,

[0610] y=Softmax(W o l3+b0)

[0611] Among them, the size of W2 is 5×M3, the size of y is 5, and the size of b o is a bias term with a size of 5 and Softmax is the activation function.

[0612] Step S6.4.5 uses the cross entropy loss function to measure the difference between the predicted probability distribution y and the actual label. The formula of the cross entropy loss function is as follows:

[0613]

[0614] in, represents the loss value, N represents the number of samples, y true,i,c represents the cth class true label of the ith sample, y i,c represents the predicted probability of the cth class of the ith sample.

[0615] Step S6.4.6 uses the back propagation algorithm to update the weights W1, W2, W3, W o and bias b1, b2, b3, b o The above process is repeated several times to obtain the final network model parameters W1, W2, W3, W o and bias b1, b2, b3, b o .

[0616] Step S6.5: Input the various indicators of the person to be graded into the network to obtain the severity level of the mental illness of the person to be graded.

[0617]

[0618] p new It is the characteristics of the personnel to be classified after data dimension reduction.

[0619] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0620] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A precise grading method based on video data, characterized in that: include: Step S1: Acquire video data of normal posture and abnormal posture with different neurological severity levels; Step S2: decomposing the acquired video data into frames and enhancing each frame image; Step S3: gait key point detection is performed on each frame of the preprocessed image to extract the key point position; Step S4: extracting main features of gait key points; Step S5: extracting gesture features; Step S6: training a deep neural network model for neural system classification based on the gesture feature vector and the gait feature vector.

2. The accurate grading method based on video data according to claim 1, characterized in that: In step S1: The total number of subjects is N, and the number of normal subjects is N n , the number of subjects is N d ; Divide the subjects into N l levels, and the number of subjects in each level is A high-definition camera was used to capture the side view of each subject under uniform lighting conditions. Each subject took a gait video with a length of N. t minute.

3. The accurate grading method based on video data according to claim 1, characterized in that: In step S2: At the time interval of T1, the gait segment video of each subject is divided into Frame image; For all frame images, the equalizeHist method in OpenCV is used to adjust the contrast of each frame image T to obtain the contrast-enhanced image Te. The hard threshold method is used to binarize Te, and the pixels greater than the threshold I are the foreground: Among them, Te t is the image after contrast enhancement in the tth frame, F t It is the binary image of frame t after removing the background.

4. The accurate grading method based on video data according to claim 1, characterized in that: In step S3: For gait, a total of Mp posture feature points are set, and for gestures, a total of Mh posture feature points are set. OpenPose is used to detect feature points for each frame of image to obtain the position coordinate information of each posture in each frame of image; the coordinate information P of the gait key point is: P={p1,p2,...,p Mp } p1={(x1,y1),(x2,y2),...,(x Nf ,y Nf )} Among them, p1 is the coordinate position information set of the first key point in all time frames, (x1, y1) is the coordinate information of the key point of the first frame image, x1 is the horizontal coordinate of the key point of the first frame, and y1 is the vertical coordinate of the key point of the first frame; p Mp is the key point position information set of the last frame image, x Nf is the horizontal coordinate of the last key single position in the image, y Nf It is the vertical coordinate of the last key single position in the image.

5. The accurate grading method based on video data according to claim 1, characterized in that: In step S4: Step S4.1: Extract static features for each key point, including the average position and its variance, as follows: in, and They represent the average horizontal and vertical positions of the key points, t is the key point index, x t is the horizontal coordinate of the tth key point, y t is the ordinate of the t-th key point, and Respectively represent the variance of the horizontal and vertical positions of the key points; Step S4.2: Extract dynamic features for each key point, including the velocity, acceleration, trajectory length and motion range of each key point, as follows: The velocity v of the key point t : Where t1 is the time interval between frames; Calculate the acceleration a of each key point t : from t =(v t -v t-1 ) / t1 Calculate the instantaneous motion range R of each feature point i : R i =max(x t )-min(x t )+max(y t )-min(y t ) Calculate the movement direction θ of the key points of each frame t : Among them, θ t represents the angle of motion direction in frame t; The rate of change of motion direction describes the change of the motion direction of the key point: Dth t =θ t -θ t-1 Among them, Δθ t represents the rate of change of motion direction in frame t; Angular velocity indicates the speed at which a key point rotates around a center point and is suitable for analyzing rotational motion: Among them, ω t represents the angular velocity at the tth frame, Δθ1 represents the rate of change of the motion direction; Motion uniformity measures how the velocity of a keypoint changes between the current frame and the previous frame: U t =|v t -v 1-1 | Among them, U t represents the motion uniformity at frame t; Step S4.3: Gait feature vector construction: Pull the features of all frames of each key point into a column to form a feature vector, then the feature vector of each key point is: f i =[f i,1,1 ,f i,1,2 ,…,f i,1,11 ,f i,2,1 ,f i,2,2 ,…,f i,2,11 ,…,f i,Nf,1 ,f i,Nf,2 ,…,f i,Nf,11 ] Among them, f i,Nf,11 is the 11th feature of the Nf-th frame image of the i-th key point; Concatenate the feature vectors of all key points to form the final input vector F: F=[f1,f2,…,f K ] Among them, K is the total number of key points in each frame of gait image.

6. The accurate grading method based on video data according to claim 1, characterized in that: In step S5: Step S5.1: Obtaining multiple reference gesture data: Display K1 fixed gestures on the mobile phone screen and record the key point coordinates of each benchmark gesture. The key point coordinates of each gesture are represented by G k , where (k=1,2,…,K1), where K1 is the number of benchmark gestures; each gesture contains N key point coordinates, represented by G k =[(x k1 ,y k1 ),(x k2 ,y k2 ),…,(x kN ,y kN )] Among them, each point (x ki ,y ki ) represents the coordinates of the i-th key point of the reference gesture k; Step S5.2: Get gesture data: Make the same multiple fixed gestures as above under the camera, use OpenPose to detect the key points of the gesture image, and record the key point coordinates P of each gesture k , each gesture contains N points, expressed as: P k =[(u k,1 ,v k,1 ),(u k,2 ,v k,2 ),…,(u k,N ,v k,N )] Among them, each point (u k,i ,v k,i ) represents the coordinates of the i-th key point of the patient's gesture k; Step S5.3: Gesture feature extraction: Step S5.3.1: Calculate each reference gesture path G using Euclidean distance k The corresponding patient gesture path P k The matching degree between k The calculation formula is: Among them, G ki =(u ki ,v ki ) represents the i-th key point of the reference gesture k, P ki =(u ki ,v ki ) represents the i-th key point of the patient's gesture k; Step S5.3.2 calculates the response time: Record the time T when the patient starts to make each gesture start,k When D k (G k ,P k )<ε, record the time T to complete the gesture end,k Response time T k The calculation formula is: T k =T end,k -T start,k Step S5.3.2: Calculate the tremor index: Calculate the high-frequency changes in the gesture path, the tremor index L k The calculation formula is: Where ΔP k,i Represents the displacement change of the i-th point, and the calculation formula is: Among them, u k,i is the horizontal coordinate of the i-th key point of the k-th gesture, v k,i is the ordinate of the i-th key point of the k-th gesture; Step S5.4: Feature vector construction For each patient gesture k, construct a gesture feature vector H k , which includes the gesture matching degree D k , response time T k , Tremor Index L k It is expressed as: H k =[D k ,T k ,L k ] Concatenate the feature vectors of all gestures to form the final comprehensive feature vector It is expressed as:

7. The accurate grading method based on video data according to claim 1, characterized in that: In step S6: Step S6.1: Fusion of gesture feature vector and gait feature vector: Among them, V is the total feature vector of a subject; And level V: The number of its elements is: Nn=33×Nf×K×K1 The gait feature vector F is standardized: Among them, μ is the feature mean, σ is the standard deviation of the feature, and Z is the standardized feature value; Step S6.2: Data dimensionality reduction: Step S6.2.1: Combine the data of N subjects to obtain the data matrix D D=[Z1,...,Z N ] T Step S6.2.2: Generate a random projection matrix R using Rademacher distribution, whose size is Nn×r, r<<Nn, where the element R ij The assignment method is as follows: Among them, P ij is the element in the i-th row and j-th column of the matrix P. The matrix P is a random matrix in which each element is generated using a random number generator and the value range is [0,1]; Step S6.2.3: Multiply the data matrix D and the projection matrix R to obtain the projected data matrix Y: Y=D×R Among them, the size of Y is N×r; Step S6.3: Construct training set and test set: The reduced-dimensional data Y is divided into a training set and a test set. The training set is used to train the model, and the test set is used to evaluate the model performance. Step S6.4: Input the normalized features into a multilayer perceptron network. The specific design is as follows: This network includes an input layer with r neurons, three hidden layers with M1, M3, M3 neurons and an output layer with 5 neurons. The specific training process is as follows: Step S6.4.1: Input the data into the first hidden layer, with the input being the feature vector V ′ , the output is the hidden layer feature l1, and the first hidden layer is set to contain M1 neurons: l1=ReLU(W1Y+b1) Among them, the size of W1 is M1×Nn, the size of l1 is M1, b1 is the bias term, its size is M1, and ReLU is the activation function; Step S6.4.2: Input the output data of step S6.4.1 into the second hidden layer, the input is the feature vector l1, the output is the hidden layer feature l2, and the second hidden layer is set to contain M2 neurons; l2=ReLU(W2l1+b2) Among them, the size of W2 is M2×M1, the size of l2 is M2, and b2 is the bias term, whose size is M2; Step S6.4.3: Input the output data in step S6.4.2 into the third hidden layer, with the input being the feature vector l2 and the output being the hidden layer feature l3; assume that the third hidden layer contains M3 neurons: l3=ReLU(W3l2+b3) Among them, the size of W2 is M3×M2, the size of l3 is M3, and b3 is the bias term, whose size is M3; Step S6.4.4: Input the output data in step S6.4.3 into the output layer, and the input is the feature vector l3, <h2 style=";text-align:left;direction:ltr">y = Softmax(W<h2 style=";text-align:left;direction:ltr"> o <h2 style=";text-align:left;direction:ltr"> l3+b0) Among them, the size of W2 is 5×M3, the size of y is 5, and the size of b o is the bias term, its size is 5, and Softmax is the activation function; Step S6.4.5: Use the cross entropy loss function to measure the difference between the predicted probability distribution y and the actual label; the formula of the cross entropy loss function is as follows: in, represents the loss value, N represents the number of samples, y true,i,c represents the cth class true label of the ith sample, y i,c represents the predicted probability of the cth category of the i-th sample; Step S6.4.6: Use the back propagation algorithm to update the weights W1, W2, W3, W o and bias b1, b2, b3, b o The above process is repeated m1 times to obtain the final network model parameters W1, W2, W3, W o and bias b1, b2, b3, b o ; m1 times are set manually; Step S6.5: Input the various indicators of the personnel to be graded into the network to obtain the severity level of the personnel to be graded p new It is the characteristics of the personnel to be classified after data dimension reduction.

8. An accurate grading system based on video data, characterized in that: include: Module M1: Acquire video data of normal posture and abnormal posture with different neurological severity levels; Module M2: decompose the acquired video data into frames and enhance each frame image; Module M3: Detect gait key points on each frame of preprocessed image and extract the key point positions; Module M4: Extract the main features of gait key points; Module M5: Extract gesture features; Module M6: Training a deep neural network model for neural system classification based on the gesture feature vector and the gait feature vector.

9. The accurate grading system based on video data according to claim 8, characterized in that: In the module M1: The total number of subjects is N, and the number of normal subjects is N n , the number of subjects is N d ; Divide the subjects into N L levels, and the number of subjects in each level is A high-definition camera was used to capture the side view of each subject under uniform lighting conditions. Each subject took a gait video with a length of N. t minute; In the module M2: At the time interval of t1, the gait segment video of each subject is divided into Frame image; For all frame images, the equalizeHist method in OpenCV is used to adjust the contrast of each frame image T to obtain the contrast-enhanced image Te. The hard threshold method is used to binarize Te, and the pixels greater than the threshold I are the foreground: Among them, Te t is the image after contrast enhancement in the tth frame, F t is the binary image of frame t after removing the background; In the module M3: For gait, a total of Mp posture feature points are set, and for gestures, a total of Mh posture feature points are set. OpenPose is used to detect feature points for each frame of image to obtain the position coordinate information of each posture in each frame of image; the coordinate information P of the gait key point is: P={p1,p2,...,p Mp } p1={(x1,y1),(x2,y2),...,(x Nf ,y Nf )} Among them, p1 is the coordinate position information set of the first key point in all time frames, (x1, y1) is the coordinate information of the key point of the first frame image, x1 is the horizontal coordinate of the key point of the first frame, and y1 is the vertical coordinate of the key point of the first frame; p Mp is the key point position information set of the last frame image, x Nf is the horizontal coordinate of the last key single position in the image, y Nf is the ordinate of the last key single position in the image; In the module M4: Module M4.1: Extract static features for each key point, including the average position and its variance, as follows: in, and They represent the average horizontal and vertical positions of the key points, t is the key point index, x t is the horizontal coordinate of the tth key point, y t is the ordinate of the t-th key point, and Respectively represent the variance of the horizontal and vertical positions of the key points; Module M4.2: Extract dynamic features for each key point, including the velocity, acceleration, trajectory length and motion range of each key point, as follows: The velocity v of the key point t : Where t1 is the time interval between frames; Calculate the acceleration a of each key point t : from t =(v t -v t-1 ) / t1 Calculate the instantaneous motion range R of each feature point i : R i =max(x t )-min(x t )+max(y t )-min(y t ) Calculate the movement direction θ of the key points of each frame t : Among them, θ t represents the angle of motion direction in frame t; The rate of change of motion direction describes the change of the motion direction of the key point: Dth t =θ t -θ t-1 Among them, Δθ t represents the rate of change of motion direction in frame t; Angular velocity indicates the speed at which a key point rotates around a center point and is suitable for analyzing rotational motion: Among them, ω t represents the angular velocity at the tth frame, Δθ t Indicates the rate of change of motion direction; Motion uniformity measures how the velocity of a keypoint changes between the current frame and the previous frame: U t =|v t -v t-1 | Among them, U t represents the motion uniformity at frame t; Module M4.3 gait feature vector construction: Pull the features of all frames of each key point into a column to form a feature vector, then the feature vector of each key point is: f i =[f i,1,1 ,f i,1,2 ,…,f i,1,11 ,f i,2,1 ,f i,2,2 ,…,f i,2,11 ,…,f i,Nf,1 ,f i,Nf,2 ,…,f i,Nf,11 ] Among them, f i,Nf,11 is the 11th feature of the Nf-th frame image of the i-th key point; Concatenate the feature vectors of all key points to form the final input vector F: F=[f1,f2,…,f K ] Among them, K is the total number of key points in each frame of gait image.

10. The accurate grading system based on video data according to claim 8, characterized in that: In the module M5: Module M5.1: Obtaining multiple reference gesture data: Display K1 fixed gestures on the mobile phone screen and record the key point coordinates of each benchmark gesture. The key point coordinates of each gesture are represented by G k , where (k=1,2,…,K1), where K1 is the number of benchmark gestures; each gesture contains N key point coordinates, represented by G k =[(x k1 ,y k1 ),(x k2 ,y k2 ),…,(x kN ,y kN )] Among them, each point (x ki ,y ki ) represents the coordinates of the i-th key point of the reference gesture k; Module M5.2: Get gesture data: Make the same multiple fixed gestures as above under the camera, use OpenPose to detect the key points of the gesture image, and record the key point coordinates P of each gesture k , each gesture contains N points, expressed as: P k =[(u k,1 ,v k,1 ),(u k,2 ,v k,2 ),…,(u k,N ,v k,N )] Among them, each point (u k,i ,v k,i ) represents the coordinates of the i-th key point of the patient's gesture k; Module M5.3: Gesture feature extraction: Module M5.3.1: Calculate each reference gesture path G using Euclidean distance k The corresponding patient gesture path P k The matching degree between k The calculation formula is: Among them, G ki =(u ki ,v ki ) represents the i-th key point of the reference gesture k, P ki =(u ki ,v ki ) represents the i-th key point of the patient's gesture k; Module M5.3.2 Calculate response time: Record the time T when the patient starts to make each gesture start,k ; When D k (G k ,P k )<ε, record the time T to complete the gesture end,k ; Response time T k The calculation formula is: T k =T end,k -T start,k Module M5.3.2: Calculate Tremor Index: Calculate the high-frequency changes in the gesture path, the tremor index L k The calculation formula is: Among them, ΔP k,i Represents the displacement change of the i-th point, and the calculation formula is: Among them, u k,i is the horizontal coordinate of the i-th key point of the k-th gesture, v k,i is the ordinate of the i-th key point of the k-th gesture; Module M5.4: Feature vector construction For each patient gesture k, construct the gesture feature vector H k , which includes the gesture matching degree D k , response time T k , Tremor Index L k It is expressed as: H k =[D k ,T k ,L k ] Concatenate the feature vectors of all gestures to form the final comprehensive feature vector It is expressed as: In the module M6: Module M6.1: Fusion of gesture feature vector and gait feature vector: Among them, V is the total feature vector of a subject; And level V: The number of its elements is: Nn=33×Nf×K×K1 The gait feature vector F is standardized: Among them, μ is the feature mean, σ is the standard deviation of the feature, and Z is the standardized feature value; Module M6.2: Data Dimensionality Reduction: Module M6.2.1: Combine the data of N subjects to obtain the data matrix D D=[Z1,...,Z N ] T Module M6.2.2: Generate a random projection matrix R using Rademacher distribution, whose size is Nn×r, r<<Nn, where the element R ij The assignment method is as follows: Among them, P ij is the element in the i-th row and j-th column of the matrix P. The matrix P is a random matrix in which each element is generated using a random number generator and the value range is [0,1]; Module M6.2.3: Multiply the data matrix D and the projection matrix R to obtain the projected data matrix Y: Y=D×R Among them, the size of Y is N×r; Module M6.3: Construct training and test sets: The reduced-dimensional data Y is divided into a training set and a test set. The training set is used to train the model, and the test set is used to evaluate the model performance. Module M6.4: Input the normalized features into a multilayer perceptron network. The specific design is as follows: This network contains an input layer with r neurons, three hidden layers with M1, M3, M3 neurons and an output layer with 5 neurons. The specific training process is as follows: Module M6.4.1: Input data into the first hidden layer, with the input being the feature vector V ′ , the output is the hidden layer feature L1, and the first hidden layer is set to contain M1 neurons: l1=ReLU(W1Y+b1) Among them, the size of W1 is M1×Nn, the size of l1 is M1, b1 is the bias term, its size is M1, and ReLU is the activation function; Module M6.4.2: Input the output data of module M6.4.1 into the second hidden layer, with the input being the feature vector l1 and the output being the hidden layer feature l2. The second hidden layer is set to contain M2 neurons; l2=ReLU(W2l1+b2) Among them, the size of W2 is M2×M1, the size of l2 is M2, and b2 is the bias term, whose size is M2; Module M6.4.3: Input the output data in module M6.4.2 into the third hidden layer, with the input being the feature vector l2 and the output being the hidden layer feature l3; assume that the third hidden layer contains M3 neurons: l3=ReLU(W3l2+b3) Among them, the size of W2 is M3×M2, the size of l3 is M3, and b3 is the bias term, whose size is M3; Module M6.4.4: Input the output data in module M6.4.3 into the output layer, with the input being feature vector l3, <h2 style=";text-align:left;direction:ltr">y = Softmax(W<h2 style=";text-align:left;direction:ltr"> o <h2 style=";text-align:left;direction:ltr"> l3+b0) Among them, the size of W2 is 5×M3, the size of y is 5, and the size of b o is the bias term, its size is 5, and Softmax is the activation function; Module M6.4.5: Use the cross entropy loss function to measure the difference between the predicted probability distribution y and the actual label; the formula of the cross entropy loss function is as follows: in, represents the loss value, N represents the number of samples, y true,i,c represents the cth class true label of the ith sample, y i,c represents the predicted probability of the cth category of the ith sample; Module M6.4.6: Use the back-propagation algorithm to update weights W1, W2, W3, W o and bias b1, b2, b3, b o The above process is repeated m1 times to obtain the final network model parameters W1, W2, W3, W o and bias b1, b2, b3, b o ; m1 times are set manually; Module M6.5: Input the various indicators of the personnel to be graded into the network to obtain the severity level of the personnel to be graded p new It is the characteristics of the personnel to be classified after data dimension reduction.

Images