A parkinsonian movement slowness quantification analysis method based on hand gesture video

Abstract

The application discloses a Parkinson motor retardation quantitative analysis method based on hand posture video, comprising the following steps: A: constructing a hand posture data set; B: constructing a Mask R-CNN deep neural network model comprising a feature extraction network, a region candidate network, a boundary box head network and a key point head network; C: training the Mask R-CNN deep neural network model to obtain a key point sequence composed of 21 hand key points; D: extracting motion features from the key point sequence to obtain corresponding motion features; E: obtaining a wave peak sequence and a wave trough sequence according to the motion features; and F: quantifying five motor retardation parameters based on the wave peak sequence and the wave trough sequence, wherein the five motor retardation parameters are slow speed, low vibration, amplitude attenuation, hesitation and pause. The application can take a hand posture video segment as a carrier to realize flexible and interpretable motor retardation quantification.

Description

TECHNICAL FIELD

[0001] The application relates to a Parkinson's motor slowness quantitative analysis method, in particular to a Parkinson's disease motor slowness quantitative analysis method based on hand gesture videos. BACKGROUND

[0002] Motor slowness, i.e. involuntary movement stop, is one of the common motor symptoms of Parkinson's disease (PD) patients and is also a major basis for clinical screening and diagnosis. At present, the gold standard for evaluating the severity of motor slowness is the Unified Parkinson's Disease Rating Scale (MDS-UPDRS). In the third part of the MDS-UPDRS motor examination, three typical hand motor slowness items are listed: finger tapping, hand movement and hand pronation-supination movement. These items require Parkinson's patients to perform the stage action as continuously and quickly as possible with the maximum amplitude, and then the evaluator assesses the speed, amplitude, gradual reduction of amplitude, and hesitation and pause of the action, and scores according to 0-4 five degrees of severity. However, since the evaluation process is essentially subjective, it is crucially dependent on the training and experience of the evaluator, resulting in a large degree of difference between different evaluators. In addition, due to the size and subtlety of hand movements, the human eye may not be able to capture every detail of the Parkinson's patient under rapid motor examination.

[0003] Existing PD motor slowness quantitative analysis methods mostly rely on wearable sensors (such as accelerometers, gyroscopes, magnetometers, etc.) to quantify motion characteristics such as amplitude, speed, acceleration, etc. However, professional sensors are expensive and complex to operate, which is not convenient for large-scale popularization and application. Recently, pose estimation technology based on deep learning has made significant progress, which can locate the key points of the patient's hand from the PD clinical video in a completely non-contact form. These key point sequences contain rich spatio-temporal information and represent the motor slowness symptoms in a compact form. However, due to the high flexibility of the hand joints, the severe self-occlusion, and the limited size and flexibility of the public hand gesture dataset, hand pose estimation is more challenging than other body parts. In addition, due to the low resolution of hand images, in order to avoid false detection of complex backgrounds, it is usually necessary to additionally deploy a deep network model to locate the hand RoI region and perform non-end-to-end pose estimation by separately cropping it.

[0004] In view of the above problems, many research works attempt to use OpenPose (an open-source pose estimation framework) as an intuitive solution. However, the OpenPose model trained on public hand gesture datasets cannot effectively cope with the unique challenges of PD clinical videos, such as clinically defined hand gestures and obvious self-occlusion. Secondly, as a bottom-up pose estimation model, OpenPose relies on human key points and hand-crafted rules to roughly estimate the hand RoI region of PD patients, making the quantitative process cumbersome. SUMMARY

[0005] The application aims to provide a Parkinson's motor slowness quantification analysis method based on hand gesture video, which can take hand gesture video clips as carriers to realize flexible and interpretable motor slowness quantification, thereby assisting early detection, regular monitoring and treatment evaluation of Parkinson's disease.

[0006] The application adopts the following technical solutions:

[0007] A Parkinson's motor slowness quantification analysis method based on hand gesture video comprises the following steps:

[0008] A: constructing a hand gesture dataset for PD clinical video hand gesture estimation;

[0009] B: constructing a Mask R-CNN deep neural network model, which comprises a feature extraction network, a region candidate network, a bounding box head network and a key point head network;

[0010] The feature extraction network is used for feature extraction of original images in the input hand gesture dataset, and fusion of different scale features, and the fused five different scale feature maps are sent to the region candidate network;

[0011] The region candidate network is used for generating region candidates according to the set initial anchor frame; then, the region candidates and the first four scale feature maps with the largest resolution are combined to generate two different resolution RoI feature maps, which are respectively sent to the parallelly arranged bounding box head network and key point head network;

[0012] The bounding box head network is used for hand target classification and bounding box regression according to the input RoI feature map, and outputs accurate hand bounding boxes;

[0013] The key point head network is used for hand target key point detection of the input RoI feature map, and finally outputs 21 hand key points;

[0014] C: training the constructed Mask R-CNN deep neural network model by using the training set in the hand gesture dataset, and obtaining a key point sequence composed of 21 hand key points of the subject in the hand gesture video;

[0015] D: extracting motion features from the key point sequence obtained in step C to obtain corresponding motion features;

[0016] E: obtaining a wave peak sequence and a wave trough sequence representing periodic action changes of the subject in the hand gesture video according to the motion features obtained in step D;

[0017] F: Based on the peak sequence and the trough sequence obtained in step E, five movement retardation parameters are quantified, and the five movement retardation parameters are slow speed, low vibration, amplitude attenuation, hesitation and pause.

[0018] The step A comprises the following specific steps:

[0019] A1: using a video acquisition device, collecting hand posture videos of the subject performing specified actions; the specified actions refer to that the subject performs three types of hand movement retardation check items of finger tapping, hand movement and hand pronation-supination movement according to the requirements in the MDS-UPDRS evaluation scale;

[0020] A2: randomly shuffle all image frames in the hand posture video, uniformly sample N image frames to construct a hand posture dataset; then randomly divide the hand posture dataset into a training set, a validation set and a test set according to a proportion, and finally manually label the hand bounding box and 21 key points.

[0021] In step B, the feature extraction network adopts ResNet-50-FPN network, extracts features from the original image, and combines the feature pyramid structure to fuse different scale features to generate P2, P3, P4, P5 and P6 five different scale feature maps, which are sent to the region candidate network.

[0022] In step B, the region candidate network sets three initial anchor boxes with different width-height ratios for all grid points in the P2, P3, P4, P5 and P6 five feature maps, and uses the RPN network to perform boundary box regression and binary classification on the initial anchor boxes to obtain region candidates; then, combined with the region candidates and the P2, P3, P4, P5 four feature maps with the largest resolution, the RoIAlign operation is performed to generate two RoI feature maps with different resolutions, which are sent into the parallel set boundary box head network and key point head network.

[0023] In step B, the boundary box head network comprises a first convolutional layer and a first fully connected layer from left to right, and the boundary box head network generates boundary box regression offset and hand classification softmax score according to the RoI feature map output by the region candidate network; then, according to the boundary box regression offset and the classification softmax score, an accurate hand bounding box is generated.

[0024] In step B, the key point head network comprises a feature extraction subnetwork, a distribution expectation decoding subnetwork and a distribution variance decoding subnetwork.

[0025] The feature extraction sub-network is used for feature extraction on the RoI feature map output by the region candidate network. The feature extraction sub-network first maps the ROI feature map to a feature map f1 through a second convolutional layer, and then further extracts features from the feature map f1 and maps the feature map f1 to a feature map f2 through a third convolutional layer;

[0026] The distribution expectation decoding sub-network is used for decoding the expectation of the underlying distribution of the key points The distribution expectation decoding sub-network first changes the channel number of the feature map f2 through a fourth convolutional layer and maps the feature map f2 to a feature map f3. The distribution expectation decoding sub-network then uses a 2x bilinear upsampling module to upsample the feature map f3 and output a probability map π. Finally, the distribution expectation decoding sub-network adopts a Soft-Argmax operation on the probability map π to decode the distribution expectation based on the following definition That is, the 21 key point coordinates of the hand 2:

[0027]

[0028] Wherein, π yi represents the probability of the predicted target appearing in y i , y i represents the discrete grid position of the probability map π;

[0029] The distribution variance decoding sub-network is used for decoding the variance of the underlying distribution of the key points The distribution variance decoding sub-network first uses a global average pooling operation on the feature map f2 to generate a one-dimensional vector v1, and then takes the one-dimensional vector v1 as the input of a second fully connected layer. Finally, the bias

[0030] In step B, during the network training process, the expectation output by the distribution expectation decoding sub-network and the variance output by the distribution variance decoding sub-network are used to model the underlying distribution of the key points using RLE Loss. The definition of RLE Loss is as follows:

[0031]

[0032] Wherein, μ g represents the key point coordinate label; represents a standard Gaussian distribution; represents the underlying distribution of the key points,

[0033] In step D, the motion feature extraction method is as follows:

[0034] In a given key point sequence Next, three different motion feature extraction methods are defined for the three types of hand movement slowness examination items of finger tapping, hand movement and hand pronation-supination movement in MDS-UPDRS;

[0035] 1) The motion feature extraction method of finger tapping is to calculate the Euclidean distance s f.t. (t) between the fingertip of the index finger and the fingertip of the thumb.

[0036]

[0037] Wherein, p4(t) represents the index finger fingertip key point coordinate, and p8(t) represents the thumb fingertip key point coordinate. The two norm formula is represented by t, which represents the video frame number.

[0038] 2) The motion feature extraction method of hand movement is to calculate the average value s h.m. (t) of the Euclidean distance between the five finger fingertips and the palm key point.

[0039]

[0040] Wherein, p i＝4,8,12,16,20 (t) respectively represent the five finger fingertip key point coordinates, and p0(t) represents the palm key point coordinate.

[0041] 3) The motion feature extraction method of hand pronation-supination movement is to calculate the horizontal offset s p.s. (t) of the little finger to the thumb key point.

[0042] s p.s. (t) = median{x1(t), x2(t) x3(t), x4(t)}

[0043] -median{x 17 (t), x 18 (t) x 19 (t), x 20 (t)}.

[0044] Wherein, x 1,2,3,4 (t) respectively represent the horizontal coordinates of the four key points on the thumb, and x 17,18,19,20 (t) respectively represent the horizontal coordinates of the four key points on the little finger, and median represents the sequence median.

[0045] The step E includes the following specific steps:

[0046] E1: First, normalize the motion features obtained in step D;

[0047] The hand boundary box diagonal length sequence output by the given boundary box head network The normalized motion characteristics of finger tapping, hand movement, and hand pronation-supination are expressed as follows:

[0048]

[0049] Where scale is the scaling factor;

[0050] E2: Apply Savitzky-Golay filtering to the three normalized motion features obtained in step E1, and the filtered three motion features are uniformly represented as follows:

[0051] Regarding the obtained s ft.norm. (t) When applying Savitzky-Golay filtering, the window width is 11 and the fitting order is 2;

[0052] Regarding the obtained s hm.norm. (t) When applying Savitzky-Golay filtering, the window width is 15 and the fitting order is 2;

[0053] Regarding the obtained s ps.norm. (t) When applying Savitzky-Golay filtering, the window width is 13 and the fitting order is 2;

[0054] E3: Filter the three motion features obtained in step E2. Treating the signal as a quasi-periodic signal, a multi-scale-based automatic peak detection algorithm is used for peak detection, yielding the corresponding peak sequences. and trough sequence Where p1 represents the first peak and v1 represents the first trough.

[0055] Step F includes the following specific steps:

[0056] F1: Based on the peak sequence obtained in step E and trough sequence Consider a peak-valley-peak cycle as one motion cycle; given a sampling frequency f, the frequency r of the i-th motion cycle is... i and amplitude a i The calculation formula is as follows:

[0057]

[0058]

[0059] in, and These represent the peak points p and p, respectively. i+1 and p i The corresponding motion feature values; representing a trough point v i corresponding motion feature value;

[0060] F2: According to the obtained frequency r i and amplitude a i , the quantification of five motion retardation parameters is defined respectively as:

[0061] Slow: Median value in sequence;

[0062] Low vibration: Median value in sequence;

[0063] Amplitude attenuation:

[0064] Hesitation: Minimum value in sequence;

[0065] Pause: Minimum value in sequence.

[0066] The present application has the following beneficial effects:

[0067] 1) The present application constructs a hand gesture dataset in a specific field, which focuses on the unique challenges of hand gestures in the clinical examination video of the subject performing the MDS-UPDRS related project, such as clinically defined gestures, video shooting angles (hand as much as possible in the central position), special self-occlusion and perspective effects (self-occlusion and perspective effects of hand gestures when the subject makes a specific action);

[0068] 2) Construct Mask R-CNN model to realize end-to-end hand gesture estimation, and redesign the key point head network structure, introduce Soft-Argmax and RLE Loss method to improve the key point regression accuracy;

[0069] 3) By analyzing the motion timing of hand key points in PD clinical video, five typical motion retardation feature parameters are quantified, which comprehensively describe the clinical symptoms described in MDS-UPDRS score, and can provide more objective and specific explanation about motion retardation of PD patients for clinicians. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 is a flowchart of the present application. DETAILED DESCRIPTION

[0071] The present application will be described in detail below in combination with the drawings and examples:

[0072] As Figure 1 shown, the Parkinson's motion retardation quantification analysis method based on hand gesture video according to the present application comprises the following steps:

[0073] A: constructing a hand gesture dataset for PD clinical video hand gesture estimation;

[0074] The step A comprises the following specific steps:

[0075] A1: using a video acquisition device to acquire a hand gesture video of a subject performing a specified action; the specified action refers to the subject performing three types of hand movement slowness check items of finger tapping, hand movement and hand pronation-supination movement of the left and right hands according to the requirements in the MDS-UPDRS evaluation scale;

[0076] Wherein, the subject is a PD patient. When performing the hand tapping action, the subject taps the thumb with the index finger 10 times at the maximum amplitude and the fastest speed; when performing the hand movement (fist experiment) action, the subject tightly clenches the fist with the elbow bent, fully opens the palm, and repeatedly stretches the palm to clench the fist 10 times at the fastest speed; when performing the hand pronation-supination movement (alternation experiment) action, the subject stretches the arm forward, with the palm downward, and alternately flips the palm 10 times at the fastest speed and the maximum amplitude; the above action requirements are the requirements in the existing MDS-UPDRS evaluation scale, which will not be described here.

[0077] In this embodiment, the video acquisition device can be a smart phone, and the shooting resolution of the video acquisition device is not less than 640x480, and the frame rate is 30 / fps. When acquiring the video, it is required that only one hand instance of the subject performing the action is included in the hand gesture video picture, and it is as possible as possible to be in the central position;

[0078] A2: using the random.shuffle function of Python to randomly shuffle all image frames in the hand gesture video, uniformly sampling N image frames to construct a hand gesture dataset; then randomly dividing the hand gesture dataset into a training set (PH-train), a verification set (PH-val) and a test set (PH-test) according to a ratio of 6:2:2, and finally using the open source key point annotation tool coco-annotator to manually annotate the hand boundary box and 21 key points.

[0079] In this embodiment, the value of N can be 1906. The hand boundary box is a rectangular box closely surrounding the hand position, which is determined by the left upper corner coordinate (x1, y1) and the right lower corner coordinate (x2, y2); the 21 key points of the hand are 21 skeletal joints of the hand. The hand boundary box and the 21 key points are conventional techniques in the art, which will not be described here.

[0080] B: constructing a Mask R-CNN deep neural network model, the Mask R-CNN deep neural network model comprising a feature extraction network, a region candidate network, a boundary box head network and a key point head network;

[0081] The feature extraction network is used for feature extraction of original images in the input hand posture data set, and different scale features are fused in combination with a feature pyramid (FPN) structure, and the fused five different scale feature maps are input into a region candidate network;

[0082] The region candidate network is used for boundary box regression and binary classification of the set initial anchor frame, to generate a region candidate; then, in combination with the region candidate and the first four scale feature maps with the largest resolution, the RoIAlign operation is performed to generate two different resolution RoI (region of interest) feature maps and input into the parallelly arranged boundary box head network and key point head network;

[0083] The boundary box head network is used for hand target classification and further boundary box regression of the input RoI feature map, and finally outputs an accurate hand boundary box;

[0084] The key point head network is used for hand target key point detection of the input RoI feature map, and finally outputs 21 key points of the hand;

[0085] In the application, the Mask R-CNN deep neural network is constructed based on the Detectron2 framework.

[0086] The feature extraction network adopts the ResNet-50-FPN network, and the original image is extracted and multi-scale feature fusion through the top-down, bottom-up and horizontal connection mode to generate five different scale feature maps P2, P3, P4, P5 and P6, which are input into the region candidate network.

[0087] The region candidate network sets three initial anchor frames with different width-height ratios for all grid points in the five feature maps P2, P3, P4, P5 and P6, and uses the RPN network to perform boundary box regression and binary (positive and negative) classification on the initial anchor frame to obtain a region candidate; then, in combination with the region candidate and the P2, P3, P4, P5 four feature maps (i.e. the first four feature maps with the largest resolution in the five feature maps), the RoIAlign operation is performed to generate two different resolution RoI feature maps 7x7 and 14x14, which are input into the parallelly arranged boundary box head network and key point head network.

[0088] The bounding box head network comprises, from left to right, four first convolutional layers with a size of 3*3*256*256 stacked and two first fully connected layers with a size of 1024*1024, and the bounding box head network generates a bounding box regression offset with a dimension of 4k and a hand classification softmax score according to a 7*7 resolution RoI feature map output by the region proposal network, wherein k is the number of detected bounding boxes; and then, according to the bounding box regression offset and the classification softmax score, an accurate hand bounding box is generated,

[0089] In the key point head network, a Soft-Argmax and RLE Loss method are introduced to improve the key point regression accuracy; the key point head network comprises a feature extraction subnetwork, a distribution expectation decoding subnetwork and a distribution variance decoding subnetwork.

[0090] The feature extraction subnetwork is used for feature extraction on the RoI feature map output by the region proposal network, and the feature extraction subnetwork first maps the ROI feature map with a size of 14*14*256 to a feature map f1 with a size of 14*14*512 through a second convolutional layer with a size of 3*3*256*512, then further extracts features of the feature map f1 through eight third convolutional layers with a size of 3*3*512*512 stacked, and maps the feature map f1 to a feature map f2 with a size of 14*14*512.

[0091] The distribution expectation decoding subnetwork is used for decoding the expectation of the bottom layer distribution position of the key point , that is, the numerical coordinate of the key point; the distribution expectation decoding subnetwork first maps the feature map f2 to a feature map f3 with a size of 14*14*21 through a fourth convolutional layer with a size of 1*1*512*21, so as to change the channel number to 21, which is consistent with the number of hand key points. In order to generate a high-resolution feature map, the distribution expectation decoding subnetwork further uses a 2 times bilinear upsampling module to upsample the feature map f3, and outputs a probability map π with a size of 28*28*21; finally, the distribution expectation decoding subnetwork adopts a Soft-Argmax operation on the probability map π, and decodes the distribution expectation based on the following definition , that is, the 21 key point coordinates of the hand:

[0092]

[0093] , wherein, π yi represents a probability that the prediction target appears at y i , y i represents a discrete grid position of the probability map π.

[0094] The distribution variance decoding subnetwork is used for decoding the variance of the bottom layer distribution scale of the key point The distribution variance decoding subnetwork first performs a global average pooling (GAP) operation on the feature map f2 to realize dimension reduction, to generate a one-dimensional vector v1 with a dimension of 512, then takes the one-dimensional vector v1 as the input of a second fully connected layer with a dimension of 512x42, and finally generates a bias with a dimension of 42 through a Sigmoid activation function

[0095] From the perspective of maximum likelihood estimation, the commonly used L1 or L2 loss function for key point coordinate regression is based on the assumption of a specific output probability distribution, so that the output obeys the Laplace distribution and the Gaussian distribution respectively. However, this unreferenced basic distribution limits the performance of regression. Therefore, during the network training process, the present application is based on the expectation and the variance output by the distribution variance decoding subnetwork uses RLE Loss to model the underlying distribution of key points, thereby facilitating the learning process. The definition of RLE Loss is as follows:

[0096]

[0097] wherein, μ g represents the key point coordinate label; represents a standard Gaussian distribution; represents the underlying distribution of key points, controls the position of the underlying distribution of key points, controls the scale of the distribution; Riemann sums can be used;

[0098] C: using the training set in the hand pose data set in step A to train the Mask R-CNN deep neural network model constructed in step B, to obtain a key point sequence composed of 21 key points of the subject's hand in the hand pose video

[0099] In step C, the Mask R-CNN model is implemented based on the Detectron2 framework, the backbone network uses the ResNet-50-FPN network, and is pre-trained on ImageNet; the stochastic gradient descent (SGD) optimizer is used, with an initial learning rate of 1e-3, and a 10-fold reduction at the 10K and 20K iterations (a total of 30K iterations); the weight decay rate, batch_size and momentum parameters are set to 0.0001, 16 and 0.9 respectively.

[0100] D: performing motion feature extraction on the key point sequence obtained in step C to obtain the corresponding motion features;

[0101] In step D, the motion feature extraction method is as follows:

[0102] According to the examination requirements of the MDS-UPDRS on the three types of hand movement slowness examination items of finger tapping, hand movement and hand pronation-supination movement, three different motion feature extraction methods are defined respectively; in the given key point sequence ,

[0103] 1) The motion feature extraction method of finger tapping is to calculate the Euclidean distance s f.t. (t) between the fingertip of the index finger and the fingertip of the thumb.

[0104]

[0105] Where p4(t) represents the key point coordinates of the index finger tip, and p8(t) represents the key point coordinates of the thumb tip. represents the two norm formula, and t represents the video frame number.

[0106] 2) The motion feature extraction method of hand movement is to calculate the average value s h.m. (t) of the Euclidean distances between the five finger tip key points and the palm key point.

[0107]

[0108] Where p i＝4,8,12,16,20 (t) respectively represent the key point coordinates of the five finger tip points, and p0(t) represents the key point coordinates of the palm.

[0109] 3) The motion feature extraction method of hand pronation-supination movement is to calculate the horizontal offset s p.s. (t) of the little finger to the thumb key point.

[0110] s p.s. (t) = median{x1(t), x2(t) x3(t), x4(t)}

[0111] s 17 (t) = median{x1(t), x2(t) x3(t), x4(t)} 18 (t), x 19 (t), x 20 (t)} (5)

[0112] Where x 1,2,3,4 (t) respectively represent the horizontal coordinates of the four key points on the thumb, and x 17,18,19,20 (t) respectively represent the horizontal coordinates of the four key points on the little finger.

[0113] E: Normalizing, Savitzky-Golay smoothing filtering and AMPD peak detection processing are performed on the motion features obtained in step D respectively, to obtain a wave peak sequence and a wave trough sequence representing periodic motion changes of the subject in the hand gesture video;

[0114] In the present application, step E includes the following specific steps:

[0115] E1: In the present application, the motion features obtained in step D are first normalized, because the distance between the video acquisition device and the subject's hand varies in different hand gesture videos, which inevitably causes changes in the original spatial values.

[0116] Hand boundary box diagonal length sequence output by the given boundary box head network The normalized motion features of finger tapping, hand motion and hand pronation-supination motion are represented as:

[0117]

[0118] wherein scale is a scale factor.

[0119] E2: Due to inevitable small prediction errors or discontinuous data labeling, the extracted motion features are prone to slight jitter (spikes, sawtooth high-frequency noise), which will cause the appearance of false local extrema. Therefore, in the present application, Savitzky-Golay filtering is also applied to the three normalized motion features obtained in step E1, with the following parameter settings:

[0120] For the obtained s ft.norm. When Savitzky-Golay filtering is applied to s

[0121] For the obtained s hm.norm. When Savitzky-Golay filtering is applied to s

[0122] For the obtained s ps.norm. When Savitzky-Golay filtering is applied to s

[0123] The three filtered motion features are uniformly represented as

[0124] E3: The three filtered motion features obtained in step E2 are subjected to AMPD peak detection processing, to obtain a wave peak sequence and a wave trough sequence representing periodic motion changes of the subject in the hand gesture video. Treating the signal as a quasi-periodic signal, peak detection was performed using an Automatic Multiscale-based Peak Detection (AMPD) algorithm, yielding the corresponding peak sequences. and trough sequence

[0125] Where p1 represents the first peak and v1 represents the first trough.

[0126] F: Based on the peak sequence obtained in step E and trough sequence Five motion sluggishness parameters were quantified, namely slow speed, low vibration, amplitude decay, hesitation, and pause.

[0127] In this invention, step F includes the following specific steps:

[0128] F1: Based on the peak sequence obtained in step E and trough sequence Consider a peak-valley-peak (PVP) cycle as one motion cycle; given the sampling frequency f, i.e. the frame rate of the hand pose video, the frequency r of the i-th motion cycle is... i and amplitude a i The calculation formula is as follows:

[0129]

[0130]

[0131] in, and These represent the peak points p and p, respectively. i+1 and p i The corresponding motion feature values; Represents the trough point v i The corresponding motion feature values;

[0132] F2: Based on the obtained frequency r i and amplitude a i The quantization of the five motion sluggishness parameters is defined as follows:

[0133] Slow: Values ​​in the sequence;

[0134] Low vibration: Values ​​in the sequence;

[0135] Amplitude attenuation:

[0136] hesitate: Minimum value of the sequence;

[0137] pause: sequence minimum.

Claims

1. A method for quantitatively analyzing Parkinsonian bradykinesia based on hand gesture video, characterized in that, The method comprises the following steps: A: constructing a hand posture dataset for PD clinical video hand posture estimation; B: constructing a Mask R-CNN deep neural network model, the Mask R-CNN deep neural network model comprising a feature extraction network, a region candidate network, a bounding box head network, and a key point head network; The feature extraction network is used for feature extraction on the original images in the input hand posture dataset, and different scale features are fused, and the fused five different scale feature maps are sent to the region candidate network; The region candidate network is used to generate region candidates according to the set initial anchor box; then, region candidates and the first four scale feature maps with the largest resolution are combined to generate two different resolution RoI feature maps, which are respectively sent to the parallelly arranged bounding box head network and key point head network; The bounding box head network is used for hand target classification and bounding box regression according to the input RoI feature map, and outputs an accurate hand bounding box; The key point head network is used for hand target key point detection on the input RoI feature map, and finally outputs 21 hand key points; The key point head network comprises a feature extraction subnetwork, a distribution expectation decoding subnetwork, and a distribution variance decoding subnetwork; The feature extraction sub-network is used for feature extraction on the RoI feature map output by the region candidate network. The feature extraction sub-network first maps the RoI feature map to a feature map through a second convolutional layer , and then further extracts features of the feature map through a third convolutional layer and maps the feature map to a feature map . The distribution expectation decoding subnetwork is used for decoding the expectation of controlling the position of the key points of the bottom layer distribution The distribution expectation decoding subnetwork first changes the channel number of the feature map by a fourth convolutional layer and maps the feature map to a feature map using a 2 times bilinear up-sampling module for up-sampling, and outputs a probability map Finally, the distribution expectation decoding subnetwork adopts a Soft-Argmax operation to decode the distribution expectation based on the following definition , that is, 21 key point coordinates of a hand ​ ； wherein, represents a probability that a prediction target appears at a discrete grid location of a probability map represents a probability map at a discrete grid location of a probability map The distribution variance decoding sub-network is used for decoding a variance of a bottom distribution scale of a control key point The distribution variance decoding sub-network first processes a feature map A global average pooling operation is adopted to realize dimension reduction, and a generated one-dimensional vector is input into a second full connection layer, and finally a deviation generated through a Sigmoid activation function ; During network training, the expectation based on the distribution expectation decoding subnetwork output and the variance based on the distribution variance decoding subnetwork output , the RLE Loss is used to model the key point underlying distribution, and the definition of RLE Loss is: ； wherein, , denotes a keypoint coordinate label; = denotes a standard Gaussian distribution; denotes a keypoint bottom distribution, ;​ C: training the constructed Mask R-CNN deep neural network model using the training set in the hand posture dataset, to obtain a key point sequence composed of 21 hand key points of a subject in a hand posture video; D: extracting motion features from the key point sequence obtained in step C to obtain corresponding motion features; E: obtaining a wave peak sequence and a wave trough sequence representing periodic action changes of a subject in a hand posture video according to the motion features obtained in step D; F: quantifying five motion retardation parameters based on the wave peak sequence and the wave trough sequence obtained in step E, the five motion retardation parameters being slow speed, low vibration, amplitude attenuation, hesitation, and pause.

2. The method of claim 1, wherein the method comprises: The step A comprises the following specific steps: A1: using a video acquisition device to collect hand posture videos of a subject performing specified actions; the specified actions refer to the subject performing three types of hand motion retardation check items of finger tapping, hand movement, and hand pronation-supination movement according to the requirements in the MDS-UPDRS evaluation scale; A2: randomly shuffling all image frames in the hand posture videos, uniformly sampling N image frames to construct a hand posture dataset; then, randomly dividing the hand posture dataset into a training set, a validation set, and a test set according to a proportion, and finally manually labeling hand bounding boxes and 21 key points.

3. The method of claim 1, wherein: In step B, the feature extraction network adopts a ResNet-50-FPN network to extract features from the original images and fuse different scale features in combination with a feature pyramid structure to generate five different scale feature maps P2, P3, P4, P5, and P6, which are sent to the region candidate network.

4. The method of claim 3, wherein the method further comprises: In step B, the region proposal network sets three initial anchor boxes with different width-height ratios for all grid points in the five feature maps P2, P3, P4, P5 and P6, adopts the RPN network to perform boundary box regression and binary classification on the initial anchor boxes, and obtains region proposals; then, the region proposals and the four feature maps P2, P3, P4 and P5 with the largest resolution are combined to perform the RoIAlign operation, thereby generating two RoI feature maps with different resolutions, which are respectively sent to the boundary box head network and the key point head network arranged in parallel.

5. The method of claim 1, wherein: In step B, the boundary box head network comprises a first convolutional layer and a first fully connected layer from left to right, and the boundary box head network generates boundary box regression offset and hand classification softmax score according to the RoI feature map output by the region proposal network; then, the accurate hand boundary box is generated according to the boundary box regression offset and the classification softmax score. 6.The method of claim 1, wherein, In step D, the motion feature extraction method is as follows: In a given sequence of key points Next, for the MDS-UPDRS, three different motion feature extraction methods are defined for the examination requirements of the finger tapping, hand movements, and hand pronation-supination movements. 1) The motion feature extraction method of finger tapping is: calculating the Euclidean distance between the tip of the index finger and the tip of the thumb ; ； wherein, represents the index finger tip key point coordinate, represents the thumb tip key point coordinate; represents the two-norm formula, represents the video frame number; 2) The motion feature extraction method of hand motion is to calculate the average value of the Euclidean distance between the fingertips of the five fingers and the palm key points ; ； wherein, respectively represent the coordinates of the fingertips of the five fingers, represents the coordinates of the palm key point; 3) The motion feature extraction method of the pronation-supination movement is to calculate the horizontal offset of the little finger to the thumb key point ; in, These represent the x-coordinates of four key points on the thumb. These represent the x-coordinates of the four key points on the little finger, and median indicates taking the midpoint of the sequence.

7. The method of claim 6, wherein the method further comprises: Step E comprises the following specific steps: E1: first, the motion feature obtained in step D is normalized; Hand bounding box skew side length sequence given bounding box head network output The normalized motion features of the finger tapping, hand motion, and hand pronation-supination motion are represented as follows, respectively: ； ； ； wherein is a scale factor; E2: applying Savitzky-Golay filter to the normalized three motion features obtained in step El respectively, and the filtered three motion features are uniformly expressed as ; For the obtained When applying Savitzky-Golay filtering, the window width is 11 and the fitting order is 2. For the obtained When applying Savitzky-Golay filtering, the window width is 15 and the fitting order is 2. For the obtained When applying Savitzky-Golay filtering, the window width is 13 and the fitting order is 2. E3: the filtered three motion features obtained in step E2 are combined to obtain a final motion feature vector The peak detection is performed by using a multi-scale based automatic peak detection algorithm, and corresponding peak sequences and valley sequences are obtained ; wherein, the first peak is represented by P1, the first valley is represented by V1.​ 8. The method of claim 7, wherein the method further comprises: Step F comprises the following specific steps: F1: Based on the peak sequence obtained in step E and trough sequence One "peak-valley-peak" cycle is considered as one motion cycle; given a sampling frequency Then the first Frequency of each motion cycle and amplitude The calculation formula is as follows: ； ； wherein, and represent a wave peak point and corresponding motion characteristic value; represent a wave trough point corresponding motion characteristic value; F2: the frequency obtained and amplitude quantifying five parameters of motor slowness, respectively: Slow: Median value in sequence; Low vibration: Sequence median value; Amplitude decay: ; Hesitation: Sequence minimum; Pauses: Sequence minimum.

Images