Hand tremor detection method based on smart phone camera and image convolution

Abstract

A hand tremor detection method based on a smartphone camera and image convolution belongs to the technical field of identification and detection, and comprises the following steps: shooting real-time image data of hand tremor of a PD patient through a built-in camera of a smartphone, carrying out a series of preprocessing on original data, and constructing a data set; and training a deep learning model to predict tremor types and severity division, and comparing and analyzing model effects. According to the method, a method of combining video graying and moving average interpolation is adopted, and noise interference in original image data is effectively removed. Meanwhile, a data enhancement mode is adopted, data features are enriched, and the generalization ability of a subsequent model is effectively improved. According to the method, the GCN iteration block is taken as a basic framework, and the GRU and the graph attention mechanism GAT are introduced, so that the feature learning efficiency of the model is effectively improved. According to the method, starting from the multi-classification task characteristic of a deep learning model, ACC and macro F1 scores are taken as main performance indexes, and five-fold cross validation is adopted, so that the superiority and robustness of the model are fully verified.

Description

Technical Field

[0001] The present invention belongs to the technical field of recognition and detection, and specifically relates to a hand tremor detection method based on a smartphone camera and graph convolution. Background Art

[0002] Parkinson's disease is a common neurodegenerative disease with typical symptoms including resting tremor, muscle rigidity, bradykinesia, etc. Among them, hand tremor is the most intuitive and one of the earliest symptoms, which has a serious impact on the patient's quality of life. Therefore, developing an efficient and convenient hand tremor detection method is of great significance for early diagnosis and disease monitoring.

[0003] In recent years, with the rapid development of smartphone technology, real-time detection of Parkinson's disease hand tremor has become possible by using the built-in camera of the mobile phone and deep learning technology, especially using graph convolutional neural networks (GCNs) to simulate the connection of key bone points and perform feature learning. This method can not only overcome the high cost and inconvenience of traditional medical equipment, but also provide relatively accurate test results, facilitate daily self-monitoring of patients and remote monitoring of doctors, and achieve a remote medical rehabilitation model. Summary of the invention

[0004] In order to solve the above-mentioned problems, the present invention proposes: a hand tremor detection method based on smartphone camera and graph convolution, which uses the built-in camera of the smartphone to capture real-time image data of hand tremor in PD patients, performs a series of preprocessing on the original data and constructs a data set, trains a deep learning model to predict tremor type and severity classification, and compares and analyzes the model effect.

[0005] Further, the following steps are included:

[0006] S1. Data collection: The hand tremor image data of PD patients were collected through the smartphone camera;

[0007] S2. Data preprocessing: grayscale and average interpolation processing are performed on the original image data. The three-dimensional coordinate points of the hand are extracted using the mediapipe_hands model, and interpolation and averaging are performed to calculate the effective hand motion parameters of acceleration, angular velocity, amplitude, and frequency, and data enhancement is performed to finally construct a multi-dimensional data set;

[0008] S3, feature extraction;

[0009] S4, build the dataset;

[0010] S5. Model building: Building a deep learning model based on TGCN-GRU-GraphAttention;

[0011] S6. Model training evaluation: Use 5-fold cross validation and take the average accuracy of the total folds. Use the ACC and F1 score indicators of the trained model on the test set to judge the accuracy performance of the model, and compare the results with existing models in the current field.

[0012] Furthermore, the step S2 includes the following steps:

[0013] Grayscale the collected video data to reduce the impact of light on the collection of video features;

[0014] The mediapipe_hands model is used to extract the three-dimensional coordinates of the key points of the hands, and the sliding average interpolation method is used to process the original coordinate data; in the time series data, for each time point, the data of the nearby points are selected and the average value is calculated as the new value of the point:

[0015]

[0016] Among them, X new [n] is the new data point, X[n] is the original data sequence, M is the window size, which indicates how many nearby points are taken to calculate the average, n is the current time point, and k is a loop variable from 0 to M-1;

[0017] Each hand of the human body is divided into 21 joints. The MH model is exported as the three-dimensional coordinates of the 21 key points of each frame and saved in a JSON file.

[0018] The acceleration, angular velocity, amplitude, frequency, and tremor extremes of each sample are calculated using the key point coordinates.

[0019] Furthermore, the calculation formulas are as follows:

[0020] Given two points P1(x1,y1,z1) and P2 in three-dimensional space ( x2, y2, z2), the Euclidean distance D between them is calculated by the following formula:

[0021]

[0022] Acceleration a is the rate of change of angular velocity with time and is obtained by taking the first-order difference of the angular velocity:

[0023]

[0024] Here, fps is the number of frames per second, which is used to convert the time interval into seconds; the angular velocity ω is calculated based on the change in the position of the key point between adjacent frames, which reflects the speed at which the key point rotates along a certain axis:

[0025]

[0026] where v x,i 、v y,i 、v z,i is the velocity of the key point in the x, y, and z directions;

[0027] The tremor extreme point refers to the change in the maximum distance between all key points between two adjacent frames in the entire video. The calculation method is as follows:

[0028]

[0029] D max =max(D i )

[0030] Where D i is the maximum distance change between the i-th frame and the i+1-th frame, D max All D i The maximum value in ;

[0031] Amplitude refers to the difference between the maximum and minimum values ​​of all key points in three-dimensional space. For each dimension (x, y, z), the amplitude A is defined as:

[0032] A x =max(x i )-min(x i )

[0033] A y =max(y i )-min(y i )

[0034] A z =max(z i )-min(z i )

[0035] The final amplitude is the average of these three values:

[0036]

[0037] The frequency is based on the fast Fourier transform FFT to analyze the frequency of all key points changing over time. The frequency F is determined by finding the frequency corresponding to the maximum amplitude in the FFT result:

[0038] F = argmax(|FFT(x i)|,|FFT(y i )|,|FFT(z i )|)

[0039] Here, FFT means fast Fourier transform, and |·| means taking the absolute value.

[0040] Furthermore, in step S3, each consecutive frame is regarded as a mutually exclusive node, the coordinates of the 21 key points of the hand in each frame are used as static time series features, and the calculated time series data such as acceleration, frequency, and amplitude are used as dynamic time series features; they are divided into two edges, space and time, as the model input of subsequent graph convolution.

[0041] Furthermore, in step S4, the original data is concatenated with the calculated timing parameters, and the corresponding static timing features and dynamic timing features are mapped through KV. The data set consists of 4 categories, normal, mild, moderate, and severe, corresponding to four labels 0, 1, 2, and 3; the data set is randomly divided into 60% training set, 20% validation set, and 20% test set; each training is independently randomly divided.

[0042] Furthermore, in step S5, the model is established:

[0043] The topological diagram of the key points of the hand is in the form of interconnected nodes, which fits the tree graph structure. It is proposed to use the graph convolutional neural network GCN as the model baseline. The deep learning model is based on the three-layer iterative GCN model to perform feature learning on static spatial features; the calculated timing parameters are sent to the timing layer GRU network as dynamic spatial features for feature learning, and the multi-head graph attention mechanism is introduced. During the entire model training and reasoning process, different weights are automatically assigned to learning features of different importance, so that the model focuses more on deeper feature learning of prominent features of different categories. Finally, the learned features are aggregated and mapped to the fully connected layer through the LeakyRelu activation function. The features are classified by the standard Softmax classifier to achieve the purpose of classifying the severity of PD hand disease level established by the model.

[0044] The beneficial effects of the present invention are:

[0045] (1) Data collection: The present invention utilizes the built-in camera of a smartphone to conveniently collect hand tremor videos in real time. Compared with traditional wearable sensors and professional equipment such as sports cameras, smartphones are more portable, simple to operate, and low-cost, making them suitable for the general public.

[0046] (2) Data processing: The present invention adopts a method combining video grayscale and sliding average interpolation to effectively remove noise interference in the original image data. At the same time, the three-dimensional key points of the hand are extracted through the mediapipe_hands model as short-term features; and effective hand motion parameters such as acceleration, angular velocity, amplitude, and frequency are calculated as long-term features. In this way, static spatial edges and dynamic temporal edges are constructed, and data enhancement is used to enrich data features, effectively improving the generalization ability of subsequent models.

[0047] (3) Model design: This paper uses the GCN iteration block as the basic architecture and introduces the GRU and graph attention mechanism (GAT). The T-GCN iteration block acts on the static spatial edges to learn short-term features; the GRU block acts on the dynamic time edges to learn long-term features. GraphAttention automatically assigns different weights to features of different importance during the training process, effectively improving the feature learning efficiency of the model.

[0048] (4) Model evaluation: Based on the multi-classification task characteristics of the deep learning model, the present invention uses ACC and macroF1 scores as the main performance indicators, adopts 5-fold cross validation, and draws the overall multi-class classification ROC curve. The total fold average accuracy is used to judge the model performance, and the effect is compared with the existing models in the current field. The superiority and robustness of the model of the present invention are fully verified. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of data collection for mobile phone cameras;

[0050] Figure 2 It is the mediapipe_hands model diagram;

[0051] Figure 3 It is the topological map of the key points of the mh hand;

[0052] Figure 4 This is a deep learning model architecture diagram. DETAILED DESCRIPTION

[0053] Example 1

[0054] In order to make the technical means adopted by the present invention and the purpose achieved easy to understand, the present invention is further explained in combination with a specific implementation method below. A hand tremor detection method based on a smartphone camera and graph convolution, which uses a built-in camera of a smartphone to capture real-time image data of hand tremors of PD patients, performs a series of preprocessing on the original data and constructs a data set, trains a deep learning model to predict tremor types and severity classifications, and compares and analyzes the model effects.

[0055] The main steps include:

[0056] 1. Data collection: The hand tremor image data of PD patients were collected through the smartphone camera.

[0057] 2. Data preprocessing: Grayscale and average interpolation are performed on the original image data. The three-dimensional coordinate points of the hands are extracted using the mediapipe_hands model, and interpolation and averaging are performed to calculate effective hand motion parameters such as acceleration, angular velocity, amplitude, and frequency. Data enhancement is performed to improve the generalization ability of the model and finally construct a multidimensional data set.

[0058] 3. Build a model: Build a deep learning model based on TGCN-GRU-GraphAttention.

[0059] 4. Model training evaluation: Use 5-fold cross validation and take the average accuracy of the total folds. Use indicators such as ACC and F1 score of the trained model on the test set to judge the accuracy performance of the model, and compare the results with existing models in the current field.

[0060] Example 2

[0061] A hand tremor detection method based on a smartphone camera and graph convolution specifically includes the following steps:

[0062] 1. Data collection (smartphone fixed shooting 60fps video)

[0063] Fix the mobile phone on the mobile phone holder, spread the green cloth flat on the table, and instruct the patient to perform the finger pinching action on the tremor hand. Figure 1 As shown in the figure, the entire back of the hand is fully exposed in the video shooting range, the hand tremor video is recorded for about 10 seconds, and the middle stable motion segment is captured as the original image dataset.

[0064] 2. Data processing

[0065] (1) The collected video data is grayed out to reduce the impact of light on the video feature collection.

[0066] (2) Use the mediapipe_hands model to extract the three-dimensional coordinates of the key points of the hands, such as Figure 2 As shown in the figure, due to the uncontrollable hand shaking process, some key points are missing, and the sliding average interpolation method is used to process the original coordinate data. It can be used to smooth data and reduce random missing noise. Its basic idea is to select the data of nearby points for each time point in the time series data and calculate their average value as the new value of the point (public

[0067] Formula 1).

[0068]

[0069] Among them, X new [n] is the new data point, X[n] is the original data sequence, M is the window size, which means how many nearby points are taken to calculate the average, n is the current time point, and k is a loop variable from 0 to M-1.

[0070] (3) Each hand of the human body is divided into 21 joints, such as Figure 3 As shown in the figure, the MH model is exported as 21 hand key point coordinates for each frame and saved in a JSON file.

[0071] (4) Calculate the acceleration, angular velocity, amplitude, frequency, and tremor extreme value of each sample through the coordinates of the key points of the hand. The calculation formulas are as follows:

[0072] 1) Euclidean Distance

[0073] Given two points P1 in three-dimensional space ( x1,y1,z1 ) and P2 9 x2,y2,z2 ) , the Euclidean distance D between them can be calculated by the following formula:

[0074]

[0075] 2) Acceleration

[0076] Acceleration a is the rate of change of angular velocity with time and can be obtained by taking the first-order difference of the angular velocity:

[0077]

[0078] Here fps is the number of frames per second, which is used to convert the time interval into seconds.

[0079] 3) Angular Velocity

[0080] The angular velocity ω is calculated based on the change in the position of the key point between adjacent frames, which reflects the speed at which the key point rotates along a certain axis:

[0081]

[0082] where v x,i 、v y,i 、v z,i is the velocity of the key point in the x, y, and z directions.

[0083] 4) Maximum Distance

[0084] The tremor extreme point refers to the change in the maximum distance between all key points between two adjacent frames in the entire video. The calculation method is as follows:

[0085]

[0086] D max =max(D i )

[0087] Where D i is the maximum distance change between the i-th frame and the i+1-th frame, D max All D i The maximum value in .

[0088] 5) Amplitude

[0089] Amplitude is the difference between the maximum and minimum values ​​of all key points in three-dimensional space. For each dimension (x, y, z), the amplitude A is defined as:

[0090] A x =max(x i )-min(x i )

[0091] A y =max(y i )-min(y i )

[0092] A z =max(z i )-min(z i )

[0093] The final amplitude is the average of these three values:

[0094]

[0095] 6) Frequency

[0096] The frequency is based on the Fast Fourier Transform (FFT) to analyze the frequency of all key points over time. The frequency F can be determined by finding the frequency corresponding to the maximum amplitude in the FFT result:

[0097] F = argmax(|FFT(x i )|,|FFT(y i )|,|FFT(z i )|)

[0098] Here, FFT means fast Fourier transform, and |·| means taking the absolute value.

[0099] 3. Feature extraction

[0100] Each consecutive frame is treated as a mutually exclusive node, the coordinates of the 21 key points of the hand in each frame are used as static time series features, and the calculated time series data such as acceleration, frequency, and amplitude are used as dynamic time series features. They are divided into two edges, space and time, as the model input for subsequent graph convolution.

[0101] 4. Build a dataset

[0102] The original data is concatenated with the calculated timing parameters, and the corresponding static timing features and dynamic timing features are mapped through KV to adapt to the corresponding selection of different networks in the subsequent model. The data set has 4 categories, normal, mild, moderate, and severe, corresponding to four labels 0, 1, 2, and 3. The data set is randomly divided into 60% training sets, 20% validation sets, and 20% test sets. Each training is independently randomly divided.

[0103] 5. Model building

[0104] Considering that the topology of the key points of the hand is in the form of interconnected nodes, which is more in line with the tree graph structure, we propose to use the graph convolutional neural network (GCN) as the model baseline. The effect is more prominent than the traditional convolutional network. The model diagram is as follows: Figure 4 shown.

[0105] This deep learning model uses three-layer iterative GCN as the model basis to learn the static spatial features; the calculated time series parameters are sent to the time series layer GRU network as dynamic spatial features for feature learning, and the multi-head graph attention mechanism (GraphAttention) (head=4) is introduced. During the entire model training and reasoning process, different weights are automatically and dynamically assigned to learning features of different importance, so that the model focuses more on deeper feature learning of prominent features of different categories, improving the overall prediction credibility of the model. Finally, the learned features are aggregated and mapped to the fully connected layer through the LeakyRelu activation function, and the features are classified through the standard Softmax classifier to achieve the purpose of classifying the severity of PD hand disease level established by the model.

[0106] 6. Model Evaluation

[0107] The model evaluation mainly looks at the performance results of various indicators on the test set after training. For this type of multi-classification task, we pay attention to the Acc, Auc, and F1-score scores of each category prediction, draw the multi-classification ROC curve according to the one-vs-many (OvR) strategy, and draw the confusion matrix to visualize the specific indicator results. The results show that the accuracy of each indicator is higher than that of the comparison model (the experimental results are shown in the following table).

[0108] Table 1: Network layer iteration comparison

[0109]

[0110] Table 2: Comparison of corresponding models

[0111] model ACC(+-) F1(+-) PD-GTSN 0.9232 0.8914 SPAP-NET 0.9090 0.9060 ST-GCN (body movement) 0.8775 0.8744 CNN-LSTM 0.8060 0.8044 Ours(TRGCN+GRU+TWGAT) 0.9204 0.9104

[0112] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A hand tremor detection method based on smartphone camera and graph convolution, characterized in that: Real-time image data of hand tremors in PD patients were captured using the built-in camera of a smartphone. A series of preprocessing was performed on the original data to construct a data set. A deep learning model was trained to predict the tremor type and severity classification, and the model effect was compared and analyzed.

2. The hand tremor detection method based on smartphone camera and graph convolution as claimed in claim 1, characterized in that: The following steps are involved: S1. Data collection: The hand tremor image data of PD patients were collected through the smartphone camera; S2. Data preprocessing: grayscale and average interpolation processing are performed on the original image data. The three-dimensional coordinate points of the hand are extracted using the mediapipe_hands model, and interpolation and averaging are performed to calculate the effective hand motion parameters of acceleration, angular velocity, amplitude, and frequency, and data enhancement is performed to finally construct a multi-dimensional data set; S3, feature extraction; S4, build the dataset; S5. Model building: Building a deep learning model based on TGCN-GRU-GraphAttention; S6. Model training evaluation: Use 5-fold cross validation and take the average accuracy of the total folds. Use the ACC and F1 score indicators of the trained model on the test set to judge the accuracy performance of the model, and compare the results with existing models in the current field.

3. The hand tremor detection method based on smartphone camera and graph convolution as claimed in claim 2, characterized in that: The step S2 includes the following steps: Grayscale the collected video data to reduce the impact of light on the collection of video features; The mediapipe_hands model is used to extract the three-dimensional coordinates of the key points of the hands, and the sliding average interpolation method is used to process the original coordinate data; in the time series data, for each time point, the data of the nearby points are selected and the average value is calculated as the new value of the point: Among them, X new [n] is the new data point, X[n] is the original data sequence, M is the window size, which indicates how many nearby points are taken to calculate the average, n is the current time point, and k is a loop variable from 0 to M-1; Each hand of the human body is divided into 21 joints. The MH model is exported as 21 key point coordinates of each frame and saved in a JSON file. The acceleration, angular velocity, amplitude, frequency, and tremor extremes of each sample are calculated using the key point coordinates.

4. The hand tremor detection method based on smartphone camera and graph convolution as claimed in claim 3, characterized in that: The calculation formulas are as follows: Given two points P1(x1,y1,z1) and P2(x2,y2,z2) in three-dimensional space, the Euclidean distance D between them is calculated by the following formula: Acceleration a is the rate of change of angular velocity with time and is obtained by taking the first-order difference of the angular velocity: Here, fps is the number of frames per second, which is used to convert the time interval into seconds; the angular velocity ω is calculated based on the change in the position of the key point between adjacent frames, which reflects the speed at which the key point rotates along a certain axis: where v x,i 、v y,i 、v z,i is the velocity of the key point in the x, y, and z directions; The tremor extreme point refers to the change in the maximum distance between all key points between two adjacent frames in the entire video. The calculation method is as follows: D max =max(Of) Where D i is the maximum distance change between the i-th frame and the i+1-th frame, D max Yes all D i The maximum value in ; Amplitude refers to the difference between the maximum and minimum values ​​of all key points in three-dimensional space. For each dimension (x, y, z), the amplitude A is defined as: A x =max(x i )-min(x i ) TO y =max(y i )-min(and i ) AND z =max(from i )-min(z i ) The final amplitude is the average of these three values: The frequency is based on the fast Fourier transform FFT to analyze the frequency of all key points changing over time. The frequency F is determined by finding the frequency corresponding to the maximum amplitude in the FFT result: F=argmax(|FFT(x i )|,|FFT(y i )|,|FFT(z i )|) Here, FFT means fast Fourier transform, and |·| means taking the absolute value.

5. The hand tremor detection method based on smartphone camera and graph convolution as claimed in claim 4, characterized in that: In step S3, each continuous frame is regarded as a mutually exclusive node, the coordinates of the 21 key points of the hand in each frame are regarded as static time series features, and the calculated time series data such as acceleration, vibration frequency, and amplitude are regarded as dynamic time series features; It is divided into two edges, spatial and temporal, as the model input for subsequent graph convolution.

6. The hand tremor detection method based on smartphone camera and graph convolution as claimed in claim 5, characterized in that: In step S4, the original data is concatenated with the calculated timing parameters, and the corresponding static timing features and dynamic timing features are mapped through KV. The data set has 4 categories: normal, mild, moderate, and severe, corresponding to four labels 0, 1, 2, and 3. The data set is randomly divided into 60% training sets, 20% validation sets, and 20% test sets. Each training is independently randomly divided.

7. The hand tremor detection method based on smartphone camera and graph convolution as claimed in claim 6, characterized in that: In step S5, the model is established: The topological graph of the key points of the hand is in the form of interconnected nodes, which fits the tree graph structure. It is planned to use the graph convolutional neural network GCN as the model baseline. This deep learning model is based on the three-layer iterative GCN model to perform feature learning on static spatial features. The calculated timing parameters are sent to the timing layer GRU network as dynamic spatial features for feature learning. At the same time, the multi-head graph attention mechanism is introduced. During the entire model training and reasoning process, different weights are automatically and dynamically assigned to learning features of different importance, so that the model can focus more on deeper feature learning of prominent features of different categories. Finally, the learned features are aggregated and mapped to the fully connected layer through the LeakyRelu activation function. The features are classified by the standard Softmax classifier to achieve the purpose of classifying the severity of PD hand disease established by the model.

Images