Evaluation method and system, storage medium and terminal for Parkinson's disease fist-clenching task

Abstract

The present invention provides an evaluation method and system, storage medium and terminal for a Parkinson's disease fist-clenching task, comprising the following steps: obtaining video information containing a Parkinson's disease patient's fist-clenching action; extracting hand skeleton sequence data from the video information, and preprocessing the hand skeleton sequence data; extracting hand joint point position features and hand bone vector features based on the preprocessed hand skeleton sequence; inputting the hand joint point position features and the hand bone vector features into a two-stream adaptive spatiotemporal graph convolutional neural network; inputting the fist-clenching action features output by the two-stream adaptive spatiotemporal graph convolutional neural network into a classifier to obtain an evaluation score for the Parkinson's disease fist-clenching task. The evaluation method for the Parkinson's disease fist-clenching task of the present invention is based on a tree pooling module and a spatiotemporal graph convolutional network, and realizes the reliability evaluation of the Parkinson's disease fist-clenching task.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning, and in particular to an evaluation method and system, storage medium and terminal for a Parkinson's disease fist-clenching task. Background Art

[0002] Parkinson's disease (PD) is one of the common neurodegenerative diseases that affects the body's motor system. Parkinson's disease usually occurs in people over 55 years old, with an incidence rate of about 1%. With the aging of the population, the number of Parkinson's patients will gradually increase. Experts from the World Health Organization predict that by 2030, the number of Parkinson's patients in China will reach 5 million. Parkinson's disease brings great distress and burden to patients and their families. Early diagnosis of the disease and appropriate treatment plans are the key to relieving symptoms and prolonging survival.

[0003] Studies have shown that early diagnosis of Parkinson's disease symptoms can effectively prolong the patient's survival period. Therefore, early screening and accurate diagnosis of Parkinson's disease are crucial. At present, there is no software or instrument for automatic diagnosis of Parkinson's disease movement disorders in clinical practice. Clinicians mainly rely on their own experience, the patient's clinical manifestations and the designated unified evaluation scale for scoring. Among the various rating scales, the Unified Parkinson's Disease Rating Scale (UPDRS) is the most commonly used standard rating scale for evaluating Parkinson's disease. It was proposed in the 1980s and revised by the Movement Disorder Society (MDS) in 2007 to solve the problems found before. The revised version is called MDS-UPDRS. The MDS-UPDRS scale provides clear and detailed descriptions of each test. It consists of four parts, of which the third part is the method of examining the patient's motor function. The score of this task is divided into five categories: 0: normal; 1: mild; 2: mild; 3: moderate; 4: severe. The main evaluation indicators are the amplitude and speed of the movement, the presence or absence of pauses and hesitations, the gradual decrease in amplitude, etc. In clinical practice, doctors usually score the severity of Parkinson's disease symptoms based on the evaluation scale and their own experience, so there are certain subjective differences. However, this method requires high professional skills of the assessors and has the following challenges:

[0004] 1) There are about 4 million Parkinson's disease patients in China. Each patient needs to undergo multiple UPDRS examinations every year. It takes about 30 minutes to complete a manual examination, which consumes a lot of manpower and time resources.

[0005] 2) Even among professional clinicians, the scoring results of different doctors may have subjective differences, which is not conducive to achieving objective consistency in disease diagnosis.

[0006] Therefore, if automatic assessment of Parkinson's disease symptoms can be achieved, it will save a lot of manpower and time resources for patients and doctors, reduce diagnostic costs, ensure the objectivity and consistency of scoring, and realize remote monitoring of the disease, thereby further improving the early diagnosis rate of Parkinson's disease and prolonging the survival period of Parkinson's disease patients.

[0007] At present, artificial intelligence technology has played a great auxiliary role in disease diagnosis, medical image recognition and segmentation, surgical planning and navigation, etc., which can greatly improve the efficiency of doctors' diagnosis and treatment. In addition to professional medical data such as medical images, EEG signals and gene sequences, video data is also a carrier containing rich information, especially suitable for problems such as movement assessment of Parkinson's patients. With the popularization of smart mobile phones and the advent of the 5G era, video shooting and data transmission will become more convenient and faster, which is more conducive to data collection. In addition, smart medical home will be the trend of the future. The popularization of video-based intelligent diagnosis technology will enable patients to enjoy timely, fast and high-quality medical services at home.

[0008] Automatic identification of the Parkinson's fist test can significantly improve the efficiency of diagnosis. It allows patients to self-assess their symptoms more frequently, providing clinicians with more useful information, allowing doctors to develop more appropriate treatment plans for patients. In addition, the computerized assessment of the fist test for Parkinson's patients can provide an objective measure of motor symptoms, avoiding the influence of subjective differences in awareness between patients and doctors in the UPDRS test.

[0009] In the automatic assessment scheme of Parkinson's disease movement disorder, the assessment scheme based on wearable devices is one of the mainstream methods. Common wearable devices include accelerometers, gyroscopes, magnetometers, etc. For example, Parisi et al. used a method based on human sensor network to automatically evaluate three tasks in the UPDRS scale, namely leg flexibility, sitting and gait. Manzanera et al. used a directional sensor with nine degrees of freedom to obtain a three-dimensional representation of movement, which was ultimately used to evaluate actions such as finger-to-toe and toe-slapping in the UPDRS scale. In addition, Salarian et al. installed gyroscopes on the forearms of Parkinson's patients, which can detect tremors and quantify the severity of tremors and bradykinesia. Giuffrida et al. placed an independent device (three-axis accelerometer and gyroscope) on a finger, which can predict the UPDRS score for rest, posture and dynamic tremor. Patel et al. used accelerometers during the UPDRS movement assessment task to record and analyze a comprehensive set of movement features and determine which movement tasks are most suitable for predicting tremor, bradykinesia and movement disorder severity.

[0010] Optical motion capture systems are also one of the common methods for evaluating Parkinson's disease movements. Motion evaluation methods based on optical motion capture systems generally refer to the use of optical motion capture devices such as optical cameras, depth cameras, infrared sensors, etc. combined with corresponding image processing algorithms to achieve the purpose of motion recognition. Early vision-based Parkinson's disease gait analysis used multi-color suits to assist in the segmentation of body parts. Cho et al. used background subtraction to detect the gait behavior of the PD patients frame by frame, and then used the resulting contours to classify PD patients and healthy controls. The finger-to-finger movement study conducted by Khan et al. required participants to hold their heads with both hands so that facial detection could be used to approximate their hand positions. Ajay et al. proposed an end-to-end system that does not require sensors for analyzing the gait of Parkinson's disease patients with the help of multi-view cameras and deep learning technology, and achieved a high accuracy rate in distinguishing the gait of Parkinson's disease patients from normal people. In addition, many studies use the built-in skeleton tracking of Microsoft Kinect to extract gait parameters and identify Parkinson's disease gait. Dror et al. used depth sensing to segment the hands in the process of extracting motion features from UPDRS hand movements (i.e., finger pointing test, fist test, and hand alternation test) to distinguish PD patients from healthy controls. However, there are still many limitations in the current research on Parkinson's disease movement assessment based on optical motion capture systems. The main reason is that it is easily affected by environmental, lighting, occlusion, etc., and the equipment of optical motion capture systems is relatively expensive, which is not suitable for the promotion and use of telemedicine in households.

[0011] The posture estimation method based on deep learning is an action recognition method that has just emerged in recent years. It uses deep learning algorithms to predict the position of the body or hand skeleton joints from image or video data. Based on the posture estimation algorithm, some researchers have also conducted research on the action recognition task of Parkinson's disease. Li et al. used the Convolutional Pose Machine (CPM) model to obtain the motion trajectory of Parkinson's patients, and then used the characteristics of the motion trajectory to train the random forest to achieve the automatic evaluation of Parkinson's patients' communication and drinking tasks and the leg flexibility and toe tapping tasks of patients with levodopa dyskinesia. Li et al. also conducted a quantitative analysis of levodopa dyskinesia using the CPM posture estimation method. According to the joint motion trajectory obtained by posture estimation, the motion features were extracted, and then the correlation with the clinician's scoring results was analyzed. Liu et al. proposed a lightweight deep learning model for human posture estimation to improve the speed of hand posture recognition. On this basis, Liu et al. defined four groups of features and used supervised learning classifiers such as support vector machine (SVM) to score finger pointing, fist clenching, and palm pronation / supination. The motion recognition method based on posture estimation does not require any additional information collection equipment. It only needs an ordinary optical camera or smart phone with recording function. It is very convenient for data collection and switching of application scenarios. Therefore, it is helpful for the remote and home management of Parkinson's disease automatic diagnosis system. Although there are few studies on Parkinson's disease automatic assessment solutions based on posture estimation, it is undoubtedly a trend in future research.

[0012] Whether the human motion information is obtained through wearable devices, optical motion capture systems, or human posture estimation algorithms, a classifier is ultimately needed to classify the severity of the disease based on the extracted features or skeleton sequences. Currently, the most commonly used classifiers can be roughly divided into traditional machine learning algorithms based on feature engineering and deep learning methods based on human skeleton sequence recognition.

[0013] Traditional machine learning algorithms based on feature engineering are the most widely used classifiers in the field of Parkinson's disease movement assessment. Specifically, researchers first extract significant features that may be related to Parkinson's disease movement scores based on data collected from wearable devices, optical motion capture systems, posture estimation systems, etc., and then use feature selection and machine learning algorithms to learn features of different categories, and finally achieve the purpose of automatic movement scoring. For example, Mohammad et al. classified the gait signals of Parkinson's patients using SVM. Nilashi et al. used incremental support vector machines and nonlinear least squares dimensionality reduction to analyze the voice signal characteristics of PD patients. Wang Xueming et al. used the minimum maximum modular support vector machine to integrate the voice information and gender information of male and female PD patients into the classifier, and realized the classification of the UPDRS rating scale. Lin Gang et al. used an iterative decision tree method to integrate the age, gender and other information of PD, effectively improving the accuracy of UPDRS prediction. Shen Tianyu et al. used the BP neural network to classify the motion features collected by wearable devices, extracted the acceleration and angular velocity of PD patients in six movements, and obtained 20 quantitative indicators for evaluating the motor functions of different body parts. Finally, they verified the feasibility of using wearable devices to quantitatively evaluate the motor symptoms of PD patients.

[0014] Traditional machine learning algorithms based on feature engineering have the characteristics of mature algorithms, easy operation, and high accuracy, so they are currently the most popular automatic classification method in the field of Parkinson's disease movement assessment. In general, the features input into the classifier (such as speed, amplitude features, etc.) need to be manually extracted. However, the movement process of the human body or hand is relatively complex, and the key information that is truly discriminative in the movements of Parkinson's patients may be difficult to describe and characterize by manually extracting simple features. Therefore, how to adaptively learn the key features of human or hand movements is the key to improving the performance of the automatic assessment system.

[0015] With the rise of posture estimation methods and the improvement of accuracy, skeleton sequence action recognition methods based on deep learning have also shown great potential and achieved good results in many action classification tasks. With the powerful learning ability of deep learning methods, deep learning classifiers can automatically learn useful information from skeleton sequence data without manually extracting key features like traditional machine learning algorithms, so that some potential useful information can be mined, further improving the accuracy of UPDRS prediction. In addition, the method of action prediction based on skeleton sequences is not easily affected by factors such as environment, lighting, and observation angle, so it is more reliable. At present, many research works have applied deep learning skeleton sequence recognition methods to the classification of sports and daily life behaviors, and have achieved good results. Li et al. proposed a new representation method for skeleton sequences and used an end-to-end deep learning framework of a two-stream convolutional neural network to train the network. Finally, its effectiveness was verified on multiple data sets. Yan et al. proposed a spatiotemporal graph convolutional network (ST-GCN), which can automatically learn temporal and spatial information from skeleton sequence data. ST-GCN has strong generalization ability and has achieved good classification results in both Kinects and NTU-RGBD datasets. Hu et al. introduced a synchronous local-nonlocal module to learn the spatiotemporal information of skeleton sequences and adopted a frequency domain attention mechanism to successfully improve the performance of the model.

[0016] At present, there are few studies on the application of deep learning skeleton sequence recognition algorithms in the field of PD motion recognition, which is almost a blank field. If posture estimation and deep learning skeleton sequence classification algorithms can be successfully and effectively applied to the motion recognition of PD patients, the additional costs brought by wearable devices, optical motion capture devices, etc. can be eliminated, and researchers do not need to manually extract key features, which will be conducive to the promotion of Parkinson's automatic recognition systems in households and mobile applications. At the same time, deep learning skeleton sequence recognition methods can also provide more accurate, stable and reliable prediction results, providing greater help for Parkinson's patients and doctors. Summary of the invention

[0017] In view of the shortcomings of the prior art mentioned above, the purpose of the present invention is to provide an evaluation method and system, storage medium and terminal for Parkinson's disease fist clenching task, which realizes the reliability evaluation of Parkinson's disease fist clenching task based on tree pooling module and spatiotemporal graph convolutional network.

[0018] To achieve the above-mentioned purpose and other related purposes, the present invention provides an evaluation method for a Parkinson's disease fist-clenching task, comprising the following steps: obtaining video information containing a fist-clenching action of a Parkinson's disease patient, wherein the fist-clenching action is a standard action required for performing the fist-clenching task evaluation; extracting hand skeleton sequence data from the video information, and preprocessing the hand skeleton sequence data; extracting hand joint point position features and hand bone vector features based on the preprocessed hand skeleton sequence; inputting the hand joint point position features and the hand bone vector features into a two-stream adaptive spatiotemporal graph convolutional neural network, wherein the two-stream adaptive spatiotemporal graph convolutional neural network includes a spatiotemporal graph convolutional network and a tree pooling module; the spatiotemporal graph convolutional network includes a graph convolutional network and a temporal convolutional network; inputting the fist-clenching action features output by the two-stream adaptive spatiotemporal graph convolutional neural network into a classifier to obtain an evaluation score for the Parkinson's disease fist-clenching task.

[0019] In one embodiment of the present invention, extracting a hand skeleton sequence from the video information includes the following steps:

[0020] The video information is divided into continuous video frames based on an OpenCV video frame extraction unit;

[0021] The hand posture is estimated for each frame of the video based on the OpenPose posture estimation unit to obtain the hand skeleton sequence data.

[0022] In one embodiment of the present invention, preprocessing the hand skeleton sequence includes the following steps:

[0023] Filling missing joint points of the hand skeleton sequence data using temporal context information;

[0024] Using a Butterworth filter and a Savitzky-Golay filter to reduce noise on the hand skeleton sequence data;

[0025] Normalize the coordinates of the denoised hand skeleton sequence data.

[0026] In one embodiment of the present invention, the graph convolutional network is trained based on a self-supervised learning strategy of contrastive learning.

[0027] In one embodiment of the present invention, the tree pooling module adopts an edge pooling module or a neighboring joint pair pooling module.

[0028] In one embodiment of the present invention, the classifier is a SVM classifier.

[0029] The present invention provides an evaluation system for a Parkinson's disease fist-clenching task, comprising an acquisition module, a processing module, an extraction module, an input module and an evaluation module;

[0030] The acquisition module is used to acquire video information containing a fist-clenching action of a Parkinson's disease patient, where the fist-clenching action is a standard action required for performing a fist-clenching task assessment;

[0031] The processing module is used to extract hand skeleton sequence data from the video information and pre-process the hand skeleton sequence data;

[0032] The extraction module is used to extract hand joint point position features and hand bone vector features based on the preprocessed hand skeleton sequence;

[0033] The input module is used to input the hand joint point position features and the hand bone vector features into a dual-stream adaptive spatiotemporal graph convolutional neural network, wherein the dual-stream adaptive spatiotemporal graph convolutional neural network includes a spatiotemporal graph convolutional network and a tree pooling module; the spatiotemporal graph convolutional network includes a graph convolutional network and a temporal convolutional network;

[0034] The evaluation module is used to input the fist-clenching action features output by the dual-stream adaptive spatiotemporal graph convolutional neural network into a classifier to obtain an evaluation score for the Parkinson's disease fist-clenching task.

[0035] The present invention provides a storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned evaluation method for the Parkinson's disease fist-clenching task is implemented.

[0036] The present invention provides an evaluation terminal for a Parkinson's disease fist-clenching task, comprising: a processor and a memory;

[0037] The memory is used to store computer programs;

[0038] The processor is used to execute the computer program stored in the memory, so that the evaluation terminal for the Parkinson's disease fist-clenching task executes the above-mentioned evaluation method for the Parkinson's disease fist-clenching task.

[0039] The present invention provides an evaluation method system for a Parkinson's disease fist-clenching task, comprising the above-mentioned Parkinson's disease fist-clenching task evaluation terminal and a video acquisition device;

[0040] The video acquisition device is used to acquire video information containing the fist-clenching action of a Parkinson's disease patient and send the video information to the evaluation terminal of the Parkinson's disease fist-clenching task.

[0041] As described above, the evaluation method and system, storage medium and terminal for the Parkinson's disease fist-clenching task of the present invention have the following beneficial effects:

[0042] (1) Adaptive graph convolution and second-order skeleton information are introduced to effectively improve the modeling flexibility of graph convolutional networks;

[0043] (2) Based on the characteristics of the tree structure of the hand, four tree pooling modules are designed to achieve feature dimensionality reduction while improving the network’s receptive field;

[0044] (3) The feature representation capability of graph convolutional networks is enhanced by using contrastive learning self-supervised training methods;

[0045] (4) Using deep learning video action recognition method, accurate prediction of UPDRS score of Parkinson's disease patients in fist clenching test was achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Shown is a flow chart of an evaluation method for a Parkinson's disease fist-clenching task in one embodiment of the present invention;

[0047] Figure 2 It is a schematic diagram showing the structure of a dual-stream adaptive spatiotemporal graph convolutional neural network in one embodiment of the present invention;

[0048] Figure 3 Shown is a schematic diagram of the joint point position features and the bone vector features in one embodiment of the present invention;

[0049] FIG. 4( a ) is a schematic diagram showing an edge pooling process in one embodiment of the present invention;

[0050] FIG4( b ) is a schematic diagram showing a pooling process of adjacent joint pairs in one embodiment of the present invention;

[0051] Figure 5 Shown is a schematic diagram of the learning process of a graph convolutional neural network based on momentum contrastive learning in one embodiment of the present invention;

[0052] FIG6( a ) shows a confusion matrix of the fist-clenching action recognition result of the evaluation method for the Parkinson's disease fist-clenching task of the present invention;

[0053] FIG6( b ) shows the ROC curve of the fist-clenching action recognition result of the evaluation method for the Parkinson's disease fist-clenching task of the present invention;

[0054] Figure 7 It is a schematic diagram showing the stability analysis of ten cross validations of the evaluation method of the Parkinson's disease fist clenching task of the present invention;

[0055] Figure 8 Shown is a schematic diagram of the performance of different tree pooling modules; Fig. 9 Shown is a schematic structural diagram of an evaluation system for Parkinson's disease fist-clenching task in one embodiment of the present invention;

[0056] Fig.10 Shown is a schematic structural diagram of an evaluation terminal for a Parkinson's disease fist-clenching task in one embodiment of the present invention;

[0057] Fig.11 Shown is a schematic structural diagram of an evaluation system for Parkinson's disease fist-clenching task in one embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0059] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0060] The evaluation method and system, storage medium and terminal of the Parkinson's disease fist-clenching task of the present invention are based on a tree pooling module and a spatiotemporal graph convolutional network, which realizes the reliability evaluation of the Parkinson's disease fist-clenching task. Among them, the spatiotemporal graph convolutional network uses graph convolution Network (GCN) to obtain spatial structure information, and uses temporal convolutional Network (TCN) to obtain temporal motion information; the tree pooling module uses the characteristics of the tree structure to downsample the hand skeleton sequence data, which can effectively improve the receptive field of the model while performing feature dimensionality reduction. At the same time, the present invention introduces a self-supervised learning strategy of momentum contrast learning, further improves the general feature learning ability of the graph convolutional network, and finally combines the classifier and feature selection method to complete the classification task of Parkinson's disease fist-clenching action.

[0061] like Figure 1 As shown, in one embodiment, the evaluation method of the Parkinson's disease fist-clenching task of the present invention comprises the following steps:

[0062] Step S1, obtaining video information containing a fist-clenching action of a Parkinson's disease patient, wherein the fist-clenching action is a standard action required for performing a fist-clenching task assessment.

[0063] Specifically, by using a video acquisition device such as a monocular camera, a camera, a smart phone, etc., a video of the patient performing the fist clenching test can be recorded to complete the data collection work of the PD patient's fist clenching task, and the data can be sent to the evaluation terminal of the Parkinson's disease fist clenching task of the present invention via wired or wireless means.

[0064] During the acquisition process, the patient needs to bend his elbow and clench his fist with his palm facing the camera. After that, the patient needs to fully open his palm and repeatedly stretch and clench his fist 10 times at the fastest speed, and test both hands separately. During the video data acquisition process, each patient was asked to sit on a chair and face the camera. The frame rate of the recorded video is 30 frames per second, and the resolution is 720P (1280×720). The video of the patient's left and right hands was intercepted separately, and then the left hand was converted into the right hand through horizontal mirror flipping to unify the data format.

[0065] Step S2: extracting hand skeleton sequence data from the video information, and preprocessing the hand skeleton sequence data.

[0066] Specifically, the OpenPose model is used to estimate the hand posture of each frame in the video information to obtain hand skeleton sequence data. The OpenPose model is composed of an OpenCV video frame extraction unit and an OpenPose posture estimation unit. First, the OpenCV video frame extraction unit divides the video information into continuous video frames. Then, the OpenPose posture estimation module estimates the hand posture for each frame of the video. The specific process is: first, the hand is preliminarily located by the wrist, elbow, and shoulder joint positions obtained by the human body posture estimation module, and then the hand posture estimation module is used to obtain the final accurate hand skeleton sequence data.

[0067] In order to solve the problems of data missing, noise interference, inconsistent coordinate system, etc. that may exist in the original hand skeleton sequence data, it is necessary to preprocess the hand skeleton sequence data. In one embodiment of the present invention, preprocessing the hand skeleton sequence data includes the following steps:

[0068] 21) Using temporal context information to fill in the missing joint points of the hand skeleton sequence data.

[0069] 22) Using a Butterworth filter and a Savitzky-Golay filter to reduce noise on the hand skeleton sequence data.

[0070] 23) Normalizing the coordinates of the denoised hand skeleton sequence data to obtain the final hand skeleton sequence data.

[0071] Step S3: extracting hand joint position features and hand bone vector features based on the preprocessed hand skeleton sequence.

[0072] Specifically, Figure 2 As shown, according to the preprocessed hand skeleton sequence, the corresponding hand joint point position features and hand bone vector features are extracted.

[0073] Step S4: input the hand joint position features and the hand bone vector features into a two-stream adaptive spatiotemporal graph convolutional neural network, wherein the two-stream adaptive spatiotemporal graph convolutional neural network includes a spatiotemporal graph convolutional network and a tree pooling module; the spatiotemporal graph convolutional network includes a graph convolutional network and a temporal convolutional network.

[0074] Specifically, Spatial Temporal-Graph Convolutional Networks (ST-GCN) have achieved excellent results in skeleton sequence action recognition tasks. However, in ST-GCN, human skeleton sequence data is usually modeled as a fixed topology spatiotemporal graph structure. This approach lacks the flexibility to model multi-level semantic information and may ignore some important implicit joint connections. In addition, ST-GCN only uses the joint position features in the skeleton data as feature input. The input features are relatively simple, which may result in insufficient information required for model learning, thereby affecting the improvement of recognition accuracy. Therefore, the present invention adopts the following method: Figure 3 The dual-stream adaptive spatiotemporal graph convolutional network (2s-AGCN) shown in the figure addresses the limitations of ST-GCN. The dual-stream adaptive spatiotemporal graph convolutional neural network includes a spatiotemporal graph convolutional network and a tree pooling module; the spatiotemporal graph convolutional network includes a graph convolutional network and a temporal convolutional network. Through the dual-stream adaptive spatiotemporal graph convolutional network, the fisting action features can be obtained based on the hand joint point position features and the hand bone vector features.

[0075] The two-stream adaptive spatiotemporal graph convolution network uses adaptive graph convolution to solve the limitations of fixed topology graph convolution. In an end-to-end learning manner, the topological structure of the graph is optimized along with the network training process, thereby effectively improving the flexibility of network modeling. Specifically, the calculation process of adaptive graph convolution is as follows:

[0076]

[0077] Among them, W k is the weight function implemented by 1×1 convolution operation, K v represents the kernel size of the spatial dimension, A k ∈R N×N (N is the total number of joints) is the original adjacency matrix, representing the physical connection of the hand joints. k ∈R N×N Yes and A kThe parameters of the mask matrix of the same size can be updated synchronously with the training of the network, so the implicit connections in the hand skeleton structure can be mined. k The elements in the matrix not only represent whether there is a connection between two joints, but also represent the strength of the connection, thus greatly improving the flexibility of network learning. k It is a connection dependency matrix, which calculates whether there is a connection between every two joint points and the connection strength based on the input data of each sample. The formula is as follows:

[0078]

[0079] Among them, θ and φ are two embedding functions. This formula calculates the similarity between two vertices through this normalized Gaussian function, and the result is used as a soft connection with the adjacency matrix A. k and B k Combined to form the final adjacency matrix. In the specific implementation process, C k It can be implemented by two 1×1 convolutions and a softmax activation function.

[0080] In addition, the dual-stream adaptive spatiotemporal graph convolutional network introduces the second-order information of bone features, namely bone vector features, based on the position features of skeleton joints. Specifically, each bone is connected to two adjacent hand joints. The joints closer to the wrist joint (origin) are defined as source joints, and the joints far from the origin are defined as target joints. The second-order features of the bones are the vectors from the source joints to the target joints. The bone vector features can represent the length and direction of the joint connection, which has higher information recognition for action recognition tasks.

[0081] In convolutional neural networks, pooling layers play a very important role. Common pooling layers include maximum pooling, average pooling, etc. Pooling layers have two main functions in convolutional neural networks:

[0082] 1) Feature dimensionality reduction.

[0083] There may be a large amount of redundant information in the input features of the convolutional neural network. The role of the pooling layer is to reduce the dimension of the features while retaining the main features, thereby reducing model parameters and calculations, preventing overfitting problems, and improving the generalization ability of the model.

[0084] 2) Improve the receptive field of the model.

[0085] During the convolution operation, the convolution kernel can only aggregate the information of adjacent nodes at a time. Therefore, when the network layer is shallow, the convolution kernel can only obtain local information of a small range, and the receptive field is limited. Adding a pooling layer can significantly increase the size of the receptive field. A larger receptive field can effectively improve the model's ability to learn global features, thereby improving the classification effect.

[0086] However, the lack of pooling modules in the spatiotemporal graph convolutional network limits the improvement of recognition effect. In the spatiotemporal graph convolutional network, the graph convolutional network can only obtain information about adjacent nodes at a time, and the number of joints of the human body or hand is fixed, so the feature dimension is also fixed. If the pooling operation is missing, when the number of network layers is shallow, the receptive field of the model is small, which will limit the improvement of classification results.

[0087] Therefore, according to the characteristics of the fist-making hand action, the present invention proposes a tree-structured skeleton data pooling module, namely a tree pooling module, which can effectively improve the receptive field of the model, thereby improving the accuracy of the fist-making action evaluation. Specifically, the hand skeleton structure can be regarded as a tree structure, and its root node is the wrist joint point. The tree itself is a special acyclic graph structure with a relatively simple structure, which is suitable for pooling operations. The tree pooling module proposed in the present invention can utilize the characteristics of the tree structure to regularly reduce the size of the subtree each time, so as to achieve the purpose of pooling.

[0088] As shown in FIG. 4( a ) and FIG. 4( b ), the tree pooling modules of the present invention are divided into two types:

[0089] 1) As shown in Figure 4(a), the edge pooling module (Edge pooling) gradually advances from leaf nodes to root nodes;

[0090] 2) As shown in Figure 4(b), the adjacent joint pair pooling module (Pair pooling).

[0091] Among them, in the edge pooling module, each pooling process will fuse the features of the leaf node and its parent node, thereby reducing the scale of the tree structure; in the adjacent joint pair pooling module, each pooling process will fuse the features of the adjacent joint points to reduce the scale of the tree structure. Each pooling module is divided into maximum pooling and average pooling, so there are four different combinations. The present invention uses a wealth of experiments to verify the effectiveness of the tree pooling module, and the experimental results show that the edge maximum pooling module has the greatest improvement effect on the classification results.

[0092] In one embodiment of the present invention, the graph convolutional network is trained based on a self-supervised learning strategy based on contrastive learning. The contrastive learning method can make full use of unlabeled data to enhance the learning ability of the network's general features, so that it can be effectively transferred to downstream subtasks. Unlike supervised training methods, contrastive learning does not train the network based on input data and labels, but performs representation learning by comparing different samples. In general, the purpose of contrastive learning is to train a dictionary encoder. Given an encoding query q and a set of key values ​​{k0, k1, k2, ...} of a set of encoding samples, assuming that there is only one key value k in the set. + matches q, then when q matches the positive sample k + When the key value is most similar but different from other key values ​​(negative samples), the loss function of contrastive learning has the lowest value. The common loss function used for contrastive learning is InfoNCE, and its formula is as follows:

[0093]

[0094] Among them, K is the total number of key values ​​in the dictionary, τ is an important hyperparameter in InfoNCE, and L q The purpose is to minimize the difference between q and k. + The distance between q and other key values ​​is increased. Usually, the result of contrastive learning self-supervised training is an encoder f that can extract common features. Given a query sample x q and the sample to be compared x k The role of the encoder is to generate the corresponding feature representation q = f(x q ) and k = f(x k ). In actual work, the sample input x can be any data organization form such as pictures, text, skeleton sequence, etc., and the structure of the encoder is not fixed. The feature extractor obtained by contrastive learning has strong generalization and can be easily transferred to downstream tasks, reaching or even exceeding the recognition effect of supervised learning.

[0095] Therefore, the present invention adopts a graph convolutional neural network based on momentum contrastive learning (MoCo) for fist-clenching action recognition, which effectively improves the feature representation ability of the network and thus improves the recognition effect. Figure 5The figure shows the learning process of the graph convolutional neural network based on momentum contrast learning. During the training process, each fist skeleton sequence data sample will first be randomly transformed (spatial stretching, rotation, time series sampling, etc.) through a random amplifier. The results of random transformation from the same sample are positive samples of each other, and the results of random transformation from other samples are negative samples of each other. The present invention uses a queue to maintain the key value of the negative sample set, thereby realizing the decoupling of the dictionary and the mini-batch, expanding the capacity of the dictionary, so that the network can fully learn the difference between positive and negative samples. In order to make the update process of the network parameters smoother, the present invention designs two adaptive graph convolutional networks with the same structure but no shared parameters, which are used as query encoders and key-value momentum encoders respectively, and uses back propagation to update the parameters θ of the encoder. q , and the momentum update method is used to update the key-value encoder parameters θ k , the update process is θ k ←mθ k +(1-m)θ q . where m∈[0,1) is the momentum parameter. In this way, θ k The update process is smoother, and the slow update of parameters is the key to making good use of the dictionary queue. Therefore, the larger the value of m is, the better the training effect.

[0096] S5. Input the fist-clenching action features output by the dual-stream adaptive spatiotemporal graph convolutional neural network into a classifier to obtain an evaluation score for the Parkinson's disease fist-clenching task.

[0097] Specifically, the fist-clenching action features are classified based on a classifier, thereby obtaining an evaluation score for the Parkinson's disease fist-clenching task.

[0098] In one embodiment of the present invention, the classifier adopts a SVM classifier to predict the UPDRS score of the fist-clenching action.

[0099] Preferably, the present invention is based on a user interface (GUI) to achieve tasks such as video recording, data importing, task selection, and recognition result display.

[0100] The following specific examples are used to verify the evaluation method of the Parkinson's disease fist-clenching task of the present invention.

[0101] Table 1 shows the recognition result statistics of the evaluation method of the Parkinson's disease fist-clenching task of the present invention in the five categories of MDS-UPDRS, including the acceptable accuracy, AUC value (area size under the ROC curve), precision, recall and F1 score of each category. It can be seen from the table that the acceptable accuracy of categories 0, 1, and 2 is above 95%, and the performance of other indicators is also relatively ideal, while the number of categories 3 and 4 is relatively small, resulting in their performance slightly worse than the first three categories, but also reaching an acceptable accuracy of more than 65%. The final average classification accuracy is 60.2%, and the acceptable accuracy is 95.4%. Figures 6 (a) and 6 (b) show the confusion matrix and ROC curve of the final classification results of the evaluation method of the Parkinson's disease fist-clenching task of the present invention, which more intuitively reflects the effectiveness of the classification model. The confusion matrices of the five MDS-UPDRS scores have high accuracy within the acceptable range, and the AUC values ​​of the ROC curves are satisfactory.

[0102] Table 1. Classification results of the five types of MDS-UPDRS scores

[0103]

[0104] The present invention was subjected to stability analysis, that is, the sample order was randomly disrupted each time, and then a five-fold crossover experiment was used to test the classification effect, which was repeated ten times. Figure 7 The results of stability analysis are shown. The accuracy of ten repeated experiments fluctuates between 59.08% and 61.03%, and the acceptable accuracy fluctuates between 94.20% and 95.49%, without any extreme cases of very poor performance. In addition, the standard deviations of the accuracy and acceptable accuracy are 0.64% and 0.39%, respectively, which further verifies the stability and robustness of the method proposed in this invention.

[0105] In order to verify the effectiveness of the tree pooling module, the present invention experimentally verifies the effects of four different tree pooling modules, and uses the two-stream spatiotemporal graph convolutional neural network without the tree pooling module as a control group for ablation comparison. Figure 8As shown, four different tree pooling modules are: Pair-Avg Pooling, Edge-Avg Pooling, Pair-Max Pooling, and Edge-Max Pooling. As can be seen from the figure, the four different tree pooling modules have a certain improvement effect on the classification results. The maximum pooling method is better than the average pooling method, and the edge pooling method is slightly better than the neighboring joint pair pooling method. Therefore, the present invention finally selects the edge maximum pooling module as the benchmark for the overall experiment. The pooling module based on the tree structure can effectively improve the receptive field of the model and feature dimensionality reduction, and can greatly improve the classification effect.

[0106] The momentum contrastive learning graph convolutional network can learn common features in the data in a self-supervised manner, which can effectively improve the network's feature representation capabilities, and can therefore be well migrated to downstream tasks. The present invention first uses the momentum contrastive learning method to train the graph convolutional action recognition network, and then uses the trained graph convolutional network as a feature extractor to extract common features in the fisting action data. On this basis, the present invention attempts two strategies to migrate it to the final fisting hand action recognition task. The two strategies are: 1) using transfer learning to fine-tune the dual-stream spatiotemporal graph convolutional neural network classifier; 2) using the graph convolutional network trained by contrastive learning as a feature generator, and then using feature selection and SVM classifiers for classification.

[0107] Table 2 shows the classification results of the two contrastive learning training strategies, and compares them with the two-stream adaptive spatiotemporal graph convolutional neural network (2s-AGCN) without contrastive learning. As can be seen from the table, the training method of using momentum contrastive learning (MoCo-AGCN) + transfer learning fine-tuning weights effectively improves the accuracy of the classification results (57.8%), while the method of using momentum contrastive learning + SVM classifier achieves the best classification accuracy of 60.2%. The above results intuitively reflect the effectiveness of the momentum contrastive learning self-supervised training strategy.

[0108] Table 2. Performance of different model training strategies

[0109]

[0110] The evaluation method of the Parkinson's disease fist-clenching task of the present invention is compared with the current mainstream skeleton sequence recognition network, as shown in Table 3. Specifically, ST-GCN (spatial-temporal graph convolutional network), Motif-GCNs (theme-based spatial-temporal graph convolutional network), two-stream CNN, 2s-AGCN (two-stream adaptive graph convolutional network) are applied to the fist-clenching action dataset, and the classification accuracy rates obtained are 35.5%, 39.9%, 48.8%, 49.8%, and 56.3%, respectively. In contrast, the contrastive learning graph convolutional network proposed in the present invention combined with the adjacent joint pair maximum pooling module improves the accuracy to 59.2%, and the accuracy combined with the edge maximum pooling module reaches 60.2%, which is much higher than the most advanced skeleton-based action recognition network.

[0111] Table 3. Comparison results of the present invention and mainstream skeleton sequence recognition methods

[0112]

[0113] The present invention also compares the applied work with the sensor-based action recognition scheme, as shown in Table 4. It can be seen that based on 157 subjects and 931 sets of fist-clenching video data, the method proposed by the present invention finally achieved a prediction accuracy of 60.2% and an acceptable accuracy of 95.4%. In addition, the Pearson correlation coefficient between the prediction result of the fist-clenching action recognition of the present invention and the manually annotated label is 0.64, which has also reached the average level based on sensor equipment, verifying the effectiveness of the graph convolutional fist-clenching action recognition model proposed by the present invention that combines the tree pooling module and contrastive learning.

[0114] Table 4. Comparison results with fist-clenching action evaluation method based on sensor equipment

[0115]

[0116] like Fig. 9 As shown, in one embodiment, the evaluation system for the Parkinson's disease fist-clenching task of the present invention includes an acquisition module 91 , a processing module 92 , an extraction module 93 , an input module 94 and an evaluation module 95 .

[0117] The acquisition module 91 is used to acquire video information containing a fist-clenching action of a Parkinson's disease patient, where the fist-clenching action is a standard action required for performing a fist-clenching task assessment.

[0118] The processing module 92 is connected to the acquisition module 91 and is used to extract hand skeleton sequence data from the video information and pre-process the hand skeleton sequence data.

[0119] The extraction module 93 is connected to the processing module 92 and is used to extract hand joint point position features and hand bone vector features based on the preprocessed hand skeleton sequence.

[0120] The input module 94 is connected to the extraction module 93, and is used to input the hand joint point position features and the hand bone vector features into a two-stream adaptive spatiotemporal graph convolutional neural network, wherein the two-stream adaptive spatiotemporal graph convolutional neural network includes a spatiotemporal graph convolutional network and a tree pooling module; the spatiotemporal graph convolutional network includes a graph convolutional network and a temporal convolutional network.

[0121] The evaluation module 95 is connected to the input module 94, and is used to input the fist-clenching action features output by the dual-stream adaptive spatiotemporal graph convolutional neural network into a classifier to obtain an evaluation score for the Parkinson's disease fist-clenching task.

[0122] Among them, the structures and principles of the acquisition module 91, the processing module 92, the extraction module 93, the input module 94 and the evaluation module 95 correspond one by one to the steps in the above-mentioned evaluation method of the Parkinson's disease fist-clenching task, so they are not repeated here.

[0123] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements, or they can all be implemented in the form of hardware, or some modules can be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example: the x module can be a separately established processing element, or it can be integrated in a certain chip of the above device. In addition, the x module can also be stored in the memory of the above device in the form of program code, and called and executed by a certain processing element of the above device. The implementation of other modules is similar. These modules can be integrated together in whole or in part, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software. The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), one or more microprocessors (DSP), one or more field programmable gate arrays (FPGA), etc. When a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. These modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0124] The storage medium of the present invention stores a computer program, which, when executed by a processor, implements the above-mentioned Parkinson's disease fist-clenching task assessment method. Preferably, the storage medium includes: ROM, RAM, disk, USB flash drive, memory card or optical disk, etc., various media that can store program codes.

[0125] like Fig.10 As shown, in one embodiment, the evaluation terminal for the Parkinson's disease fist-clenching task of the present invention includes: a processor 101 and a memory 102 .

[0126] The memory 102 is used to store computer programs.

[0127] The memory 102 includes: ROM, RAM, disk, USB flash drive, memory card or optical disk, etc., which can store program codes.

[0128] The processor 101 is connected to the memory 102 and is used to execute the computer program stored in the memory 102 so that the evaluation terminal for the Parkinson's disease fist clenching task executes the above-mentioned evaluation method for the Parkinson's disease fist clenching task.

[0129] Preferably, the processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0130] like Fig.11 As shown, in one embodiment, the evaluation system for the Parkinson's disease fist-clenching task of the present invention includes the above-mentioned Parkinson's disease fist-clenching task evaluation terminal 111 and the video acquisition device 112.

[0131] The video acquisition device 112 is connected to the evaluation terminal 111 of the Parkinson's disease fist-making task, and is used to acquire video information containing the toe-slapping action of the Parkinson's disease patient and send it to the evaluation terminal 111 of the Parkinson's disease fist-making task.

[0132] In summary, the evaluation method and system, storage medium and terminal of the Parkinson's disease fist-clenching task of the present invention introduce adaptive graph convolution and second-order skeleton information, which effectively improves the modeling flexibility of the graph convolution network; according to the characteristics of the tree structure of the hand, four tree pooling modules are designed to achieve feature dimension reduction while improving the network's receptive field; the contrast learning self-supervised training method is used to enhance the feature representation ability of the graph convolution network; and the deep learning video action recognition method is used to achieve accurate prediction of the UPDRS score of the Parkinson's disease patient's fist-clenching test. Therefore, the present invention effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0133] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. An evaluation method for the fist clenching task in Parkinson's disease, Features: The following steps are involved: Acquiring video information containing a fist-clenching action of a Parkinson's disease patient, wherein the fist-clenching action is a standard action required for performing a fist-clenching task assessment; Extracting hand skeleton sequence data from the video information, and preprocessing the hand skeleton sequence data; Extract hand joint position features and hand bone vector features based on the preprocessed hand skeleton sequence; Inputting the hand joint point position features and the hand bone vector features into a two-stream adaptive spatiotemporal graph convolutional neural network, wherein the two-stream adaptive spatiotemporal graph convolutional neural network includes a spatiotemporal graph convolutional network and a tree pooling module; the spatiotemporal graph convolutional network includes a graph convolutional network and a temporal convolutional network; Inputting the fist-clenching action features output by the dual-stream adaptive spatiotemporal graph convolutional neural network into a classifier to obtain an evaluation score for the Parkinson's disease fist-clenching task; Training the graph convolutional network using a self-supervised learning strategy based on contrastive learning; The tree pooling module adopts an edge pooling module or a neighboring joint pair pooling module.

2. The Parkinson's disease fist-clenching task evaluation method according to claim 1, Features: Extracting a hand skeleton sequence from the video information comprises the following steps: The video information is divided into continuous video frames based on an OpenCV video frame extraction unit; The hand posture is estimated for each frame of the video based on the OpenPose posture estimation unit to obtain the hand skeleton sequence data.

3. The Parkinson's disease fist-clenching task evaluation method according to claim 1, Features: Preprocessing the hand skeleton sequence includes the following steps: Filling missing joint points of the hand skeleton sequence data using temporal context information; Using a Butterworth filter and a Savitzky-Golay filter to reduce noise on the hand skeleton sequence data; Normalize the coordinates of the denoised hand skeleton sequence data.

4. The Parkinson's disease fist-clenching task evaluation method according to claim 1, Features: The classifier adopts SVM classifier.

5. An evaluation system for the Parkinson's disease fist clenching task, Features: It includes an acquisition module, a processing module, an extraction module, an input module and an evaluation module; The acquisition module is used to acquire video information containing a fist-clenching action of a Parkinson's disease patient, where the fist-clenching action is a standard action required for performing a fist-clenching task assessment; The processing module is used to extract hand skeleton sequence data from the video information and pre-process the hand skeleton sequence data; The extraction module is used to extract hand joint point position features and hand bone vector features based on the preprocessed hand skeleton sequence; The input module is used to input the hand joint point position features and the hand bone vector features into a dual-stream adaptive spatiotemporal graph convolutional neural network, wherein the dual-stream adaptive spatiotemporal graph convolutional neural network includes a spatiotemporal graph convolutional network and a tree pooling module; the spatiotemporal graph convolutional network includes a graph convolutional network and a temporal convolutional network; The evaluation module is used to input the fist-clenching action features output by the dual-stream adaptive spatiotemporal graph convolutional neural network into a classifier to obtain an evaluation score for the Parkinson's disease fist-clenching task; Training the graph convolutional network using a self-supervised learning strategy based on contrastive learning; The tree pooling module adopts an edge pooling module or a neighboring joint pair pooling module.

6. A storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the evaluation method of the Parkinson's disease fist-clenching task according to any one of claims 1 to 4 is implemented.

7. An evaluation terminal for Parkinson's disease fist clenching task, It is characterized in that include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory, so that the evaluation terminal for the Parkinson's disease fist-clenching task performs the evaluation method for the Parkinson's disease fist-clenching task according to any one of claims 1 to 4.

8. An evaluation method system for Parkinson's disease fist clenching task, Features: An evaluation terminal and a video acquisition device for the Parkinson's disease fist-clenching task according to claim 7; The video acquisition device is used to acquire video information containing the fist-clenching action of a Parkinson's disease patient and send the video information to the evaluation terminal of the Parkinson's disease fist-clenching task.

Images