Evaluation method, system, storage medium and terminal for Parkinson's disease toe tapping task

Through graph neural network and supervised contrast learning strategy, combined with multi-stream combined sparse learning mechanism, automatic and objective evaluation of toe-shot ground task in Parkinson's disease patients is achieved, and the problems of inaccurate evaluation and limitations of feature engineering in traditional methods are solved. Improves diagnostic efficiency and accuracy.

CN113901892BActive Publication Date: 2024-12-06SHANGHAI JIAOTONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111108294.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-22
Publication Date
2024-12-06
Estimated Expiration
2041-09-22

AI Technical Summary

Technical Problem

The prior art is difficult to accurately and objectively evaluate the toe-stripping task of Parkinson's patients, resulting in low diagnostic and treatment efficiency, and the inconvenience and feature engineering limitations of sensors make it difficult to popularize and implement automatic evaluation.

Method used

The graph neural network scheme is used for end-to-end feature extraction and classification, combined with the supervised comparison learning strategy and the multi-stream joint sparse learning mechanism, extract joint position and velocity characteristics through video information, and design the spatiotemporal interactive graph convolution module to mine the joint nodes. Remote dependencies enable automatic evaluation of Parkinson's toe-shot task.

Benefits of technology

A fine-grained evaluation of the toe shooting task of Parkinson's disease is achieved. The evaluation results are highly correlated with doctor's scores. You can get objective scores by shooting videos with ordinary cameras, which improves diagnostic accuracy and efficiency, and overcomes the limitations of traditional methods. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113901892B_ABST
    Figure CN113901892B_ABST
Patent Text Reader

Abstract

The present invention provides an evaluation method and system for a Parkinson's disease toe tapping task, a storage medium and a terminal, comprising the following steps: obtaining video information containing a toe tapping action of a Parkinson's disease patient, wherein the toe tapping action is a standard action required for performing the toe tapping task evaluation; extracting a human joint point coordinate sequence from the video information, and obtaining joint position information and joint velocity information based on the human joint coordinate sequence; inputting the joint position information and the joint velocity information into a joint position flow network and a joint velocity flow network respectively; inputting the output scores of the joint position flow network and the joint velocity flow network into a fully connected layer, and selecting the evaluation score with the highest probability value as the evaluation score of the Parkinson's disease toe tapping task. The evaluation method and system, storage medium and terminal for the Parkinson's disease toe tapping task of the present invention realize automatic, objective and fine-grained evaluation of the Parkinson's disease patient toe tapping task.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning, and in particular to an evaluation method and system, storage medium and terminal for a Parkinson's disease toe tapping task. Background Art

[0002] Bradykinesia is one of the main characteristics of Parkinson's Disease (PD), which can seriously affect the patient's mobility and quality of life. Accurate assessment of bradykinesia is very important for the diagnosis and treatment of PD. Neurologists use clinical rating scales to assess the severity of bradykinesia symptoms. The most widely used is the new version of the World Movement Disorder Society Parkinson's Disease Rating Scale (MDS-UPDRS). Its third part (motor function examination) defines a series of related items to evaluate the bradykinesia symptoms of PD.

[0003] However, this clinical assessment method has the following three major limitations, which may adversely affect the accurate assessment and timely monitoring of bradykinesia in PD;

[0004] (1) The assessment mainly relies on the subjective judgment of neurologists combined with clinical experience, so the consistency of scores among different neurologists is low;

[0005] (2) The number of doctors who have been specially trained and certified in scoring is small, while the number of patients is large, which is not conducive to improving medical services;

[0006] (3) It is not convenient to realize remote diagnosis and home monitoring of PD, which affects the improvement of patient management level.

[0007] Currently, it is more urgent and necessary to realize the automatic assessment of bradykinesia in PD. The toe tapping task is an essential part of the motor function examination in MDS-UPDRS, and is also an important component of the assessment of lower limb bradykinesia. Its angular velocity and other indicators can effectively quantify lower limb bradykinesia. Patients need to tap their toes with the largest amplitude and fastest speed, and doctors score according to the speed, amplitude, occurrence of pauses, and the changing trend of each feature over time. Therefore, an accurate and objective method is needed to realize the automatic assessment of the PD toe tapping task and improve the accuracy and efficiency of PD diagnosis.

[0008] At present, according to the source of the movement signal, researchers' research on automatic assessment of PD bradykinesia can be divided into three categories:

[0009] 1) Sensor-based methods

[0010] Samà et al. used a support vector machine (SVM) to extract the gait-related portion of the signal provided by a waist-worn triaxial accelerometer and then determined the occurrence and severity of gait delay using a regression model. Dai et al. used the parameters of a six-axis electromagnetic tracking system to train a machine learning algorithm to quantify bradykinesia, and the results showed superior performance to that of neurologists.

[0011] 2) Methods based on optical devices

[0012] Bank et al. used optical hand tracking (OHT) to quantify the motor components of bradykinesia from three hand subtasks, demonstrating the potential of OHT to quantify upper limb bradykinesia in PD. et al. analyzed the motion parameters of the finger-to-finger test using a non-contact three-dimensional motion capture system and showed that amplitude attenuation and maximum opening speed can effectively identify PD patients.

[0013] 3) Methods based on joint point coordinate sequences extracted from videos

[0014] Liu et al. designed a human posture estimation algorithm to extract the kinematic features of the patient's hand, and then used SVM to generate scores for three items related to upper limb bradykinesia. Pang et al. used OpenPose and combined two cameras to obtain a 3D coordinate sequence, and then extracted the kinematic features of the finger joints through wavelet analysis to achieve effective identification of PD patients.

[0015] Currently, there are only three studies on the automatic evaluation of the toe-slapping task, which are mainly divided into two categories: sensor-based and video-based methods. In the first category of sensor-based methods, Martinez-Manzanera et al. collected signals through a nine-degree-of-freedom sensor and calculated relevant motion features, and finally achieved a classification accuracy of 62.5-64.5% through SVM. Kim et al. measured the movement of the ankle joint using a gyro sensor, and then analyzed the correlation between the features and the clinical scores by the Spearman rank correlation coefficient, with a correlation coefficient of 0.72-0.81. In the second category of video-based methods, there is only one study so far. Li et al. used the coordinates of the patient's ankle joint points extracted from the video to infer the task bounding box, and then used dense optical flow to capture the action signal, and extracted 95 kinematic and spectral features. Finally, random forest regression analysis was used to obtain the predicted value, and the Pearson correlation between it and the clinician's score was 0.372.

[0016] Although the above work is of reference significance for subsequent research, there are still the following difficulties:

[0017] 1) Inconvenience of sensors

[0018] The sensor is fixed on the patient's instep, which may affect the patient's execution of movements. In addition, the deployment and calibration of the sensor requires strong professionalism, which is not conducive to the popularization and remote implementation of PD automatic assessment.

[0019] 2) Limitations of feature engineering

[0020] The above methods all use feature engineering to extract features from motion signals, and then predict task scores through machine learning or perform correlation analysis through statistical methods. However, it is challenging to manually define discriminative features for fine-grained classification.

[0021] Deep learning is a powerful automated feature engineering and extraction tool that can learn complex task-related features by itself. Currently, the related work of deep learning technology in automatic assessment of PD motor function can be divided into two categories:

[0022] 1) Deep learning technology is used in video-based joint point extraction. That is, the human skeleton sequence is extracted from the PD motion video using a human posture estimation model, and then the relevant features are extracted and analyzed through feature engineering. Finally, a machine learning algorithm is used for classification or regression prediction.

[0023] 2) Deep learning techniques were used in PD motion feature extraction and classification. To achieve automatic detection of gait freezing based on vision, Hu et al. constructed a novel graph convolutional neural network, embedded a data-driven adjacency matrix estimation layer and a weakly supervised learning strategy, and finally achieved an AUC value of 0.887. Afterwards, they proposed a graph sequence recurrent neural network to simultaneously model the structural graph and the temporal graph, and obtained an AUC value of 0.90. To achieve automatic evaluation of vision-based leg flexibility tasks, Guo et al. proposed a sparse adaptive graph convolutional network to achieve adaptive mining of discriminative human spatial structural relationships, and finally obtained an accuracy of 70.34% and an AUC value of 0.90. They also proposed a two-stream spatiotemporal attention graph convolutional network to achieve the evaluation of gait motion disorder videos, with an accuracy of 65.66%. In addition, Li et al. proposed a three-stream fine-grained convolutional neural network model to achieve automatic fine-grained classification of finger-to-finger tests, achieving an accuracy of 72.4%.

[0024] However, according to investigations, there is no research on the use of deep learning technology to achieve feature extraction and classification of the toe tapping task for PD patients. This implementation model can get rid of the limitations of manually defined features and realize the automatic mining of discriminative features.

[0025] Fine-grained actions have large inter-class similarities, and the changes in participants and backgrounds lead to large intra-class differences. Therefore, fine-grained human action recognition has always been one of the challenges in the field of action recognition. Existing studies usually explore and utilize aspects such as human-object interaction, temporal context, human body part location, detail representation ability, and adaptability to unlabeled environments to improve the performance of fine-grained action recognition. For example, in order to model the interaction between people and objects, Zhou et al. proposed an interactive action component mining method to form interactive components by generating target region candidates, creating spatiotemporal region graphs, and obtaining strongly connected subgraphs. In order to learn the interaction between actions in the temporal dimension, Singh et al. used a multi-stream convolutional neural network to create a joint representation of the full frame and the frame of the person's bounding box, and then learned the long-term temporal context of the action through a bidirectional long short-term memory layer. In order to utilize the prior knowledge of the location of human body parts, Ma et al. used a pose estimation method to obtain human body patches as the input of the convolutional neural network, and then encoded the output of the last pooling layer in the convolutional neural network to generate discriminative action descriptors. In order to improve the network's ability to represent fine-grained details, Martinez et al. improved the final classification stage of the model and designed local feature branches and discriminative filter groups to enable the model to focus on key fine-grained local areas in the video. In order to address environmental bias in fine-grained action recognition datasets, Munro et al. integrated adversarial training of each modality and multimodal self-supervision methods, and proposed a multimodal unsupervised domain adaptation strategy to improve the model's ability to adapt to unlabeled environments.

[0026] Although the above studies have explored the challenges in the field of fine-grained action recognition from multiple perspectives and achieved good results, they did not fully utilize the inter-class similarities and intra-class differences in the feature extraction process, that is, the specificity between features of different categories is insufficient.

[0027] Early methods often build multi-stream frameworks based on RGB frames and optical flow information. For example, Simonyan et al. used a single static video frame as the input of the spatial stream and the cross-frame motion information as the input of the temporal stream, and finally obtained the classification result through the softmax score fusion method based on SVM. Wang et al. introduced RGB difference and distorted optical flow as one of the inputs of the network, and fused the spatial and temporal stream networks through weighted averaging.

[0028] Since RGB frames and optical flow information are easily disturbed by complex backgrounds and changing appearances, human skeleton data has strong adaptability and robustness, which has attracted widespread attention. Wang et al. proposed a two-stream recurrent neural network architecture based on the spatial and temporal relationship of the skeleton, using the joint point sequence to model the spatial structure and learn the temporal dynamics at the same time. In order to simulate the spatial structure of the human skeleton more naturally, more and more works have begun to introduce graph neural networks to build effective models for recognizing the spatial features of skeleton sequences, and many of them have adopted a two-stream framework. Si et al. designed an attention-enhanced graph convolutional long short-term memory network and extended it to a hybrid model based on joints and parts. Shi et al. represented the skeleton data as a directed acyclic graph, thereby constructing a two-stream directed graph neural network framework based on spatial and motion information. More works believe that the length and direction attributes of the bones can provide some important supplementary information, and designed a two-stream framework based on joints and bones on the basis of the basic network structure, achieving good recognition results.

[0029] Most current two-stream frameworks use different features to train separate models with the same network architecture, and then add the output scores of the two streams through soft voting to get the final predicted category. This method does not exploit the potential similarity of the two-stream features, and the features extracted by each stream are redundant. Summary of the invention

[0030] In view of the shortcomings of the prior art mentioned above, the purpose of the present invention is to provide an evaluation method and system, storage medium and terminal for the Parkinson's disease toe tapping task, to achieve end-to-end feature extraction and classification based on a graph neural network solution, to improve performance based on a supervised contrast learning strategy and a multi-stream joint sparse learning mechanism, and to achieve automatic, objective and fine-grained evaluation of the toe tapping task for Parkinson's patients.

[0031] To achieve the above-mentioned purpose and other related purposes, the present invention provides an evaluation method for a Parkinson's disease toe tapping task, comprising the following steps: obtaining video information containing a toe tapping action of a Parkinson's disease patient, wherein the toe tapping action is a standard action required for performing the toe tapping task evaluation; extracting a human joint point coordinate sequence from the video information, and obtaining joint position information and joint velocity information based on the human joint coordinate sequence; inputting the joint position information and the joint velocity information into a joint position flow network and a joint velocity flow network, respectively; the joint position flow network and the joint velocity flow network both include three space-time-channel units, one spatiotemporal interactive graph convolution module, one average pooling layer and one fully connected layer; inputting the output scores of the joint position flow network and the joint velocity flow network into the fully connected layer, and selecting the evaluation score with the highest probability value as the evaluation score of the Parkinson's disease toe tapping task.

[0032] In one embodiment of the present invention, a human joint point coordinate sequence is extracted from the video information based on the human pose estimation model OpenPose.

[0033] In one embodiment of the present invention, a supervised contrastive learning strategy is applied to the output features of the average pooling layer, which uses category prior knowledge to bring the features of samples of the same category closer and separate the features of samples of different categories to construct and strengthen the specific feature space of each fine-grained category.

[0034] In one embodiment of the present invention, the weights of the fully connected layers of the joint position flow network and the joint velocity flow network are constrained through a multi-stream joint sparse learning mechanism to enhance the similarity and discriminability of position features and velocity features.

[0035] In one embodiment of the present invention, the space-time-channel unit is composed of a spatial graph convolution, a temporal convolution, and a Squeeze-and-Excitation module for the channel.

[0036] In one embodiment of the present invention, in the spatiotemporal interaction graph convolution module, the spatiotemporal interaction graph is represented by G = {V, E}, all the relevant nodes required for the task extracted from a video constitute the vertex set V of the spatiotemporal interaction graph, and the edge set E is formed by calculating the similarity of the attributes of different vertices; the output feature graph f of the spatiotemporal interaction graph convolution module out =Af in W, where A is the spatiotemporal similarity adjacency matrix, f in For the input feature map, W is the weight function implemented by a 1×1 convolution operation.

[0037] The present invention provides an evaluation system for a Parkinson's disease toe tapping task, comprising an acquisition module, an extraction module, a processing module and an evaluation module;

[0038] The acquisition module is used to acquire video information containing toe tapping movements of Parkinson's disease patients, wherein the toe tapping movements are standard movements required for toe tapping task assessment;

[0039] The extraction module is used to extract a human joint point coordinate sequence from the video information, and obtain joint position information and joint velocity information based on the human joint coordinate sequence;

[0040] The processing module is used to input the joint position information and the joint velocity information into a joint position flow network and a joint velocity flow network respectively; the joint position flow network and the joint velocity flow network each include 3 space-time-channel units, 1 spatiotemporal interaction graph convolution module, 1 average pooling layer and 1 fully connected layer;

[0041] The evaluation module is used to input the output scores of the joint position flow network and the joint velocity flow network into a fully connected layer, and select the evaluation score with the highest probability value as the evaluation score of the Parkinson's disease toe tapping task.

[0042] The present invention provides a storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned evaluation method for the Parkinson's disease toe tapping task is implemented.

[0043] The present invention provides an evaluation terminal for a Parkinson's disease toe tapping task, comprising: a processor and a memory;

[0044] The memory is used to store computer programs;

[0045] The processor is used to execute the computer program stored in the memory, so that the evaluation terminal for the Parkinson's disease toe tap task executes the above-mentioned evaluation method for the Parkinson's disease toe tap task.

[0046] The present invention provides an evaluation system for a Parkinson's disease toe tapping task, comprising the above-mentioned Parkinson's disease toe tapping task evaluation terminal and a video acquisition device;

[0047] The video acquisition device is used to acquire video information containing the toe tapping action of a Parkinson's disease patient and send the video information to the evaluation terminal of the Parkinson's disease toe tapping task.

[0048] As described above, the evaluation method and system, storage medium and terminal for the Parkinson's disease toe tapping task of the present invention have the following beneficial effects:

[0049] (1) For the first time, a graph neural network solution was used in the toe tapping task to achieve end-to-end feature extraction and classification, and a supervised contrastive learning strategy and a multi-stream joint sparse learning mechanism were embedded to improve performance. The supervised contrastive learning strategy was used to constrain the feature extraction process under fine-grained tasks, and the discriminative feature space of different categories was constructed, thereby enhancing the specificity and reliability of the features of each category. By applying joint sparse constraints to the weights of the fully connected layers of each stream, the similarity and discriminability of the two-stream features were enhanced, thereby achieving sparse selection of classification features. By designing a spatiotemporal interactive graph convolution module, the long-range dependency relationship between joint points in the spatiotemporal domain was mined according to the similarity of features, and effective spatiotemporal feature modeling was achieved with the help of the information propagation mechanism.

[0050] (2) The toe tapping task has the largest dataset and the best evaluation effect. The predicted evaluation results have the highest correlation with the doctors’ clinical scores, and the results are relatively more reliable.

[0051] (3) Objective scoring results can be obtained by simply taking a video of the patient performing the toe tapping task with a regular camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Shown is a flow chart of an embodiment of a method for evaluating a Parkinson's disease toe tapping task according to the present invention;

[0053] Figure 2 Shown is a schematic diagram of a PD patient performing a toe tapping action in an embodiment;

[0054] Figure 3 It is a schematic diagram showing the structure of an evaluation method for the Parkinson's disease toe tapping task in one embodiment of the present invention;

[0055] Figure 4 It is a schematic diagram showing the structure of a spatiotemporal interactive graph convolution module in one embodiment of the present invention;

[0056] FIG5( a ) is a schematic diagram showing the ROC curve of the classification results of various categories of the experimental data set of the present invention;

[0057] FIG5( b ) is a schematic diagram showing a confusion matrix of classification results of various categories of the experimental data set of the present invention;

[0058] Figure 6 Shown is a schematic diagram of the distribution of results of ten repeated experiments of the Parkinson's disease toe tapping task evaluation method of the present invention;

[0059] Figure 7 Shown is a schematic structural diagram of an evaluation system for the Parkinson's disease toe tapping task in one embodiment of the present invention;

[0060] Figure 8 Shown is a schematic structural diagram of an evaluation terminal for the Parkinson's disease toe tapping task in one embodiment of the present invention;

[0061] Fig. 9 FIG. 4 is a schematic structural diagram of another embodiment of the Parkinson's disease toe tapping task evaluation system of the present invention. DETAILED DESCRIPTION

[0062] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0063] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0064] The evaluation method and system, storage medium and terminal of the Parkinson's disease toe tapping task of the present invention extract the human body's joint point coordinate sequence from the toe tapping task video with the help of an advanced human posture estimation algorithm, and then use the position information and speed information represented by the joint point coordinate sequence as the input of a two-stream network, i.e., a joint position stream and a joint speed stream. Each stream is composed of three space-time-channel units, one spatiotemporal interactive graph convolution module, one average pooling layer and one fully connected layer. Among them, a supervised contrast learning strategy is applied to the output features of the average pooling layer, and the category prior knowledge is used to construct and strengthen the specific feature space of each fine-grained category. The weights of the two stream fully connected layers are constrained by a multi-stream joint sparse learning mechanism, which enhances the similarity and discriminability of the position and speed features. By designing a spatiotemporal interactive graph convolution module, the long-range dependency between joint points in the spatiotemporal domain is mined according to the similarity of the features, and effective spatiotemporal feature modeling is achieved with the help of an information propagation mechanism.

[0065] like Figure 1 As shown, in one embodiment, the evaluation method of the Parkinson's disease toe tapping task of the present invention comprises the following steps:

[0066] Step S1, obtaining video information containing a toe-slapping action of a Parkinson's disease patient, wherein the toe-slapping action is a standard action required for performing a toe-slapping task assessment.

[0067] Specifically, the data collection work of the PD patient toe tapping task can be completed through a video acquisition device with an ordinary camera, such as a video camera, a smart phone, a tablet, etc., and sent to the evaluation terminal of the Parkinson's disease toe tapping task of the present invention in a wired or wireless manner.

[0068] During the collection process, Figure 2 As shown, the video acquisition device needs to be fixed in front of the PD patient and remain motionless. The PD patient needs to sit on a straight-backed chair with armrests with both feet on the ground. Both feet are tested separately. The PD patient puts the heel on the ground in a suitable position, and then taps the ground with the toes 10 times with the largest amplitude and fastest speed.

[0069] Step S2: extracting a human joint point coordinate sequence from the video information, and acquiring joint position information and joint velocity information based on the human joint coordinate sequence.

[0070] Specifically, in order to unify the format of input data, the videos of the left foot performing the task were uniformly flipped horizontally to videos of the right foot performing the task. Then, the advanced human posture estimation model OpenPose was used to process the video of PD patients performing the toe-slapping task frame by frame to extract the coordinate sequence of the foot joints of PD patients. Then, the first 150 frames of each video were empirically set as the time series length of the input data, and the y coordinate sequence of two joints (right big toe and right ankle) closely related to the toe-slapping action was selected as the input data, and z-score normalization was performed.

[0071] In addition, according to the extracted human joint point position coordinate sequence, the coordinates of the same joints in adjacent frames are subtracted to obtain the motion coordinates, which contain the velocity information of the corresponding joints, thereby obtaining the joint velocity information of the human joint points. According to the human joint point position coordinate sequence, the joint position information can be obtained.

[0072] Step S3, inputting the joint position information and the joint velocity information into the joint position flow network and the joint velocity flow network respectively; the joint position flow network and the joint velocity flow network both include 3 space-time-channel units, 1 spatiotemporal interactive graph convolution module, 1 average pooling layer and 1 fully connected layer.

[0073] Specifically, Figure 3 As shown, the joint position information and the joint velocity information are respectively input into a dual-stream network consisting of a joint position stream network and a joint velocity stream network. Both the joint position stream network and the joint velocity stream network include 3 space-time-channel units (STC), 1 spatiotemporal interactive graph convolution module (ST-IGCM), 1 average pooling layer (GAP) and 1 fully connected layer (FC). Among them, the space-time-channel unit is composed of spatial graph convolution, temporal convolution and a Squeeze-and-Excitation module for the channel. After these three space-time-channel units, the input features of all previous units will be spliced ​​in the channel dimension, thereby promoting the effective transmission of features through a densely connected mode.

[0074] In one embodiment of the present invention, supervised contrast (Supervised Contrastive Learning, SCL) is applied to the output features of the average pooling layer, and the features of samples of the same category are brought closer and the features of samples of different categories are separated by category prior knowledge to construct and strengthen the specific feature space of each fine-grained category. Specifically, in supervised classification tasks, the category prior knowledge of samples can provide rich supervisory information for learning discriminative sample features, which helps to improve the performance of fine-grained classification tasks. Contrastive learning methods have been applied in many self-supervised related works. Without using category prior knowledge, it constructs positive and negative pairs of input samples by designing pre-tasks. Model learning brings positive pairs as close as possible in the embedding space and negative pairs as far apart as possible, which ultimately enables the model to extract good sample features.

[0075] In theory, the supervised contrastive learning strategy should achieve the following effect as much as possible: S(f(X i ),f(X j ))>>S(f(X i ),f(X k )). Among them, X i and X j are samples of the same category, forming a positive pair, X i and X k are samples of different categories, constituting negative pairs. f is the mapping function, and S(·,·) is the scoring function for calculating the similarity of samples. In an ideal situation, positive pairs have a higher similarity and thus a higher score, while negative pairs have a lower score. Therefore, this idea can guide the model to maximize the consistency between the features of similar samples.

[0076] Assume that the high-level features extracted by the sample through the network body are Where N is the batch size and C is the number of channels. First, the feature matrix of each sample (such as ) performs normalization using the Euclidean norm on the channel dimension:

[0077]

[0078] Among them, ε is a very small value to avoid the denominator being 0. Then, for sample i, the similarity between it and other samples is calculated by the inner product, so as to obtain its loss function value in the supervised contrastive learning strategy

[0079]

[0080] in, is the number of samples in the batch that are of the same category as sample i. [·]is an indicator function. If the formula in [·] holds, it is equal to 1, otherwise it is equal to 0. <·,·> represents the inner product operation. In the above formula, only the similarity calculation results between the positive samples are retained. The minimization of represents the maximization of the similarity of the positive samples. After that, the supervised contrastive learning loss of N samples in the same batch is summarized as follows:

[0081]

[0082] Since the model of the present invention adopts a dual-stream framework, the supervised contrastive learning strategy is applied to each stream respectively, and then summed up to obtain the final supervised contrastive learning loss function:

[0083] Therefore, by designing a supervised contrastive learning strategy applied to the dual-stream framework of the present invention, it is possible to use category supervision information to construct a discriminative category feature space in fine-grained classification tasks, so that the model can learn to bring the features of samples of the same category closer and the features of samples of different categories farther apart, thereby enhancing the specificity of features of different categories.

[0084] In one embodiment of the present invention, the weights of the fully connected layers of the joint position stream network and the joint velocity stream network are constrained by a multi-stream joint sparse learning mechanism (MJSL) to enhance the similarity and discriminability of position features and velocity features. Specifically, since the ultimate goal of the joint position stream and the joint velocity stream is the same fine-grained classification task, there is a potential correlation between the two streams. In addition, although the backbone of the network has extracted high-level spatiotemporal features, the features of each stream are still redundant. Therefore, it is necessary to enhance the similarity and discriminability of the joint position stream and velocity stream features. Therefore, the multi-stream joint sparse learning mechanism of the present invention integrates the joint sparse constraint term into the loss function of the model, and constrains the weight matrix of the fully connected layer of each stream to promote similarity between streams and sparsity within streams. Assume that the weight matrices of the fully connected layers of multiple streams are respectively Where C is the number of input features of the fully connected layer, and K is the number of categories of the classification task, then the joint sparse constraint term can be expressed as:

[0085]

[0086] in, It is the matrix corresponding to the kth category in the fully connected layer weights of S streams Make stitching.

[0087] For the fully connected layer weights of each category, the multi-stream joint sparse learning mechanism first calculates the l2-norm in the multi-stream dimension to promote the similarity between multiple streams, then calculates the l1-norm in the channel dimension to promote the sparsity of feature selection, and finally summarizes and averages the calculation results of each category. As the network is optimized, the weight value connecting the redundant features and the category prediction score will be minimized to 0. If the cth row of the channel dimension of the weight matrix corresponding to category k is reduced to 0, it means that the features of this row are redundant for the prediction of category k, so the prediction output will not be affected by the corresponding input features. Formally, the constraint process of each category is similar to the calculation process of l2,1-norm. As a robust feature selection method, the effectiveness of l2,1-norm has been verified in many studies.

[0088] Therefore, the present invention, by designing a multi-stream joint sparse learning mechanism that constrains the weights of the fully connected layer, can enable the network to adaptively eliminate potential redundant features, achieve sparser discriminative feature selection, and make the model learning more robust and stable.

[0089] In addition, most of the research on skeleton-based human action recognition using graph neural networks follows the basic idea proposed by Yan et al.: the human skeleton sequence extracted from the video is constructed as a space-time graph, and then the high-level spatiotemporal features are gradually extracted through the space-time graph convolution network. The basic spatiotemporal unit of the network is composed of spatial graph convolution and temporal convolution. This approach not only breaks the unified information exchange between the spatial domain and the temporal domain, but also hinders the direct information exchange across spatiotemporal joints.

[0090] The present invention proposes Figure 4 The spatiotemporal interaction graph convolution module shown in Figure 1 is used to mine and explicitly model complex cross-temporal long-range dependencies. Assume that the spatiotemporal interaction graph can be represented as G = {V, E}. All the relevant nodes required for the task extracted from a video constitute the vertex set V of the spatiotemporal interaction graph. The edge set E is constructed by calculating the similarity of the attributes (i.e., features) of different vertices. Specifically, given the input feature graph is Among them C in , V are the number of channels and the number of vertices respectively, and then f in Through two 1×1 convolution operations ξ, Mapped to the low-dimensional feature space, the intermediate feature map f is obtained ξ , Among them, f ξ =W ξ f in , W ξ and are two 1×1 convolution operations ξ and Parameters.

[0091] Finally, the spatiotemporal similarity adjacency matrix is ​​obtained through matrix multiplication and softmax normalization operation as shown below

[0092]

[0093] Among them, element A ij Represents the vertex v in the space-time domain i and v j Then, according to the basic formula of graph convolution operation proposed by Kipf et al., we further use formula f out =Af in W defines the graph convolution operation in the spatiotemporal interaction graph convolution module, and obtains the output feature map of the spatiotemporal interaction graph convolution module Among them, W is the weight function implemented by 1×1 convolution operation.

[0094] Therefore, the spatiotemporal interactive graph convolution module can calculate and utilize the similarity of each node feature in the spatiotemporal domain, adaptively construct long-range direct dependencies across time and space, and realize direct communication of information, thus forming a powerful spatiotemporal feature extractor.

[0095] Finally, the total loss function of the two-stream model of the present invention is composed of the cross entropy term Supervised Contrastive Learning and multi-stream joint sparse items It consists of three parts, namely

[0096]

[0097] Among them, y and Represent the true label and predicted probability of the sample respectively, λ1 and λ2 are balanced and Hyperparameters of .

[0098] Step S4: input the output scores of the joint position flow network and the joint velocity flow network into a fully connected layer, and select the evaluation score with the highest probability value as the evaluation score of the Parkinson's disease toe tapping task.

[0099] Specifically, the probabilities of each score value (0, 1, 2, 3, 4) output by the joint position flow network and the joint velocity flow network are input into a fully connected layer, and the probabilities of the 0, 1, 2, 3, and 4 categories are calculated by softmax. The category with the largest probability value is the evaluation result of the current video, that is, the score of the current patient performing the toe tapping task.

[0100] In addition, according to the output probability value of the evaluation method for the Parkinson's disease toe tapping task of the present invention, the predicted category of the current video can be obtained. After multiple experiments, the predicted category can be compared with the doctor's score, and the evaluation indicators such as the accuracy, acceptable accuracy, precision, recall, F1 score and area under the ROC curve of the evaluation method for the Parkinson's disease toe tapping task of the present invention can be calculated to evaluate the performance of the evaluation method for the Parkinson's disease toe tapping task of the present invention.

[0101] The evaluation method for the Parkinson's disease toe tapping task of the present invention is verified by using specific examples below.

[0102] Table 1 shows the classification performance of the evaluation method for the Parkinson's disease toe tapping task of the present invention, namely, the accuracy (Acc), acceptable accuracy (Acceptable Acc), precision (Prec), recall (Rec), F1 score (F1) and area under the receiver operating characteristic curve (ROC) (AUC) of each category, as well as the total accuracy and total acceptable accuracy of all categories. As can be seen from the table, the evaluation method for the Parkinson's disease toe tapping task of the present invention achieves an accuracy of 70.04% and an acceptable accuracy of 98.70%, and achieves an acceptable accuracy of more than 89% in each category.

[0103] Table 1 Classification performance of the evaluation method for the Parkinson's disease toe tapping task of the present invention

[0104]

[0105]

[0106] FIG5(a) shows the ROC curves of the classification results of each category of the experimental data set of the present invention, indicating that the AUC values ​​of each category are relatively high, with an average AUC value of 0.89. The confusion matrix of the classification results is shown in FIG5(b), indicating that the classification effects of 1, 2, and 4 points are relatively good, while the classification effects of 0 and 3 points are not good. These results reflect the excellent performance of the proposed method in the fine-grained classification task of the present invention.

[0107] The present invention conducts a wealth of ablation experiments to comprehensively evaluate the effectiveness and coordination of each component in the proposed method. The experimental results are shown in Table 2, and each component has an effective improvement in accuracy. The baseline network is a two-stream network with each stream consisting of three space-time-channel units.

[0108] Table 2. Ablation experiment results

[0109]

[0110] In order to prove the stability of the present invention, the five-fold cross validation experiment was repeated 10 times, and the order of samples was randomly shuffled in each experiment. The distribution of the obtained results is as follows: Figure 6 As shown in the figure, the fluctuation ranges of the accuracy and acceptable accuracy of the ten experiments are relatively small, which are 1.16% and 0.43% respectively. These results further prove that the present invention has high reliability.

[0111] The proposed method is compared with five other advanced skeleton-based action recognition models, including CNN models (i.e., Two-stream CNN) and GCN models (i.e., ST-GCN, Js-AGCN, Bs-AGCN, 2s-AGCN, and Motif-STGCN). Table 3 shows the comparison results with these models, indicating that the proposed model achieves the highest accuracy and the best values ​​in other evaluation indicators compared with other CNN and GCN models. These comparisons further demonstrate the superior performance of the proposed method in the toe-strike fine-grained evaluation task.

[0112] Table 3. Comparison results of the present invention with other advanced skeleton-based action recognition models

[0113]

[0114] The present invention also investigates other existing studies on automatic evaluation of toe-slap tasks and makes a comprehensive comparison. Since Kim et al. and Li et al. both used correlation analysis in their evaluation, the present invention also conducted a correlation analysis on the prediction results of the spatiotemporal interactive graph convolutional network combined with contrast and sparse learning and the clinical scores provided by doctors, and the obtained Pearson correlation coefficient was 0.84. Compared with sensor-based methods, the scheme of the present invention achieved the highest accuracy / correlation coefficient in classification / correlation analysis. Video-based methods all use human posture estimation technology to extract joints in videos. However, Li et al. then extracted manually defined features and used machine learning classifiers, while the present invention directly designed a deep learning model to achieve end-to-end feature extraction and classification. The results show that the evaluation results of the scheme of the present invention have significant advantages. Therefore, compared with other existing related studies, the scheme of the present invention achieves optimal performance on the largest toe-slap dataset.

[0115] Table 4. Comparison results of the automatic evaluation study of the present invention and other toe tap tasks

[0116]

[0117] like Figure 7As shown, in one embodiment, the evaluation system for the Parkinson's disease toe tapping task of the present invention includes an acquisition module 71 , an extraction module 72 , a processing module 73 and an evaluation module 74 .

[0118] The acquisition module 71 is used to acquire video information containing toe tapping movements of Parkinson's disease patients, where the toe tapping movements are standard movements required for toe tapping task assessment.

[0119] The extraction module 72 is connected to the acquisition module 71, and is used to extract a human joint point coordinate sequence from the video information, and acquire joint position information and joint velocity information based on the human joint coordinate sequence.

[0120] The processing module 73 is connected to the extraction module 72, and is used to input the joint position information and the joint velocity information into the joint position flow network and the joint velocity flow network respectively; the joint position flow network and the joint velocity flow network both include 3 space-time-channel units, 1 spatiotemporal interaction graph convolution module, 1 average pooling layer and 1 fully connected layer.

[0121] The evaluation module 74 is connected to the processing module 73, and is used to input the output scores of the joint position flow network and the joint velocity flow network into the fully connected layer, and select the evaluation score with the highest probability value as the evaluation score of the Parkinson's disease toe tapping task.

[0122] Among them, the structures and principles of the acquisition module 71, the extraction module 72, the processing module 73 and the evaluation module 74 correspond one to one to the steps in the above-mentioned evaluation method of the Parkinson's disease toe tapping task, so they are not repeated here.

[0123] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements, or they can all be implemented in the form of hardware, or some modules can be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example: the x module can be a separately established processing element, or it can be integrated in a certain chip of the above device. In addition, the x module can also be stored in the memory of the above device in the form of program code, and called and executed by a certain processing element of the above device. The implementation of other modules is similar. These modules can be integrated together in whole or in part, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software. The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), one or more microprocessors (DSP), one or more field programmable gate arrays (FPGA), etc. When a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. These modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0124] The storage medium of the present invention stores a computer program, which, when executed by a processor, implements the above-mentioned evaluation method for the Parkinson's disease toe tapping task. Preferably, the storage medium includes: ROM, RAM, disk, U disk, memory card or CD, etc., which can store program codes.

[0125] like Figure 8 As shown, in one embodiment, the evaluation terminal for the Parkinson's disease toe tapping task of the present invention includes: a processor 81 and a memory 82 .

[0126] The memory 82 is used to store computer programs.

[0127] The memory 82 includes: ROM, RAM, disk, USB flash drive, memory card or CD and other media that can store program codes.

[0128] The processor 81 is connected to the memory 82 and is used to execute the computer program stored in the memory 82 so that the evaluation terminal for the Parkinson's disease toe tap task executes the above-mentioned evaluation method for the Parkinson's disease toe tap task.

[0129] Preferably, the processor 81 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0130] like Fig. 9 As shown, in one embodiment, the evaluation system for the Parkinson's disease toe tapping task of the present invention includes the above-mentioned Parkinson's disease toe tapping task evaluation terminal 91 and a video acquisition device 92.

[0131] The video acquisition device 92 is connected to the evaluation terminal 91 of the Parkinson's disease toe tapping task, and is used to acquire video information containing the toe tapping action of the Parkinson's disease patient and send it to the evaluation terminal 91 of the Parkinson's disease toe tapping task.

[0132] In summary, the evaluation method and system, storage medium and terminal of the Parkinson's disease toe tapping task of the present invention use a graph neural network solution for the first time in the toe tapping task to realize end-to-end feature extraction and classification, and embed a supervised contrast learning strategy and a multi-stream joint sparse learning mechanism to improve performance; wherein the supervised contrast learning strategy is used to constrain the feature extraction process under fine-grained tasks, and constructs a discriminative feature space of different categories, thereby enhancing the specificity and reliability of features of each category; by applying a joint sparse constraint to the weights of the fully connected layers of each stream, the similarity and discriminability of the dual-stream features are enhanced, thereby realizing sparse selection of classification features; by designing a spatiotemporal interactive graph convolution module, the long-range dependency relationship between joint points in the spatiotemporal domain is mined according to the similarity of features, and effective spatiotemporal feature modeling is realized with the help of an information propagation mechanism; the data set of the toe tapping task is the largest, the evaluation effect is the best, the predicted evaluation results have the highest correlation with the doctor's clinical score, and the results are relatively more reliable; objective scoring results can be obtained by only taking a video of a patient performing a toe tapping task with an ordinary camera. Therefore, the present invention effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

[0133] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. An evaluation method for the toe tapping task in Parkinson's disease, Features: The following steps are involved: Acquiring video information containing a toe-slapping action of a Parkinson's disease patient, wherein the toe-slapping action is a standard action required for performing a toe-slapping task assessment; Extracting a human joint point coordinate sequence from the video information, and acquiring joint position information and joint velocity information based on the human joint coordinate sequence; The joint position information and the joint velocity information are input into a joint position flow network and a joint velocity flow network respectively; the joint position flow network and the joint velocity flow network each include 3 space-time-channel units, 1 spatiotemporal interaction graph convolution module, 1 average pooling layer and 1 fully connected layer; Input the output scores of the joint position flow network and the joint velocity flow network into a fully connected layer, and select the evaluation score with the highest probability value as the evaluation score of the Parkinson's disease toe tapping task; The space-time-channel unit is composed of a spatial graph convolution, a temporal convolution, and a squeeze-and-excitation module for the channel; In the spatiotemporal interaction graph convolution module, the spatiotemporal interaction graph is represented as G = {V, E}, all the relevant nodes required for the task extracted from a video constitute the vertex set V of the spatiotemporal interaction graph, and the edge set E is formed by calculating the similarity of the attributes of different vertices; the output feature graph f of the spatiotemporal interaction graph convolution module out =Af in W, where A is the spatiotemporal similarity adjacency matrix, f in For the input feature map, W is the weight function implemented by a 1×1 convolution operation.

2. The method for evaluating the Parkinson's disease toe tapping task according to claim 1, Features: A human joint point coordinate sequence is extracted from the video information based on the human pose estimation model OpenPose.

3. The method for evaluating the Parkinson's disease toe tapping task according to claim 1, Features: A supervised contrastive learning strategy is applied to the output features of the average pooling layer, which uses category prior knowledge to bring the features of samples of the same category closer and separate the features of samples of different categories, so as to construct and strengthen the specific feature space of each fine-grained category.

4. The method for evaluating the Parkinson's disease toe tapping task according to claim 1, Features: The weights of the fully connected layers of the joint position stream network and the joint velocity stream network are constrained through a multi-stream joint sparse learning mechanism to enhance the similarity and discriminability of position features and velocity features.

5. An evaluation system for the toe tapping task in Parkinson's disease, Features: It includes an acquisition module, an extraction module, a processing module and an evaluation module; The acquisition module is used to acquire video information containing toe tapping movements of Parkinson's disease patients, wherein the toe tapping movements are standard movements required for toe tapping task assessment; The extraction module is used to extract a human joint point coordinate sequence from the video information, and obtain joint position information and joint velocity information based on the human joint coordinate sequence; The processing module is used to input the joint position information and the joint velocity information into a joint position flow network and a joint velocity flow network respectively; the joint position flow network and the joint velocity flow network each include 3 space-time-channel units, 1 spatiotemporal interaction graph convolution module, 1 average pooling layer and 1 fully connected layer; The evaluation module is used to input the output scores of the joint position flow network and the joint velocity flow network into a fully connected layer, and select the evaluation score with the highest probability value as the evaluation score of the Parkinson's disease toe tapping task; The space-time-channel unit is composed of a spatial graph convolution, a temporal convolution, and a squeeze-and-excitation module for the channel; In the spatiotemporal interaction graph convolution module, the spatiotemporal interaction graph is represented as G = {V, E}, all the relevant nodes required for the task extracted from a video constitute the vertex set V of the spatiotemporal interaction graph, and the edge set E is formed by calculating the similarity of the attributes of different vertices; the output feature graph f of the spatiotemporal interaction graph convolution module out =Af in W, where A is the spatiotemporal similarity adjacency matrix, f in For the input feature map, W is the weight function implemented by a 1×1 convolution operation.

6. A storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method for evaluating the Parkinson's disease toe tapping task according to any one of claims 1 to 4 is implemented.

7. An evaluation terminal for the Parkinson's disease toe tapping task, It is characterized in that include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory, so that the Parkinson's disease toe tap task evaluation terminal performs the Parkinson's disease toe tap task evaluation method according to any one of claims 1 to 4.

8. An evaluation system for the toe tapping task in Parkinson's disease. Features: An evaluation terminal and a video acquisition device for the Parkinson's disease toe tapping task according to claim 7; The video acquisition device is used to acquire video information containing the toe tapping action of a Parkinson's disease patient and send the video information to the evaluation terminal of the Parkinson's disease toe tapping task.