A method for evaluating Parkinson's disease based on video gait analysis

By combining graph convolution neural network and Transformer model, video gait data is processed, and excessive information and occlusion problems in video gait analysis are solved, and the accurate assessment of gait in patients with Parkinson's disease is achieved, providing a more objective and Accurate evaluation methods improve the quality of motor function assessment in patients with Parkinson's disease.

CN118072385BActive Publication Date: 2024-11-12HARBIN INSTITUTE OF TECHNOLOGY (SHENZHEN) (INSTITUTE OF SCIENCE AND TECHNOLOGY INNOVATION HARBIN INSTITUTE OF TECHNOLOGY SHENZHEN)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410103176.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-24
Publication Date
2024-11-12
Estimated Expiration
2044-01-24

AI Technical Summary

Technical Problem

In the prior art, video gait analysis has problems such as excessive information, more interference and strong subjective scores, which leads to inadequate evaluation of motor function in patients with Parkinson's disease.

Method used

Using a method based on video gait analysis, human posture recognition and key point extraction are performed through the combination of graph convolutional neural network and Transformer model, masks and random noise are used to deal with occlusion problems, and multi-dimensional features are extracted to build a classification model to realize Automatic evaluation of Parkinson's disease gait.

Benefits of technology

It improves the accuracy and objectivity of gait assessment in patients with Parkinson's disease, reduces the impact of occlusion, and provides a more intuitive and accurate condition rating classifier that can better assist clinical diagnosis and treatment decisions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118072385B_ABST
    Figure CN118072385B_ABST
Patent Text Reader

Abstract

The present invention provides a Parkinson's disease assessment method based on video gait analysis, which includes obtaining gait video data to be evaluated, processing the gait video data, and establishing a video data set; performing human posture recognition on a single-frame video image, and obtaining a heat map of key points of the human body for each frame image; then using graph convolution to complete key point recognition based on the time domain and space domain; and building a classification model by combining a dual-stream spatiotemporal adaptive graph convolutional neural network and a transformer model. The present invention is mainly used to solve the problems existing in the prior art, such as excessive video information, more interference, and strong subjectivity in scoring.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of medical detection technology, and in particular to a Parkinson's disease assessment method based on video gait analysis. Background Art

[0002] Parkinson's disease (PD) is a common chronic neurodegenerative disease that affects the central nervous system, mainly affecting the motor nervous system. Its symptoms usually appear slowly over time, and the most obvious symptoms in the early stage are tremor, limb stiffness, motor dysfunction and gait abnormality. The classification of motor symptom subtypes of PD is generated through clinical motor data, and is mainly divided into tremor dominant PD (TD-PD) and postural disorder / gait instability based on symptom manifestations. Parkinson's disease (PD) is a progressive neurodegenerative disease, and movement disorders are the typical clinical features of the disease. At present, the main clinical evaluation standard for PD patients is the Unified Parkinson's Disease Rating Scale (UPDRS), which was named MDS-UPDRS by the Movement Disorder Society (MDS) in 2008. Movement disorder symptoms are scored from 0 to 4 points, with 0 points indicating normal and 4 points indicating severe. The clinical practice of this scale faces two major problems: 1) A comprehensive motor assessment by a well-trained and experienced neurologist is time-consuming, requiring at least half an hour, and the results obtained are subjective with large inter-rater variability; 2) Travel restrictions, timely follow-up of progressive diseases, and sparse distribution of PD experts also create challenges in providing medical care for PD patients. Therefore, an objective and automated remote monitoring motor function assessment system for Parkinson's disease patients is urgently needed, which has become a research hotspot in academia and clinical circles.

[0003] Gait dyskinesia is one of the common movement disorders in patients with Parkinson's disease, which is usually evaluated by clinical analysis of the patient's stride, speed, turning, and arm swing. Traditional evaluation methods have the problems of subjectivity and invasiveness because they rely on the subjective judgment of doctors and patients need to wear sensors. In recent years, automated evaluation methods based on video technology, deep learning algorithms, and high-performance computing technologies have presented new prospects. Video technology has the advantages of being non-contact, scalable, and non-invasive. Instead of wearing sensors, patients use cameras to evaluate their movements, which improves comfort and convenience. This allows video technology to be extended to patients' daily life environments, not just limited to use scenarios in medical institutions. The introduction of deep learning algorithms further enhances the degree of automation and accuracy. These algorithms can automatically learn and extract features, reduce the reliance on manual feature engineering, and are expected to improve the accuracy of gait dyskinesia evaluation. By combining video technology and deep learning algorithms, effective automatic quantification of gait dyskinesia in patients with Parkinson's disease can be achieved, providing more comprehensive information for disease monitoring and treatment.

[0004] The main problems existing in the prior art are as follows: 1) There are many interferences: In the analysis of monocular video, since the video contains more information, the computational cost is high and contains various environmental information and information of other non-target objects such as patients' family members and doctors, which may have a greater negative impact on our results. The video should be partially preprocessed as the input for feature extraction, rather than the original video image, because the latter may be sensitive to the subject's shoes, clothes and background, while the former is robust to most of these factors. Therefore, this requires that the people participating in the shooting cannot wear skirts and loose clothes, the background and the color of the clothes cannot be too close, and the floor leather is laid on the ground to prevent reflection. The current human posture recognition algorithms are all based on public data sets and are not necessarily perfectly suitable for the patient's gait analysis. Due to the blurring of some frames caused by the patient's movement during the shooting, the self-occlusion of the patient's joints caused by the front camera during the patient's movement, and the patient's gait disorder is more serious. When the family members accompany him, the elbow or the lens is blocked when turning around, etc., the patient's body key points are blocked. Therefore, based on this background, this application hopes to refine and improve the human posture point recognition after the mainstream human posture recognition.

[0005] 2) High requirements for front key point data: When obtaining the time series coordinates of the key points of human posture, the data represents the different motion states of the patient and contains spatial information and time information, which makes the overall data complex and presents characteristics of different dimensions. Because sometimes the patient's entire body cannot be fully photographed during the shooting process, some of the key points of the human body are lost.

[0006] 3. Subjectivity of scoring: In practice, gait disorder scoring is subjectively scored by experts based on the MDS-UPDRS gait scale standard, and the results are highly subjective. Although the UPDRS score has been widely used as a standard worldwide, there is also interference from strong subjective factors of the evaluator and the person being evaluated. The evaluations of doctors of different seniority often have certain differences, and there is often a large bias. This bias often affects the accurate classification of PD disease, the formulation of treatment plans, the adjustment of drugs, and the decision of whether to perform surgical treatment. Therefore, standardized disease symptom identification and condition assessment play an extremely important role in the diagnosis, differential diagnosis, and formulation of treatment plans for PD patients. Therefore, a more objective motion scoring algorithm is needed to perform a Parkinson's disease gait scoring system based on the extracted features. It is also necessary to compare traditional methods such as support vector machines and random forests, as well as deep learning graph convolutional neural networks, and compare and analyze multiple models based on accuracy, recall, precision, and F-measure to select the optimal model. Summary of the invention

[0007] The purpose of the present invention is to provide a Parkinson's disease assessment method based on video gait analysis, which is used to solve the problems existing in the prior art, such as excessive video information, much interference, and strong subjectivity in scoring.

[0008] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0009] A method for evaluating Parkinson's disease based on video gait analysis, the method comprising the following steps:

[0010] Acquire gait video data to be evaluated, process the gait video data, and establish a video data set;

[0011] Perform human body posture recognition on a single frame of video and obtain the heat map of key points of the human body in each frame. Then use graph convolution to complete key point recognition based on time and space domains.

[0012] A classification model is built by combining a dual-stream spatiotemporal adaptive graph convolutional neural network and a transformer model.

[0013] According to a Parkinson's disease assessment method based on video gait analysis provided by the present invention, the heat map is processed, including:

[0014] A mask is set for the heat map identified in each frame of the gait video to reduce negative effects such as occlusion. The masking method includes:

[0015] The entire video frame is 0 or the heat map is all 0 to randomly mask several frames, indicating that the entire frame is occluded or has low confidence;

[0016] Point-by-point occlusion is achieved by randomly setting the heat maps of some key points to zero to simulate the occlusion of some joint points;

[0017] Since the output of the 2D pose estimator is not strictly Gaussian, random noise is introduced into the heat map of the input sequence.

[0018] According to a Parkinson's disease assessment method based on video gait analysis provided by the present invention, partial data is extracted from the corresponding human skeleton key point coordinates, and features are extracted from the partial data. The step length is defined based on the left and right feet in the body25 model, which is expressed as formula (1):

[0019]

[0020] Among them, P9P 11 is the pixel distance of the left leg, P 12 P 14 is the pixel distance of the right leg.

[0021] According to a Parkinson's disease assessment method based on video gait analysis provided by the present invention, extracting the corresponding features extracted based on the neural network includes:

[0022] Convolutional neural network features: Convolutional neural networks learn the spatial hierarchy in images through convolution and pooling layers. Convolution layers extract features of different sizes and complexities, while pooling layers reduce the dimensionality of feature maps.

[0023] Features of recurrent neural networks: Recurrent neural networks model time series data, and the hidden state of the recurrent neural network contains the contextual information in the sequence;

[0024] Autoencoder features: The autoencoder compresses the input data into a low-dimensional representation and then reconstructs the input to learn useful features of the data, which are used for tasks such as dimensionality reduction, denoising, and anomaly detection.

[0025] Transfer learning features: Use a pre-trained neural network model and use the output of its intermediate layers as a feature extractor.

[0026] According to a Parkinson's disease assessment method based on video gait analysis provided by the present invention, the method further performs: extracting similarity features of time series data, including:

[0027] The DTW algorithm is used to adjust the time alignment of the sampling points of the two time series to find the best match between the two, that is, to calculate the minimum cumulative distance between the two time series. After DTW matching, the DTW features are extracted to represent the similarity between the two time series.

[0028] According to a Parkinson's disease assessment method based on video gait analysis provided by the present invention, the DTW features include:

[0029] DTW distance: the minimum cumulative distance between two time series, used to indicate the similarity between the time series;

[0030] DTW path: a path that indicates how to align two time series for visualization and further analysis;

[0031] Matching point: a sampling point on the DTW path, used to represent the matching relationship between two time series;

[0032] DTW feature vector: a set of statistical values ​​used to describe the distribution of sampling points on the DTW path.

[0033] According to a Parkinson's disease assessment method based on video gait analysis provided by the present invention, the constraints of the DTW algorithm include:

[0034] Boundedness: Two sequences C and Q of unequal lengths start at (1, 1) and end at (m, n), where m is the length of sequence C and n is the length of sequence Q;

[0035] Monotonicity: The previous point P(k, l) and the next point P(i, j) in the DTW path.

[0036] According to a Parkinson's disease assessment method based on video gait analysis provided by the present invention, the following steps are further performed:

[0037] The DTW distance between the step length curve of each walking after the patient takes the medicine and the walking before taking the medicine is selected to represent a feature, that is, the DTW distance between the gait curve before taking the medicine and the first step length curve after taking the medicine becomes the first feature, and the DTW distance between the gait curve before taking the medicine and the second gait curve after taking the medicine becomes the second feature. And so on, the gait curve of patients with Parkinson's disease is compared with the step length curve of the normal control group to obtain the DTW distance as a reference feature.

[0038] According to a Parkinson's disease assessment method based on video gait analysis provided by the present invention, the classification model includes:

[0039] The human skeleton key points extracted from the gait video are used as the input of the classification model, and are divided into joint flow and bone flow. The transformer's self-attention mechanism is combined with the adaptive graph convolutional neural network as the basic block, and the adjacent basic blocks are connected through stacked hourglass to construct a two-stream classification model, so as to obtain the gait score of Parkinson's patients; wherein the joint flow corresponds to the pixel coordinate points of the key parts of the patient's body in each frame of each video, and the bone flow corresponds to the angle and pixel length of each part of the patient's body in each frame of each video, wherein two identical adaptive graph convolutional neural networks are used for shunting, and each shunting is an independent spatiotemporal adaptive graph convolution transformer stacked hourglass neural network, which is used to simulate the graphic structure of the human body key points and finally fuse the manually designed features to obtain refined action classification;

[0040] When simulating a graphic structure, A is defined as an adjacency matrix to simulate the human body structure, A1 is defined as the key point connection of the human body itself, A2 and A3 are defined as learnable parameters in the adaptive module, and the gait features extracted based on the coordinates of the key points of the human skeleton are input into the fully connected layer together with the output based on the spatiotemporal adaptive graph convolutional neural network to obtain the classification model; wherein, A=A1+A2+A3.

[0041] It can be seen that compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. The present invention performs human posture training by masking the training images and processing the training heat map, which effectively improves the accuracy of the algorithm based on occlusion (the patient is accompanied by his family members). In addition, since the existing popular human posture recognition algorithms are based on the training results of pictures and do not use the context in the video for processing, the present invention can make full use of the spatial and temporal information in the video to solve the problems of motion blur and occlusion.

[0043] 2. The present invention can still achieve a high accuracy in identifying key points when Parkinson's patients are exercising and may have self-occlusion. It can reflect motion features that are clinically intuitive to doctors and effective for disease grade classifiers. It can use the motion trajectory curve and motion features obtained by the posture estimation algorithm to more accurately distinguish Parkinson's patients and their gait disorder levels, and build a precise auxiliary assessment platform for Parkinson's symptoms based on computer vision technology.

[0044] 3. When dealing with the gait classification task for Parkinson's patients, the present invention proposes a transformer-based dual-stream adaptive graph convolutional neural network. The fusion of the adaptive graph convolutional neural network and the transformer can obtain multi-level feature representation, extract the spatiotemporal graph structure nodes and edge features, and globally integrate the nodes through the transformer's self-attention mechanism. This structure can better integrate the local features extracted by the adaptive graph convolutional neural network with the global dependency features extracted by the transformer into a powerful model.

[0045] The present invention also provides an electronic device, comprising:

[0046] a memory storing computer executable instructions;

[0047] a processor configured to execute the computer executable instructions,

[0048] Wherein, when the computer executable instructions are executed by the processor, the steps of any one of the above-mentioned methods for assessing Parkinson's disease based on video gait analysis are implemented.

[0049] The present invention also provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, it is used to implement the steps of any one of the above-mentioned methods for assessing Parkinson's disease based on video gait analysis.

[0050] It can be seen that the present invention also provides an electronic device and storage medium for a Parkinson's disease assessment method based on video gait analysis, which includes: one or more memories, and one or more processors. The memory is used to store program codes and intermediate data generated during program operation, storage of model output results, and storage of models and model parameters; the processor is used for processor resources occupied by code operation and multiple processor resources occupied when training models.

[0051] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The present invention is a flow chart of an embodiment of a method for evaluating Parkinson's disease based on video gait analysis.

[0053] Figure 2 It is a flow diagram of an embodiment of a method for evaluating Parkinson's disease based on video gait analysis of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0056] See also Figure 1 and Figure 2 The present invention provides a method for evaluating Parkinson's disease based on video gait analysis, the method comprising the following steps:

[0057] Step S1, obtaining gait video data to be evaluated, processing the gait video data, and establishing a video data set;

[0058] Step S2, perform human posture recognition on a single frame of the video, obtain a heat map of human key points in each frame; then use graph convolution to complete key point recognition based on the time domain and space domain; that is, statically refine human key point recognition based on the time dimension of the video stream, and perform mask processing on the input heat map; then extract multi-dimensional features based on human key points;

[0059] Step S3, build a classification model by combining a two-stream spatiotemporal adaptive graph convolutional neural network and a transformer model. That is, a two-stream spatiotemporal attention graph convolutional network (2S-ST-AGCN) is constructed, which simultaneously models static spatial information and dynamic temporal changes and integrates manually designed features for video assessment of PD gait movement disorders.

[0060] Among them, human posture recognition is performed on a single frame image in a gait video to obtain the coordinates of the key points of the human skeleton in each frame image of the gait video, and time series recognition is performed on the key point coordinates of the gait video to obtain the time dimension composed of the key point coordinates of each frame image; a heat map is constructed for the key point coordinates based on Gaussian distribution, and the heat map is processed; the connection between joints is established by constructing an attention mechanism through graph convolution; and the recognition result of the current frame is processed based on the time dimension using time information;

[0061] Extracting multi-dimensional gait features based on the coordinates of key points of the human skeleton, wherein the features include statistical features based on kinematics, corresponding features extracted based on neural networks, and similarity features of time series data;

[0062] The features of joint and bone streams are obtained by fitting a fully connected layer on the multi-dimensional feature vector, and the final classification model is obtained by fusing the multi-dimensional features and the features of the two-stream spatiotemporal graph convolutional neural network through a fully connected layer.

[0063] The gait video to be evaluated is analyzed using a classification model based on a two-stream spatiotemporal graph convolutional neural network to obtain the evaluation results.

[0064] In the above step S1, for human key point recognition based on spatiotemporal information, firstly, an input video is obtained through any prior art detector and tracker, such as Mask R-CNN and Yolov5. The position of the patient in the video is located in the human posture recognition algorithm and the human positioning algorithm. However, since it is difficult to directly evaluate the gait of the patient in the video, this embodiment chooses to use skeleton analysis to estimate the posture of the patient, such as Hrnet or Openpose.

[0065] After analyzing and evaluating multiple groups of videos, this embodiment formulates corresponding shooting requirements for data collection and shooting to obtain a data set with better recognition results. The relevant requirements are as follows: (1) The patient's entire body should be in the video; (2) The shooting device should be kept as still as possible during shooting to reduce errors caused by shaking and the camera should be 1.5m to 6m away from the person to ensure that the person appears in the middle of the video; (3) When shooting, only one patient should be in the video as much as possible, if necessary, to accompany the later research; (4) The patient should not leave the device as much as possible or occupy a small proportion in the video to prevent low credibility of bone points caused by low image resolution; (5) Shooting should be done from a frontal perspective; (6) Data volume: videos of about 70 patients in different medication states.

[0066] In the above step S2, the heat map is processed, including:

[0067] A mask is set for the heat map identified in each frame of the gait video to reduce negative effects such as occlusion. The masking method includes:

[0068] The entire video frame is 0 or the heat map is all 0 to randomly mask several frames, indicating that the entire frame is occluded or has low confidence;

[0069] Point-by-point occlusion is achieved by randomly setting the heat maps of some key points to zero to simulate the occlusion of some joint points;

[0070] Since the output of the 2D pose estimator is not strictly Gaussian, random noise is introduced into the heat map of the input sequence.

[0071] Specifically, human posture recognition is first performed on a single frame of a video using mainstream methods, such as Openpose, which proposes a method for effectively detecting the 2D posture of multiple people in an image. This embodiment will propose a new multi-frame human posture estimation framework that uses video frames with rich time cues to improve the accuracy of key point detection. After obtaining the key points of each frame in the video, a heat map is constructed based on the Gaussian distribution of the key points and the recognition results are improved through the following points to avoid the influence of adverse factors such as motion blur, target size, and occlusion:

[0072] 1. Set a mask for the heat map identified in each frame of the video to reduce negative effects such as occlusion. The main masking methods are as follows:

[0073] (1) The entire frame is 0 or the heat map is all 0 to randomly mask several frames, indicating that the entire frame is occluded or has low confidence. (2) Point-by-point occlusion is achieved by randomly setting the heat map of certain key points to zero (simulating the occlusion of certain joints). (3) Since the output of the 2D pose estimator is not strictly Gaussian, random noise is introduced in the heat map of the input sequence.

[0074] 2. Establish the connectivity between joints by constructing the attention mechanism through graph convolution.

[0075] 3. Use deep learning methods such as graph convolutional neural networks or deformable convolutions to refine the current frame recognition results based on the time dimension using time information.

[0076] In practical applications, in the field of Parkinson's disease research, supervised learning methods are often used to evaluate and classify Parkinson's disease levels. This embodiment uses a support vector machine classification model, a random forest classification model, and a neural network model to evaluate the gait classification of Parkinson's patients, and compares and analyzes the performance of the three models based on accuracy, recall, precision, and F-measure.

[0077] This embodiment will focus on realizing a more objective classification model and has a significant improvement over the traditional classification model. The bone sequence extracted from the gait video of PD patients is used as the input of the model, and it is divided into a joint stream and a feature stream (human skeleton length and angle). Each stream is a combination of an independent spatiotemporal adaptive graph convolutional neural network (2s-AGCN) and transformer, which is used to naturally simulate the graphic structure of human key points and capture the characteristics of the spatiotemporal domain.

[0078] Specifically, the classification model based on the dual-stream spatiotemporal graph convolutional neural network in this embodiment includes:

[0079] The human skeleton key points extracted from the gait video are used as the input of the classification model, and are divided into joint flow and bone flow. The transformer's self-attention mechanism is combined with the adaptive graph convolutional neural network as a basic block, and the adjacent basic blocks are connected through stacked hourglass to construct a two-stream classification model, so as to obtain the gait score of Parkinson's patients; wherein the joint flow corresponds to the pixel coordinate points of the key parts of the patient's body in each frame of each video, and the bone flow corresponds to the angle and pixel length of each part of the patient's body in each frame of each video, wherein two identical adaptive graph convolutional neural networks are used for shunting, and each shunt is an independent spatiotemporal adaptive graph convolutional neural network, which is used to simulate the graphic structure of the key points of the human body;

[0080] When simulating the graphic structure, we define A as the adjacency matrix to simulate the human body structure, define A1 as the key point connection of the human body itself and add virtual connections, define A2 and A3 as learnable parameters in the adaptive module, and input the gait features extracted based on the coordinates of the key points of the human skeleton into the fully connected layer together with the output of the spatiotemporal adaptive graph convolutional neural network to obtain the classification model; wherein, A=A1+A2+A3. Then pass it to the Transformer to globally interact and integrate the multi-level features. In each graph convolution transformer, we use stacked hourglass to learn features of different scales and restore the detailed information through upsampling. Specifically, our model framework mainly consists of 5 agcn and transformer modules, and in the two adjacent modules, we use stacked hourglass connections to learn information of different scales. In the transformer, our query (Q) and key (K) are calculated as Q=XW Q ,K=XW K , the attention weight can be calculated as The final weighted sum is Y = AX. The attention mechanism occurs in both the temporal and spatial domains, allowing the embedding of global class representations by associating different temporal windows as well as spatial windows.

[0081] In summary, this embodiment effectively improves the accuracy of the algorithm based on occlusion (patients are accompanied by family members) by masking the training images and processing the training heat map for human posture training, and because the existing popular human posture recognition algorithm is based on the training results of the picture and does not use the context in the video for processing, this embodiment can make full use of the spatial and temporal information in the video to solve the problems of motion blur and occlusion. This embodiment divides the entire gait process of the patient into three parts, namely walking in front of the camera, turning around and walking with the back to the camera, and filters the gait curve of the drawn Parkinson's patient to make its motion trajectory smoother. This embodiment extracts some key point features based on the human posture key point sequence in accordance with medicine, kinematics and statistics. This embodiment can still achieve a high key point recognition accuracy in the face of the posture of Parkinson's patients during movement and possible self-occlusion; it can reflect the motion features that are intuitive to doctors and effective for the disease level classifier; it can use the motion trajectory curve and motion features obtained by the posture estimation algorithm and can accurately distinguish Parkinson's patients and the gait disorder level of Parkinson's patients through the classification model, and build a precise auxiliary evaluation platform for Parkinson's symptoms based on computer vision technology. This embodiment uses deep learning methods such as graph convolutional neural networks or variable convolution to refine the current frame recognition result using time information.

[0082] In one embodiment, an electronic device is provided, which may be a server. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a Parkinson's disease assessment method based on video gait analysis is implemented.

[0083] Those skilled in the art will understand that the electronic device structure shown in this embodiment is only a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than those shown in this embodiment, or combine certain components, or have a different arrangement of components.

[0084] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0085] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0086] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0087] It can be seen that this embodiment also provides an electronic device and storage medium for a Parkinson's disease assessment method based on video gait analysis, which includes: one or more memories, and one or more processors. The memory is used to store program codes and intermediate data generated during program operation, storage of model output results, and storage of models and model parameters; the processor is used for processor resources occupied by code operation and multiple processor resources occupied when training models.

[0088] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0089] The above-mentioned embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and substitutions made by technicians in this field on the basis of the present invention shall fall within the scope of protection required by the present invention.

Claims

1. A method for evaluating Parkinson's disease based on video gait analysis, characterized in that: The method comprises the following steps: Acquire gait video data to be evaluated, process the gait video data, and establish a video data set; Perform human body posture recognition on a single frame of video and obtain the heat map of key points of the human body in each frame. Then use graph convolution to complete key point recognition based on time and space domains. A classification model is built by combining a dual-stream spatiotemporal adaptive graph convolutional neural network and a transformer model; During the training process, the heat map is processed, including: For the heat map generated for each frame in the gait video, the following methods are used to reduce negative effects such as occlusion; Randomly block a few frames by setting the entire video frame to 0 or the entire heat map to 0, indicating that the entire frame is blocked or has low confidence; By randomly setting the heat maps of some key points to zero, occluding them point by point to simulate the occlusion of some joint points; Since the output of the 2D pose estimator is not strictly Gaussian, random noise is introduced into the heat map of the input sequence; Among them, the corresponding features based on the neural network are extracted, including: Convolutional neural network features: Convolutional neural networks learn the spatial hierarchy in images through convolution and pooling layers. Convolution layers extract features of different sizes and complexities, while pooling layers reduce the dimensionality of feature maps. Recurrent Neural Network Features: Recurrent Neural Network models time series data, and the hidden state of the recurrent neural network contains the contextual information in the sequence; Autoencoder features: The autoencoder compresses the input data into a low-dimensional representation and then reconstructs the input to learn useful features of the data, which are used for tasks such as dimensionality reduction, denoising, and anomaly detection. Transfer learning features: Use a pre-trained neural network model and use the output of its intermediate layers as a feature extractor; Wherein, the classification model includes: The human skeleton key points extracted from the gait video are used as the input of the classification model, and are divided into joint flow and bone flow. The transformer's self-attention mechanism is combined with an adaptive graph convolutional neural network as a basic block, and adjacent basic blocks are connected through stacked hourglass to construct a two-stream classification model, so that the gait score of Parkinson's patients can be obtained; wherein the joint flow corresponds to the pixel coordinate points of the key parts of the patient's body in each frame of each video, and the bone flow corresponds to the angle and pixel length of each part of the patient's body in each frame of each video, wherein two identical adaptive graph convolutional neural networks are used for shunting, and each shunt is an independent spatiotemporal adaptive graph convolutional neural network, which is used to simulate the graphic structure of the key points of the human body.

2. The method according to claim 1, characterized in that: Human posture recognition is performed on a single frame image in the gait video to obtain the coordinates of the key points of the human skeleton in each frame image of the gait video. Partial data is extracted from the corresponding coordinates of the key points of the human skeleton, and features are extracted from the partial data. The step length is defined based on the left and right feet in the body25 model, which is expressed as formula (1): (1) in, is the pixel distance of the left leg, is the pixel distance of the right leg.

3. The method according to claim 1, characterized in that Also performed: Extracting similarity features of time series data, including: The DTW algorithm is used to adjust the time alignment of the sampling points of the two time series to find the best match between the two, that is, to calculate the minimum cumulative distance between the two time series. After DTW matching, the DTW features are extracted to represent the similarity between the two time series.

4. The method according to claim 3, characterized in that: The DTW features include: DTW distance: the minimum cumulative distance between two time series, used to indicate the similarity between the time series; DTW path: a path that indicates how to align two time series for visualization and further analysis; Matching point: a sampling point on the DTW path, used to represent the matching relationship between two time series; DTW feature vector: a set of statistical values ​​used to describe the distribution of sampling points on the DTW path.

5. The method according to claim 4, characterized in that: The constraints of the DTW algorithm include: Boundedness: Two sequences C and Q of unequal lengths start at (1, 1) and end at (m, n), where m is the length of sequence C and n is the length of sequence Q; Monotonicity: The previous point P(k, l) and the next point P(i, j) in the DTW path.

6. The method according to claim 5, characterized in that Also execute: The DTW distance between the step length curve of each walking after the patient takes the medicine and the walking before taking the medicine is selected to represent a feature, that is, the DTW distance between the gait curve before taking the medicine and the first step length curve after taking the medicine becomes the first feature, and the DTW distance between the gait curve before taking the medicine and the second gait curve after taking the medicine becomes the second feature. And so on, the gait curve of patients with Parkinson's disease is compared with the step length curve of the normal control group to obtain the DTW distance as a reference feature.

7. The method according to claim 1, characterized in that: When simulating the graphic structure, the human body structure is simulated by defining A as an adjacency matrix, defining A1 as the key point connection of the human body itself, and adding virtual connections related to gait movement to A1 to represent the implicit correlation of each key point in the movement, defining A2 and A3 as learnable parameters in the adaptive module, and inputting the gait features extracted based on the coordinates of the key points of the human skeleton and the output based on the spatiotemporal adaptive graph convolutional neural network into the fully connected layer to obtain the classification model; wherein, A=A1+A2+A3.