Auxiliary diagnosis method and system for Parkinson's disease based on static and dynamic features of facial expression

Abstract

The present invention discloses a method and system for assisting diagnosis of Parkinson's disease based on static and dynamic features of facial expressions, and relates to the technical field of image processing. The method comprises the following steps: obtaining video data of a patient to be tested completing multiple facial expressions; extracting multiple facial expression images with the best state of each facial expression after preprocessing; synthesizing a happy facial expression image using a generative network based on a neutral facial expression image, and performing similarity discrimination in combination with the extracted happy facial expression image; calculating the distance between multiple facial key points in multiple facial expression images to obtain multiple key features; splicing features to form static features; calculating the degree of coordinate change of multiple facial key points of eyelids and mouth based on multiple facial expression images of multiple facial expressions to obtain dynamic features; and outputting the classification prediction results of Parkinson's disease based on the spliced ​​features. The present invention comprehensively captures the degree of expression loss and facial stiffness, and effectively improves the accuracy of Parkinson's disease diagnosis.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a Parkinson's disease auxiliary diagnosis method and system based on static and dynamic features of facial expressions. Background Art

[0002] Parkinson's disease (PD) is the second most common neurodegenerative disease, primarily affecting the motor control area of ​​the central nervous system. The disease brings great difficulties to patients' daily lives, as they experience a range of motor and non-motor symptoms. A prominent symptom of Parkinson's disease is facial stiffness, which can range from reduced facial expressions in mild cases to complete loss of facial expressions in severe cases, a symptom also known as "hypomimia".

[0003] Traditional Parkinson's disease diagnosis usually requires the use of assessment scales such as the Non-Motor Symptom Quest (NMSQuest), the Non-Motor Symptom Scale (NMSS), and the Unified Parkinson's Disease Rating Scale (UPDRS) to comprehensively assess the patient's condition. Although these scales are convenient, their reliance on the doctor's subjective medical ability leads to inconsistency in clinical assessment. In addition, patients may not be in the best physiological and psychological state when they visit the doctor, which affects the accuracy and robustness of these scales.

[0004] Objective auxiliary diagnostic systems such as positron emission tomography (PET), electroencephalography (EEG), and magnetic resonance imaging (MRI) provide additional support for diagnosis. Although these methods are contact-based and help doctors develop treatment plans based on the actual situation of the patient, this will also increase the psychological pressure and economic burden on patients. Non-contact PD detection methods target modalities such as voice, gait signals, and facial expressions to assist in the diagnosis of Parkinson's disease based on their significant clinical symptoms. These methods not only reduce the burden on patients when seeking medical treatment, but also have higher accuracy and can provide a preliminary assessment of the patient's Parkinson's disease condition. Among them, the facial expression-based method has a particular advantage due to the ease of data acquisition, because this method only requires ordinary recording equipment to capture clear images or videos of facial expressions, and this is also valuable for the detection of early Parkinson's disease, because subtle facial expression loss often appears in the early stages.

[0005] At present, deep learning, especially generative adversarial networks (GANs), has been increasingly used in PD-assisted diagnosis. There are studies that use generative networks to synthesize normal facial expression images, train ResNet-based networks, and use the trained networks to predict PD. In addition, there are other studies that incorporate facial key points and action units (AUs) into PD diagnosis and extract facial key points from images to calculate expression factors; due to problems such as insufficient PD face datasets, some methods have also begun to use statistical analysis and automatic emotion detection. However, most existing models are based on analysis and processing of static facial images, rarely pay attention to the dynamic characteristics of facial expressions, or rarely use continuous video data / image data for PD-assisted diagnosis. The accuracy of PD diagnosis based solely on static facial image features is still low. Summary of the invention

[0006] To address the deficiencies of the above-mentioned prior art, the present invention provides a method and system for auxiliary diagnosis of Parkinson's disease based on static and dynamic features of facial expressions. The method extracts dynamic features and static features from a video continuously imitating facial expressions to comprehensively capture the degree of expression loss and facial stiffness / rigidity, and then performs auxiliary diagnosis of PD by fusing the two features, thereby effectively improving the accuracy of Parkinson's disease diagnosis.

[0007] In a first aspect, the present invention provides a method for auxiliary diagnosis of Parkinson's disease based on static and dynamic features of facial expressions.

[0008] A method for assisting diagnosis of Parkinson's disease based on static and dynamic features of facial expressions, comprising:

[0009] Obtain video data of the patient completing various facial expressions;

[0010] Preprocess the video data to extract multiple facial expression images with the best state for each facial expression;

[0011] Based on the neutral facial expression image, a generative network is used to synthesize a happy facial expression image of the patient under test in a healthy state, and similarity judgment is performed in combination with the extracted happy facial expression image to obtain a similarity feature; the distance between multiple facial key points in multiple facial expression images is calculated to obtain multiple key features; the similarity feature and multiple key features are spliced ​​to form a static feature;

[0012] Based on multiple facial expression images of various facial expressions, the degree of coordinate change of multiple facial key points of the eyelids and mouth is calculated to obtain dynamic features;

[0013] After balancing the dimensions of static features and dynamic features, feature splicing is performed, and the classification prediction results of Parkinson's disease are output based on the spliced ​​features.

[0014] In a second aspect, the present invention provides a Parkinson's disease auxiliary diagnosis system based on static and dynamic features of facial expressions.

[0015] A Parkinson's disease auxiliary diagnosis system based on static and dynamic features of facial expressions, comprising:

[0016] A data acquisition module, used to acquire video data of the patient to be tested completing various facial expressions;

[0017] A data preprocessing module is used to preprocess the video data and extract multiple facial expression images with the best state for each facial expression;

[0018] The static feature extraction module is used to synthesize the happy facial expression image of the patient to be tested in a healthy state based on the neutral facial expression image by using the generative network, and perform similarity judgment in combination with the extracted happy facial expression image to obtain similarity features; calculate the distance between multiple facial key points in multiple facial expression images to obtain multiple key features; splice the similarity features and multiple key features to form static features;

[0019] A dynamic feature extraction module, for calculating the degree of change of coordinates of multiple facial key points of the eyelids and the mouth based on multiple facial expression images of multiple facial expressions, and obtaining dynamic features;

[0020] The classification prediction module is used to balance the dimensions of static features and dynamic features, perform feature splicing, and output the classification prediction results of Parkinson's disease based on the spliced ​​features.

[0021] In a third aspect, the present invention further provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps of the method described in the first aspect are completed.

[0022] In a fourth aspect, the present invention further provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the method described in the first aspect.

[0023] One or more of the above technical solutions have the following beneficial effects:

[0024] 1. The present invention provides a method and system for auxiliary diagnosis of Parkinson's disease based on static and dynamic features of facial expressions. The method extracts dynamic features and static features from a video that continuously imitates facial expressions to comprehensively capture the degree of expression loss and facial stiffness / rigidity. The two features are then integrated to perform auxiliary diagnosis of PD, thereby effectively improving the accuracy of Parkinson's disease diagnosis.

[0025] 2. In the present invention, a generative network and facial key points are used to analyze static images of Parkinson's disease patients imitating facial expressions, and static facial features are constructed to quantify and characterize the degree of facial expression loss; the movement of key facial areas is extracted from continuous Parkinson's disease patient expression videos as dynamic features to quantify and characterize the degree of facial rigidity; the data difference problem between static features and dynamic features is solved through a static-dynamic feature balanced classification network, thereby improving the accuracy of PD classification prediction. Compared with the existing non-contact Parkinson's disease diagnosis method, the method proposed by the present invention achieves higher prediction accuracy (0.94) and recall rate (0.98) due to the comprehensive analysis of static and dynamic facial expression features. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0027] Figure 1 A schematic diagram of the structure of a classification prediction model in an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of marking the positions of different facial regions and facial key points within the regions in a facial expression image according to an embodiment of the present invention;

[0029] Figure 3 Schematic diagram of the Pearson correlation coefficient between static and dynamic characteristics and Parkinson's disease in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] It should be noted that the following detailed descriptions are exemplary only, are intended to describe specific embodiments, are intended to provide further explanation of the present invention, and are not intended to limit exemplary embodiments according to the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art to which the present invention belongs. In addition, it should also be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0031] Embodiment 1

[0032] Parkinson's disease is a common neurodegenerative disease with complex symptoms, which is difficult to diagnose directly and accurately. To this end, this embodiment proposes a Parkinson's disease auxiliary diagnosis method based on static and dynamic features of facial expressions, which detects the main symptom of Parkinson's disease, the lack of facial expression, by analyzing facial expressions. The method mainly includes two key parts: first, a complex generative network is used to generate a synthetic image of a patient showing a basic expression such as a "happy" expression under healthy conditions, and the synthetic image is compared with the patient's actual expression, and the completeness of the facial expression is extracted from the patient's other facial expression images to form a static feature for evaluating the lack of facial expression; secondly, the facial movements of the patient in the expression video are analyzed to form dynamic features for evaluating the degree of facial muscle stiffness. Finally, through a special static-dynamic feature balance classification network, the static features and dynamic features constructed above are integrated to complete PD auxiliary diagnosis and discrimination. This method can effectively help patients achieve more convenient and accurate PD auxiliary diagnosis.

[0033] The Parkinson's disease auxiliary diagnosis method based on static and dynamic features of facial expressions provided in this embodiment specifically includes the following steps:

[0034] Obtain video data of the patient completing various facial expressions;

[0035] Preprocess the video data to extract multiple facial expression images with the best state for each facial expression;

[0036] Based on the neutral facial expression image, a generative network is used to synthesize a happy facial expression image of the patient under test in a healthy state, and similarity judgment is performed in combination with the extracted happy facial expression image to obtain a similarity feature; the distance between multiple facial key points in multiple facial expression images is calculated to obtain multiple key features; the similarity feature and multiple key features are spliced ​​to form a static feature;

[0037] Based on multiple facial expression images of various facial expressions, the degree of coordinate change of multiple facial key points of the eyelids and mouth is calculated to obtain dynamic features;

[0038] After balancing the dimensions of static features and dynamic features, feature splicing is performed, and the classification prediction results of Parkinson's disease are output based on the spliced ​​features.

[0039] The following content introduces the Parkinson's disease auxiliary diagnosis method based on static and dynamic features of facial expressions proposed in this embodiment in more detail.

[0040] Step S1, obtaining video data of a patient to be tested performing a variety of facial expressions.

[0041] When acquiring video data, the patient to be tested sits in front of a screen that displays images of seven basic facial expressions, with three images for each expression, from the public dataset Oulu_CASIA. During data collection, the images are presented to the patient in turn, and the patient imitates the displayed images. This process is recorded using a camera, and auxiliary lighting is required to ensure that the patient's face is illuminated enough to ensure stable and clear video images. The seven facial expressions include: neutral, happy, sad, surprised, fearful, angry, and disgusted.

[0042] Step S2: pre-process the video data to extract multiple optimal facial expression images for each facial expression state.

[0043] The entire video data obtained through the above step S1 includes seven basic facial expressions. During the data processing, the entire video is divided into seven parts, namely seven independent video screens, each of which corresponds to a different facial expression.

[0044] For each facial expression, separate videos are used to capture the best facial state for each expression and obtain the best facial expression image for each facial expression state. This process faces many problems, which are solved through data preprocessing, including:

[0045] Question 1: The patient's face was shaking during the video recording.

[0046] Facial key points can be divided into relatively fixed points and dynamic flexible points. The former are used as baseline points, while the latter help measure the displacement of baseline points. For example, the points around the nose are relatively fixed points. In order to offset the slight facial tremor during recording, the facial key points (i.e. Figure 2 The facial key points are converted into relative coordinates relative to the fixed point, and then all video frame images are normalized. Here, the formula for normalizing facial key points is:

[0047]

[0048]

[0049] x i ′=|x i -x center |

[0050] y i ′=|y i -y center |

[0051] In the above formula, x i ,yi represents the horizontal and vertical coordinates of the i-th facial key point; x 28 ,y 28 、x 29 ,y 29 、x 30 ,y 30 、x 31 ,y 31 、x 34 ,y 34 The horizontal and vertical coordinates of the facial key points numbered 28, 29, 30, 31, and 34 are respectively represented, and these points are points around the nose; i ′、y i ′ represents the horizontal and vertical coordinates of the i-th facial key point after coordinate transformation.

[0052] Through the above coordinate transformation, the problem of patient facial shaking during video recording can be effectively solved.

[0053] Issue 2: During recording, the patient's movement toward or away from the camera causes their distance from the camera to change.

[0054] K images are uniformly selected from each patient's neutral expression independent video, and the average coordinates of each facial key point in these K neutral facial expression images are calculated to obtain the key point coordinates of the average neutral face (ANF). This method reduces the influence of inconsistent coordinates caused by changes in the distance between the patient and the camera, thereby solving problem 2.

[0055] Problem 3: Patients are unable to maintain their best facial appearance in expression videos.

[0056] In order to extract the frame that best represents the facial expression state, for the video frame images in each independent expression video, the distance between each facial key point in each image and the corresponding facial key point in the average neutral face ANF is calculated, and according to the sum of the distances of all points in each image, all the video frame images in each independent expression video are sorted in descending order, and the first L frames (the first 5 frames are selected in this embodiment) are selected as the optimal images of the facial expression state, thereby solving problem 3.

[0057] Through the above data preprocessing process, a total of 35 facial expression images were obtained for each patient. Finally, OpenFace was used for facial registration and the images were cropped to 128×128 pixels.

[0058] Step S3, feature extraction. In this embodiment, the static features and dynamic features of the facial expression video of the Parkinson's disease patient are extracted and analyzed, so as to complete the diagnosis of Parkinson's disease. The entire classification prediction process is actually implemented by the classification prediction model, such as Figure 1As shown in the figure, first, a complex generative network is used to synthesize images of healthy and happy expressions of Parkinson's patients, and then the images are compared with the happy expressions imitated by themselves. Then, the degree of completeness of the expression is calculated according to the imitation of the expression by the patient, and these results are regarded as static features; then, the movement of the facial parts in their expression videos that may reflect the symptoms of facial stiffness of the patient is analyzed, and these results are regarded as dynamic features. Among them, the static features reflect the completeness of the facial expression of the patient, while the dynamic features reflect the degree of facial stiffness.

[0059] Step S3.1, static feature extraction. In this embodiment, static features are extracted from two aspects: on the one hand, a generative network is used to synthesize a happy expression image of the patient in a healthy state, and the training strategy of the network is modified to improve the quality of the synthesized image. That is, the patient's neutral expression image and the action unit sequence image representing the happy expression selected from the data set are input into the generative network, and then the happy expression image in a healthy state is output; the similarity judgment result between the synthesized image and the patient's happy image is used as part of the static feature; on the other hand, based on the key features of different facial expressions, the patient's expression completion is extracted, and on this basis, the facial expression completion of the happy, sad, surprised and angry expressions in the patient's image is extracted as another part of the static feature.

[0060] Furthermore, by analyzing the facial expression images of patients, static features were extracted from two aspects. First, the smiles of Parkinson's patients often appear "unnatural" because they are not accompanied by cheek lifts. To solve this problem, the generative network was used to synthesize images of patients showing happy expressions in a healthy state, and these images were compared with their actual happy expression images. Secondly, the expression completion of Parkinson's patients when imitating happy, sad, surprised and angry expressions was quantified. These calculation results can be used as static features of Parkinson's patients, representing the lack of facial expressions.

[0061] Step S3.1.1: Based on the neutral facial expression image, a generative network is used to synthesize a happy facial expression image of the patient to be tested in a healthy state, and similarity judgment is performed on the extracted happy facial expression image to obtain a similarity feature.

[0062] First, the generative model and the discriminative model are trained through an adversarial process to create a high-quality image generator. After a lot of testing, this embodiment chooses GANimation as the generative model. GANimation is based on action unit (AUs) annotations and can capture facial movements that define human expressions in a continuous manifold. An important ability of this model is that it can focus on synthesizing key parts of the face while other parts of the image remain unchanged.

[0063] Secondly, training datasets for training GANimation networks are constructed, including the CelebA dataset and the seeprettyface dataset. In order to enhance the training set with diverse facial images and improve the robustness of the network under various lighting conditions, the diverse, large-capacity, and richly annotated CelebA dataset is added. CelebFaces AttributesDataset (CelebA) is a large facial attribute dataset containing more than 200,000 celebrity images, each of which is annotated with 40 attributes, and these images show various poses and backgrounds. On this basis, the seeprettyface dataset is also added to improve the performance of the network on different faces. This dataset consists of high-resolution images generated by the StyleGAN network to ensure that there are no repeated images in the dataset.

[0064] The robustness of the generative network is enhanced by adding a category face dataset from the seeprettyface dataset, and category a faces are taken as the target of the PD-assisted diagnosis method, and Openface is used to extract AUs of face images.

[0065] Finally, the training dataset is used to train the generative network model. For GANimation, the goal is to learn a mapping (M) that can transform the reference image (I yr ) is converted to target AU(y g ) is the output image (I yg ). In order to synthesize more diverse results, the original network uses random AU as (y g ), however, this embodiment modifies the training strategy, focusing on generating happy expression images from neutral images. To this end, a "happy AUs database" is created from 8,000 happy expression images and their extracted AUs. During training, there is a 60% probability of using the previous random AU generation method and a 40% chance of selecting AUs from this "happy AUs database". To prevent overfitting, random noise is introduced when selecting AUs from the "happy AUs database", thereby enhancing the diversity of the network.

[0066] The generative network trained as above is used to synthesize a happy facial expression image of the patient under the healthy state, and then the synthesized happy facial expression image is judged similarly with the actually extracted happy facial expression image to obtain a similarity feature.

[0067] Specifically, after the images are synthesized, image similarity evaluation is performed. The main methods for extracting facial image features include image descriptors, deep learning-based methods, and facial expression dynamic analysis.

[0068] Existing studies have shown that humans mainly express happiness through changes in their mouths, with the corners of the mouth raised and lips slightly open as their characteristics. Therefore, we selected M synthetic happy images and N images of patients showing happy expressions, extracted the coordinates of the facial key points of the mouth in these images, and compared the coordinates of each image. The comparison results provide a quantitative measure of the lack of facial expression in patients.

[0069] The Euclidean distance between two points is defined as:

[0070]

[0071] Among them, x and y represent the coordinates of point p.

[0072] Then, the Euclidean distance of the same facial key point between each synthetic image and each extracted image is calculated to obtain the similarity judgment result, which is defined as:

[0073]

[0074] where i and j are the coordinates representing the synthesized and extracted happy facial expression images, respectively. i,mouth ) and (p j,mouth ) represents the key points of the mouth, that is, from key point 49 to key point 68.

[0075] The similarity determination result obtained by the above calculation is the similarity feature.

[0076] Step S3.1.2, calculate the distances between multiple facial key points in multiple facial expression images to obtain multiple key features.

[0077] Parkinson's disease can cause cognitive impairment and affect the ability to express, leading to cortical or peripheral nerve trauma, limiting the generation of facial expressions and emotion recognition. In order to calculate the degree of completeness of facial expressions of Parkinson's patients, the positional relationship of key facial areas in the image when the patient imitates different expressions is analyzed. After multiple experimental analyses, in this embodiment, four representative expressions are selected: happiness, sadness, surprise and anger, to evaluate the degree of completeness of expressions, including:

[0078] For the happy expression characterized by the upturned corners of the mouth and the slightly open mouth, it includes: for the upturned corners of the mouth, calculate the points near the corners of the mouth (the points are preset points, i.e. Figure 2 The change in distance between the point framed by the corner of the mouth shown in the figure and the relatively fixed point; for a slightly open mouth, the change in distance between the upper and lower lips is calculated. It should be noted that the distance change mentioned in this section is relative to the neutral expression image, such as: the distance change here refers to the change in the distance between the upper and lower lips in the happy expression image relative to the distance between the upper and lower lips in the neutral expression image.

[0079] For a sad expression characterized by drooping eyebrows and upturned mouth corners, it includes: calculating the distance change between the eyebrows and a relatively fixed point, and the distance change between a point near the mouth corner and the relatively fixed point, so as to describe the characteristics of the sad expression.

[0080] For the surprised expression characterized by the increase of the distance between the eyebrows and the eyes, the widening of the eyes and the opening of the mouth, including: calculating the change of the distance between the eyebrows and the lower part of the eyes, the change of the distance between the upper and lower eyelids, and the change of the distance between the upper and lower lips to describe the characteristics of the surprised expression.

[0081] For angry expressions characterized by frowning eyebrows (manifested as drooping eyebrows and shortened distance between eyebrows and eyes), the method includes: for drooping eyebrows, calculating the change in distance between eyebrows and a relatively fixed point; for shortened distance between eyebrows and eyes, calculating the change in distance between eyebrows and eye centers.

[0082] The locations of different facial regions and key points within the regions are as follows: Figure 2 As shown in the figure, the left column is the expression images in the public Oulu-CASIA dataset (the images in this dataset are public images and do not involve privacy issues), and the right column is the distribution of their corresponding facial key points; Table 1 below shows the key facial areas used to calculate the degree of facial expression completion of Parkinson's disease patients. The distance between the key points in the key facial areas is used to quantify the degree of facial expression loss of the patient, so as to accurately determine the spatial relationship between them.

[0083] Table 1 Facial features and corresponding areas

[0084] Facial features Region A Region B Corners of mouth rise Corners of mouth Relative fixed point Mouth slightly open Upper lip Lower lip Lowered eyebrows Eyebrow Relative fixed point Mouth corners down Corners of mouth Relative fixed point Increased eyebrow distance Eyebrow Eye Center Eyes wide open Upper eyelid Lower eyelid Open your mouth Upper lip Lower lip Lowered eyebrows Eyebrow Relative fixed point Reduced eyebrow distance Eyebrow Eye Center

[0085] However, considering that the location of facial regions varies from person to person, analyzing only the patient's expression images may affect the robustness of the algorithm. Therefore, when calculating the facial position relationship, the same calculation is performed on the patient's imitation expression images and neutral expression images at the same time, and the difference in the calculation results is regarded as a static feature to represent the degree of facial expression loss in Parkinson's patients and eliminate the individual differences in facial features.

[0086] Step S3.1.3: Concatenate similarity features and multiple key features to form static features. In the above manner, each patient finally generates a 9-dimensional vector as a static feature vector.

[0087] Step S3.2, dynamic feature extraction. The face of a Parkinson's patient is often described as a mask face, with little or no expression, and in addition to the absence of facial expression, the patient's facial expression also shows obvious dullness, especially the reduction in the number of spontaneous blinks. Neurophysiological studies on Parkinson's disease have shown that the amplitude of the patient's voluntary facial movement is small and the speed is slow. Therefore, to evaluate the degree of dullness of the facial expressions of Parkinson's patients, it is necessary to extract dynamic facial features, focusing on the movement of the eyes and mouth. In this embodiment, in the video of the patient making expressions, based on multiple facial expression images of multiple facial expressions, the variance is used to calculate the degree of change in the coordinates of multiple facial key points of the eyelids and mouth, which is used as a dynamic feature.

[0088] Step S3.2.1, extract the coordinates of facial key points in multiple expression images of multiple facial expressions; based on the extracted key point coordinates, calculate the positional relationship between the upper and lower eyelids, and use variance calculation to determine the degree of change in the coordinates of facial key points in the eye area. Specifically, although Parkinson's disease affects the patient's eye movement, the correlation between blinking frequency and other Parkinson's disease symptoms is not significant. Therefore, blinking frequency should not be used to quantify the degree of facial stiffness. In fact, each patient completes seven basic facial expressions, and each expression requires a change in the eye area. Through the experimental results, it was found that there was a difference in the magnitude of change in eyeball size between Parkinson's patients and healthy controls. Therefore, for each patient and each expression, the coordinates of facial key points were extracted from 35 facial images of each patient (5 images for each expression), and the positional relationship between the upper and lower eyelids was calculated; then, variance analysis was used to determine the degree of change in the eye area.

[0089] Among them, for n samples, its variance is defined as:

[0090]

[0091] If it represents the variance of n points, it is defined as:

[0092]

[0093] Among them, x i and i Represents the coordinates of the i-th point among n points.

[0094] Then, the distance E between the upper and lower eyelids of the i-th facial image is i Defined as:

[0095]

[0096] Among them, p i,n Represents the key points of the eyelid area in the i-th facial image.

[0097] Changes in the eye region are defined as:

[0098] D 11 =Var(E)

[0099] Where E represents the distance between the upper and lower eyelids in each image of the patient's facial expression.

[0100] Step S3.2.2, based on the extracted key point coordinates, calculate the movement of each facial key point in the mouth area, and use variance calculation to determine the degree of change of the facial key point coordinates in the mouth area. Specifically, the distribution of emotional information on the face is not uniform. Generally speaking, when humans discern facial expressions, the mouth is used more than the eyes. When making various expressions, the patient's mouth needs to make corresponding changes. However, due to the facial stiffness of Parkinson's patients, the amplitude of mouth movement will decrease during continuous facial changes. Therefore, this embodiment extracts the key point coordinates of the mouth area and calculates the movement of each point. By analyzing the variance, the amplitude of mouth movement shown by the patient in a series of continuous facial expression changes is obtained, and the calculation formula is:

[0101]

[0102] Among them, p i represents the i-th key point in each image of the patient's facial expression.

[0103] Step S3.2.3: Take each calculation result as a feature and obtain multiple dynamic features. That is, in order to estimate the degree of stiffness of the patient's face, two results are obtained: the change of the eyes and the movement of the mouth area. These two calculation results are two dynamic features.

[0104] Step S4: After balancing the dimensions of static features and dynamic features, feature splicing is performed, and a classification prediction result of Parkinson's disease is output based on the spliced ​​features.

[0105] In fact, the weights of static and dynamic features should be equal for PD-assisted diagnosis. However, after feature extraction, the actual number of data points of static and dynamic features is not equal. Therefore, in order to keep the weights of static and dynamic features equal, these features need to be input into a feature balancing network, which converts the two sets of features into six dimensions respectively to balance the weights between static and dynamic features. This process balances the weights of static and dynamic features in the PD diagnosis model without changing the number of data points. Then, the balanced features are concatenated and PD is classified to complete the PD-assisted diagnosis task.

[0106] Furthermore, each patient’s facial expression video will generate a 12-dimensional vector, where D1 to D 10is a static feature, D 11 To D 12 It is a dynamic feature, and the meaning of each data point is shown in Table 2. Afterwards, the feature data is input into the designed static-dynamic feature balanced classification network for PD classification.

[0107] Table 2 Meaning of static and dynamic characteristics data

[0108] data meaning <![CDATA[D1]]> Similarity of synthetic images <![CDATA[D2]]> The degree to which the corners of the mouth rise when expressing happiness <![CDATA[D3]]> How open the mouth is in a happy expression <![CDATA[D4]]> The degree to which the eyebrows are wrinkled in a sad expression <![CDATA[D5]]> The degree to which the corners of the mouth droop in a sad expression <![CDATA[D6]]> The distance between eyebrows and eyes in surprised expressions <![CDATA[D7]]> The degree of mouth opening in the surprised expression <![CDATA[D s ]]> The degree of eye dilation in a surprised expression <![CDATA[D9]]> The degree to which the eyebrows are furrowed in anger expressions <![CDATA[D 10 ]]> Distance between eyebrows and eyes in angry expressions <![CDATA[D 11 ]]> The degree of change in mouth movement <![CDATA[D 12 ]]> Changes in eyelid distance

[0109] As another implementation, this embodiment uses technologies such as generative networks and facial key points to carefully study the facial expression data of Parkinson's patients. The data of each patient is encapsulated in a one-dimensional vector (static dynamic feature splicing) consisting of 12 data points. This embodiment also explores the correlation between these data points and Parkinson's disease by calculating the Pearson correlation coefficient. The Pearson correlation coefficient quantifies the strength and direction of the linear relationship between two variables. Its absolute value exceeds 0.2, indicating moderate correlation, and exceeds 0.4, indicating strong correlation. Figure 3 The results in show that most of the data points in this embodiment have a significant correlation with Parkinson's disease. Therefore, the static and dynamic features extracted in this embodiment have certain rationality.

[0110] In the present embodiment, in order to solve the problem of lacking a public Parkinson's disease patient facial expression data set, sufficient training data is collected, and a Parkinson's disease patient facial expression video data set (PDFEV) is constructed, and the data set is used to train the above-mentioned classification prediction model. Specifically, the data set contains videos of 62 Parkinson's disease patients, including 25 males, 37 females, and an average age of 62.8 years old (standard error ± 7.6 years old). In addition, 23 healthy control group subjects are also included, including 11 males, 12 females, and an average age of 58.6 years old (standard error ± 6.1 years old). Each subject will be recorded with seven basic facial expressions: neutral, happy, sad, surprised, fear, anger and disgust. When collecting data, the above-mentioned step S1 is used to collect, wherein, considering that the facial expression ability of Parkinson's disease patients is usually weak, verbal guidance on how to complete each expression is also provided for this purpose. In addition, to protect the privacy of patients, the eye area is deleted from the original image and the synthesized image of the facial expression of Parkinson's disease patients.

[0111] Furthermore, based on the above data set, the feature balance network is trained in combination with a neural network classifier to improve the prediction accuracy. In this embodiment, a five-fold cross validation is used to evaluate the performance of the network, and the average accuracy reaches 94.1%, which has a high classification accuracy and effectively improves the accuracy of Parkinson's disease diagnosis.

[0112] In order to further verify the superiority of the method proposed in this embodiment, further verification and explanation are carried out through the following experiments.

[0113] The improved GANimation generative network is trained on the CelebA dataset and the seeprettyface dataset. The training set includes 231,990 images and the test set includes 25,000 images. The training batchsize of the GANimation model is 16, with a total of 50 epochs, and each epoch contains 14,499 steps; the moment estimate exponential decay rate of the Adam optimizer is η1=0.5 and η2=0.999; the learning rate begins to decay after the initial 20 epochs; the model is implemented using Pytorch and is run on a 13th generation Intel(R) Core(TM) i9-13900K CPU and NVIDIA TM ( All experiments were conducted on a server with a GeForce RTX4090 GPU. The improved GANimation network successfully generated images of happy expressions of healthy people using a tailored training strategy, and after adding the A-category face dataset, the network became more stable in generating images with category A facial features.

[0114] 12 data were collected for each patient to be tested, with the aim of classifying Parkinson's disease. In addition to the static-dynamic feature balanced classification network, a variety of classification algorithms were used, including Bayesian classifier (BC), random forest (RF), decision tree (DT), support vector machine (SVM) and deep neural network (DNN). In order to evaluate the robustness of the model, five-fold cross validation was used, and the results are shown in Table 3 below. The static-dynamic feature balanced classification network (SDFEC) of this embodiment showed the superiority of balanced learning of static and dynamic features, including the highest accuracy (0.94) and recall (0.98), making it particularly suitable for this task and producing better classification accuracy. In addition, SDFEC also achieved the highest Cohen's Kappa coefficient (0.83), which shows that the method of this embodiment achieves better consistency across the entire data set. In addition, the results of classification using only static or dynamic features are also listed in Table 3, indicating that combining static and dynamic features in Parkinson's disease classification is more accurate than using a single feature.

[0115] Table 3 Performance comparison experimental results of using different models to classify different features

[0116]

[0117] Embodiment 2

[0118] This embodiment provides a Parkinson's disease auxiliary diagnosis system based on static and dynamic features of facial expressions, including:

[0119] A data acquisition module, used to acquire video data of the patient to be tested completing various facial expressions;

[0120] A data preprocessing module is used to preprocess the video data and extract multiple facial expression images with the best state for each facial expression;

[0121] The static feature extraction module is used to synthesize the happy facial expression image of the patient to be tested in a healthy state based on the neutral facial expression image by using the generative network, and perform similarity judgment in combination with the extracted happy facial expression image to obtain similarity features; calculate the distance between multiple facial key points in multiple facial expression images to obtain multiple key features; splice the similarity features and multiple key features to form static features;

[0122] A dynamic feature extraction module, for calculating the degree of change of coordinates of multiple facial key points of the eyelids and the mouth based on multiple facial expression images of multiple facial expressions, and obtaining dynamic features;

[0123] The classification prediction module is used to balance the dimensions of static features and dynamic features, perform feature splicing, and output the classification prediction results of Parkinson's disease based on the spliced ​​features.

[0124] Embodiment 3

[0125] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the Parkinson's disease auxiliary diagnosis method based on static and dynamic features of facial expressions as described above are completed.

[0126] Embodiment 4

[0127] This embodiment also provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps in the Parkinson's disease auxiliary diagnosis method based on static and dynamic features of facial expressions as described above are completed.

[0128] The steps involved in the above embodiments 2 to 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0129] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0130] The above description is only a preferred embodiment of the present invention. Although the specific implementation mode of the present invention is described in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present invention, various modifications or deformations that can be made by technical personnel in this field without creative work are still within the scope of protection of the present invention.

Claims

1. A method for assisting diagnosis of Parkinson's disease based on static and dynamic features of facial expressions, characterized in that: include: Obtain video data of the patient completing various facial expressions; Preprocess the video data to extract multiple facial expression images with the best state for each facial expression; Based on the neutral facial expression images, the generative network is used to synthesize the happy facial expression images of the patients under the healthy state, and the similarity is judged by combining the extracted happy facial expression images to obtain the similarity features. Calculate the distances between multiple facial key points in multiple facial expression images to obtain multiple key features; Concatenate similarity features and multiple key features to form static features; Based on multiple facial expression images of various facial expressions, the degree of coordinate change of multiple facial key points of the eyelids and mouth is calculated to obtain dynamic features; Through the static-dynamic feature balanced classification network, the dimensions of static features and dynamic features are balanced, and feature splicing is performed, and the classification prediction results of Parkinson's disease are output based on the spliced ​​features; The pre-processing comprises: Split the entire video into multiple independent videos, and each independent video corresponds to a different facial expression; For each continuous video frame image of each independent video, the facial key points in the image are marked, the coordinates of each facial key point are converted into relative coordinates relative to a relative fixed point, and all video frame images are normalized; Uniformly select K images from the neutral expression independent video, calculate the average value of the key point coordinates of each face in the K images, and obtain the key point coordinates of the average neutral face; For each video frame image in each independent expression video, calculate the distance between each facial key point in each image and the corresponding facial key point in the average neutral face. According to the sum of the distances of all points in each image, sort all the video frame images of each independent expression video in descending order, and select the first L images as the images with the best facial expression state; The multiple facial expressions include: neutral, happy, sad, surprised, fearful, angry, and disgusted; The facial key points include relatively fixed points and dynamic flexible points, and the relatively fixed points include points around the nose; The similarity determination includes: Obtaining M synthesized happy facial expression images and N extracted happy facial expression images; Extracting the coordinates of facial key points of the mouth in the acquired image; Calculate the Euclidean distance of the same facial key point between each synthetic image and each extracted image to obtain a similarity judgment result, which is the similarity feature; Calculate the distance between facial key points in multiple facial expression images to obtain multiple key features, including: For happy facial expression images, calculate the distance change between the point near the mouth corner and the relatively fixed point, and the distance change between the upper and lower lips; For sad facial expression images, calculate the distance change between the eyebrows and the relatively fixed point, and the distance change between the point near the mouth corner and the relatively fixed point; For surprised facial expression images, calculate the distance change between the eyebrows and the lower part of the eyes, the distance change between the upper and lower eyelids, and the distance change between the upper and lower lips; For angry facial expression images, calculate the distance change between eyebrows and relative fixed points, and the distance change between eyebrows and eye centers; Taking the difference between each calculation result of the synthesized image and the extracted image as a key feature, obtaining a plurality of key features; The distance change is relative to the neutral expression image; Based on multiple facial expressions, the coordinate changes of the key points of the eyelids and mouth are calculated to obtain dynamic features, including: Extracting facial key point coordinates from multiple expression images of multiple facial expressions; Based on the extracted key point coordinates, the positional relationship between the upper and lower eyelids is calculated, and the degree of change of the coordinates of the facial key points in the eye area is determined by variance calculation; Based on the extracted key point coordinates, the movement of each facial key point in the mouth area is calculated, and the degree of change of the facial key point coordinates in the mouth area is determined by variance calculation; Each calculation result is used as a feature to obtain multiple dynamic features.

2. A Parkinson's disease auxiliary diagnosis system based on static and dynamic features of facial expressions, characterized in that: include: A data acquisition module, used to acquire video data of the patient to be tested completing various facial expressions; A data preprocessing module is used to preprocess the video data and extract multiple facial expression images with the best state for each facial expression; The static feature extraction module is used to synthesize the happy facial expression image of the patient under test in a healthy state based on the neutral facial expression image by using the generative network, and perform similarity judgment in combination with the extracted happy facial expression image to obtain the similarity feature; Calculate the distances between multiple facial key points in multiple facial expression images to obtain multiple key features; concatenate similarity features and multiple key features to form static features; A dynamic feature extraction module, for calculating the degree of change of coordinates of multiple facial key points of the eyelids and the mouth based on multiple facial expression images of multiple facial expressions, and obtaining dynamic features; A classification prediction module is used to balance the static and dynamic features through a classification network, perform feature concatenation after balancing the dimensions of static features and dynamic features, and output a classification prediction result of Parkinson's disease based on the concatenated features; The pre-processing comprises: Split the entire video into multiple independent videos, and each independent video corresponds to a different facial expression; For each continuous video frame image of each independent video, the facial key points in the image are marked, the coordinates of each facial key point are converted into relative coordinates relative to a relative fixed point, and all video frame images are normalized; Uniformly select K images from the neutral expression independent video, calculate the average value of the key point coordinates of each face in the K images, and obtain the key point coordinates of the average neutral face; For each video frame image in each independent expression video, calculate the distance between each facial key point in each image and the corresponding facial key point in the average neutral face. According to the sum of the distances of all points in each image, sort all the video frame images of each independent expression video in descending order, and select the first L images as the images with the best facial expression state; The multiple facial expressions include: neutral, happy, sad, surprised, fearful, angry, and disgusted; The facial key points include relatively fixed points and dynamic flexible points, and the relatively fixed points include points around the nose; The similarity determination includes: Obtaining M synthesized happy facial expression images and N extracted happy facial expression images; Extracting the coordinates of facial key points of the mouth in the acquired image; Calculate the Euclidean distance of the same facial key point between each synthetic image and each extracted image to obtain a similarity judgment result, which is the similarity feature; Calculate the distance between facial key points in multiple facial expression images to obtain multiple key features, including: For happy facial expression images, calculate the distance change between the point near the mouth corner and the relatively fixed point, and the distance change between the upper and lower lips; For sad facial expression images, calculate the distance change between the eyebrows and the relatively fixed point, and the distance change between the point near the mouth corner and the relatively fixed point; For surprised facial expression images, calculate the distance change between the eyebrows and the lower part of the eyes, the distance change between the upper and lower eyelids, and the distance change between the upper and lower lips; For angry facial expression images, calculate the distance change between eyebrows and relative fixed points, and the distance change between eyebrows and eye centers; Taking the difference between each calculation result of the synthesized image and the extracted image as a key feature, obtaining a plurality of key features; The distance change is relative to the neutral expression image; Based on multiple facial expressions, the coordinate changes of the key points of the eyelids and mouth are calculated to obtain dynamic features, including: Extracting facial key point coordinates from multiple expression images of multiple facial expressions; Based on the extracted key point coordinates, the positional relationship between the upper and lower eyelids is calculated, and the degree of change of the coordinates of the facial key points in the eye area is determined by variance calculation; Based on the extracted key point coordinates, the movement of each facial key point in the mouth area is calculated, and the degree of change of the facial key point coordinates in the mouth area is determined by variance calculation; Each calculation result is used as a feature to obtain multiple dynamic features.

3. An electronic device, characterized in that: The invention comprises a memory and a processor and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the Parkinson's disease auxiliary diagnosis method based on static and dynamic features of facial expressions as claimed in claim 1 are completed.

4. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of the Parkinson's disease auxiliary diagnosis method based on static and dynamic features of facial expressions as claimed in claim 1.

Images