Intelligent Parkinson's disease assessment method fusing facial expression and behavior gait data

Abstract

The invention relates to the technical field of data processing, in particular to an intelligent Parkinson's disease assessment method fusing facial expressions and behavior gait data, which comprises the following three steps of: 1, performing data cleaning on a behavior gait video of a Parkinson's disease patient, segmenting the patient from the video by using a YOLOv8 model, extracting skeleton key points by using HRNet, and extracting the skeleton key points; acquiring features by using STGCN + +; 2, using StyleGAN to generate a pre-illness image retaining patient identity information, training a plurality of deep learning models to perform facial expression classification on collected facial expression images of six basic emotions of each person, and using a pre-training model with the best effect to extract features of facial expressions of the patient; and 3, carrying out multi-modal feature fusion on the extracted gait features and facial expression features for Parkinson's disease prediction. According to the method, the diagnosis accuracy is remarkably improved, and the problems of missed diagnosis and misdiagnosis are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a Parkinson's disease intelligent assessment method integrating facial expression and behavior gait data. Background Art

[0002] Parkinson's disease (PD) is a chronic progressive disease. It has become a neurodegenerative disease second only to Alzheimer's disease in prevalence. It is known as the "never-ending tremor". The average age of onset of Parkinson's disease in the population is around 60 years old, and most of the patients are middle-aged and elderly. Although the prevalence of Parkinson's disease is not high, it has the characteristics of rapid progression and high disability. Therefore, early screening is required for early intervention treatment and management. Although there is no effective solution to prevent or cure Parkinson's disease, related drugs such as levodopa can effectively improve the symptoms of the disease and improve the quality of life of patients.

[0003] One way to diagnose Parkinson's disease is to use the Unified Parkinson's Disease Rating Scale (MDS-UPDRS) of the Movement Disorders Society (MDS). However, this assessment tool cannot capture subtle changes in motor function. In addition, the assessment process of MDS-UPDRS relies on the patient's subjective self-report and the doctor's experience and judgment, and has defects in objectivity and quantification, which makes the screening misdiagnosis rate of Parkinson's disease high. Another commonly used method for diagnosing Parkinson's disease is in vivo diagnosis, which mainly relies on professional medical imaging technology and diagnoses and differentiates patients based on the imaging characteristics of Parkinson's disease. This method requires patients to go to professional medical institutions for diagnosis, which is not convenient for elderly people with limited mobility. Secondly, the diagnostic cost of professional equipment is high, which imposes an economic burden on the family. Compared with in vivo diagnosis, in vitro diagnostic methods are more convenient and less costly. Among the commonly used in vitro biomarkers, diagnosis through voice signals may be affected by language type or even dialect, gait signals require wearing professional gait signal sensors, and dynamic note collection solutions currently lack unified standards. The collection of facial expressions as biomarkers requires patients to make multiple expressions when photographing. This method is relatively convenient and economical. However, in the in vitro diagnosis of Parkinson's disease, not all patients have certain unimodal symptoms. For example, the voices of some early patients are not much different from those when the patients are not sick, which leads to diagnostic errors in models trained with unimodal data. Therefore, unimodal diagnostic methods have limitations in the early diagnosis of Parkinson's disease.

[0004] At present, deep learning is performing better and better in computer vision and other fields. Due to the strong fitting ability of deep learning, it can train a diagnostic model with excellent generalization through data-driven, and only needs to be fine-tuned to adapt to downstream tasks, and can be deployed and applied without spending a lot of time and financial resources. At the same time, we found that by using deep learning to extract facial expression features and gait video features, the shortcomings of speech signals and handwriting signals can be avoided, and the feature fusion technology can make up for the low generalization problem encountered in single-modal diagnosis. Therefore, the present invention proposes a research method for intelligent auxiliary diagnosis of Parkinson's disease by extracting facial expression images and gait video features of patients through deep learning models for multi-modal fusion. The detection has the advantages of convenient data collection, lightweight model, good generalization performance and easy deployment and promotion. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a Parkinson's disease intelligent assessment method that integrates facial expression and behavioral gait data to improve the reliability of diagnosis.

[0006] A Parkinson's disease intelligent assessment method integrating facial expression and behavioral gait data comprises the following steps:

[0007] S1. Behavioral gait feature extraction based on skeleton key points:

[0008] The YOLOv8 tracking and segmentation model is used to track the patients in the video, and the mask is used to segment the video containing only the behavior and gait of the PD patients; the YOLOv8 segmentation detection task includes segmentation-related losses and classification losses inherited from the target detection task, and rectangular box regression losses, namely CIoU loss and DFL;

[0009] S1.1, CIoU loss: the adversarial loss in the WGAN-GP model is used to replace the original adversarial loss in the StarGAN model.

[0010] Loss, the adversarial loss function is as follows:

[0011]

[0012] Where IoU is the intersection over union ratio between the predicted bounding box and the true bounding box; ρ(b, b g ) is the Euclidean distance between the center point of the predicted bounding box and the center point of the true bounding box; c is the bounding box b around the predicted bounding box and the true bounding box b g The diagonal length of the minimum rectangular box; α is an adjustment parameter used to balance the weight of IoU loss and center point distance loss; v is the aspect ratio loss, defined as:

[0013]

[0014] S1.2. DFL is defined as:

[0015]

[0016] in and is the probability distribution of two adjacent discrete points predicted by the model, y i and i+1 is the regression target value of the true bounding box, and y is the bounding box regression value predicted by the model;

[0017] S1.3, classification loss and segmentation loss: classification loss and segmentation loss are both based on BCEWithLogitsLoss. Segmentation loss is mainly the binary cross entropy loss calculated pixel by pixel between the predicted segmentation area and the ground truth segmentation area. BCEWithLogitsLoss is defined as:

[0018] l bn，c =-w bn，c [p c y bn，c ·log(σ(x bn，c ))+(1-y bn，c )·log(1-σ(x bn，c ))] (4)

[0019] Among them l bn，c is the bounding box classification loss for the b-th batch, n-th sample, c-th class (c>1 for multi-label binary classification, c=1 for single-label binary classification); p c is the true label of the cth class; y bn，c is the true label of the cth class of the nth sample in the bth batch, and its value is 0 or 1; x bn，c is the prediction score of the b-th batch, n-th sample, and c-th class output by the model; logσ(x bn，c ) is to use the sigmoid function to convert x bn，c The prediction score is converted into a probability value; w bn，c Is a weight factor used to control the loss weight of different categories or different samples;

[0020] Finally, the four losses calculated above are weighted summed to obtain the total loss l all , defined as:

[0021] l all =w1l CloU +w2l DFL +w3l class +w4l seg (5)

[0022] Among them l CloU is IoU loss, l DFL is the DFL loss, l class is the classification loss, l seg To split the loss;

[0023] After obtaining data containing only video clips of Parkinson's disease patients after tracking and segmentation extraction, a behavioral gait feature extraction method based on skeleton key points was adopted; the first part is to use HRNet to extract video clips into skeleton key point data. The loss function used by HRNet is MSE, which is defined as:

[0024]

[0025] Where n is the number of samples, y i is the true target value of the i-th sample, is the predicted value of the i-th sample; and OKS is used as the evaluation indicator, which is defined as:

[0026]

[0027] Where i represents the i-th key point, represents the Euclidean distance between the predicted coordinates and the true coordinates of the i-th key point, s represents the scale of the image, usually the height or width of the human body, represents the scale-adaptive scaling factor of the key point, which is used to balance the importance of different key points, v i Represents the visibility mark of the i-th key point;

[0028] The second part is to use the extracted skeleton key points to extract features through the excellent STGCN++ network;

[0029] S2. Synthesize facial expressions and extract features based on generative adversarial networks:

[0030] Use StyleGAN to synthesize virtual facial expression images of PD patients in their pre-disease state;

[0031] S2.1. StyleGAN generates latent vectors: A pre-trained StyleGAN generator is combined with an encoder network. During the training process, the parameters of the encoder network are fine-tuned through a carefully designed similarity loss function, which is a weighted combination of the VGG-16 perceptual loss and the per-pixel mean square error MSE loss. The loss function is defined as follows:

[0032]

[0033] where c represents the target latent vector, I0 represents the input image, G represents the pre-trained StyleGAN generator, N represents the total number of image pixels, and λ mse is a weight hyperparameter; the perceptual loss L percept A pre-trained VGG network is used to measure the perceptual difference between the original image I0 and the synthesized image G(c); this perceptual loss is defined as follows:

[0034]

[0035] Among them C j (·) represents the feature map output of the jth convolutional layer of the VGG-16 network, k represents the total number of convolutional layers, and N j represents the total number of pixels in the feature map of the jth convolutional layer, λ j is a weight hyperparameter;

[0036] S2.2, StyleGAN calculates the facial expression direction vector: obtains latent vectors that share consistent identity information but show different facial expressions A and B;

[0037] Then, the transition direction vector from expression A to expression B is calculated. Specifically, the potential vector sets of expressions A and B are first obtained, which are denoted as label A and label B , and assign them labels latent A =0 and latent B =1; then construct the mapping function in is considered as the normal vector from expression A to expression B, and the term (1-2label x ) is used to control the direction;

[0038] Next, a logistic regression model P = σ(f) is constructed to output the predicted class label probability, whose value is between [0,1], and the normal vector that can manipulate the transition from A to B is determined by optimizing the binary cross entropy (BCE) loss. this is the desired direction vector n AB ;

[0039] Similarly, by adopting the above calculation algorithm, the direction vectors between multiple facial expression states are obtained, so that different types of facial expressions can be synthesized while preserving the identity;

[0040] S2.3, multi-expression synthesis and deep feature extraction: According to the method described in step S2.2, multiple direction vectors capable of controlling different facial expressions can be successfully obtained; the process of facial expression generation can be formalized as follows:

[0041]

[0042] Among them G enc and G dec denote the encoder and decoder of G, respectively, I denotes a facial image with expression A, and I ′ represents the image of the same person with expression B, is the direction vector from expression A to B, and λ controls the degree of expression change;

[0043] By adopting the above facial expression synthesis method, facial expression images representing six basic emotions can be generated from a single neutral facial expression image of a Parkinson's disease patient, capturing the expression before the onset of the disease; these images will be used as a control group to construct an enhanced Parkinson's disease facial expression training dataset; then, a deep learning model is used on this extended training dataset to train a discriminant model; finally, the model is used as a feature extractor and six facial images with different expressions are used as input, and the trained expression classifier is used as a pre-training model to fine-tune the model parameters to obtain the image features of facial expressions;

[0044] S3, multimodal feature fusion:

[0045] Using the appropriate gait video data processing method and feature extraction model found in the first two steps, as well as the facial expression feature extraction model, gait features and facial expression features are obtained. In order to facilitate subsequent feature fusion, the dimensions of gait features and facial expression features are unified to m dimensions through two fully connected layers. The specific method is as follows:

[0046] S3.1, Feature-level fusion: In the feature extraction stage, we directly concatenate the m-dimensional gait features and the m-dimensional facial expression features into a 2*m-dimensional high-dimensional feature vector, and then output a two-dimensional feature vector through a fully connected layer;

[0047] S3.2, Decision-level fusion: We fuse gait features and facial expression features by weighted averaging and other methods, giving weights to the two modalities w1 and w2 respectively, and obtain an m-dimensional fusion feature vector; then output a two-dimensional feature vector through a fully connected layer;

[0048] S3.3, Hybrid-level fusion: We input the two modal features into the fully connected layer separately to obtain their respective scores; then we concatenate the two scores into an m+1-dimensional fusion feature, and finally input it into the same fully connected layer for model training, output two two-dimensional features, and add them together to obtain the final result;

[0049] Finally, the transfer learning method is used to fuse the features of the two modalities through a fully connected layer, and finally a vector consisting of two elements is output, representing the probabilities of PD and non-PD respectively.

[0050] Furthermore, in step S1.3, if the key point is visible, it is 1, otherwise it is 0, indicating the Kronecker delta function, such as v i >0 is 1, otherwise it is 0; the final OKS value is between 0 and 1, the closer it is to 1, the closer the predicted key point is to the real key point; the key point used here is the standard of COCO17.

[0051] Furthermore, in step S2.2, another feature of StyleGAN is the use of progressive layers. The lower the layer and resolution, the coarser the features it affects. In the latent space, each latent vector corresponds to a generated face image. By translating or superimposing these expression direction vectors on the basis of the original latent vector, new latent vectors with different expressions but the same identity can be obtained.

[0052] Furthermore, in step S2.2, when x = A, the term (1-2label A ) evaluates to 1, and the calculated direction is On the contrary, when x=B, the term in f (1-2label B ) evaluates to -1, and the calculated direction is Indicates the opposite direction from expression B to expression A.

[0053] Further, the six basic emotions include anger, disgust, fear, sadness, happiness, and surprise.

[0054] The beneficial effects of the present invention are:

[0055] The present invention proposes a new multimodal in vitro diagnostic scheme for PD based on facial expressions and behavioral gait. It is the first time that a method for in vitro diagnosis of PD using daily mobile phone video data is proposed. By introducing gait features, compared with the previous diagnostic method that only uses facial expressions, the accuracy of diagnosis is significantly improved, and missed diagnoses and misdiagnoses are reduced, confirming the excellent performance of multimodal in vitro diagnosis of PD.

[0056] In order to reduce the difficulty of patient diagnosis and meet the requirement that mobile phones can be used for gait video data collection without limiting the environment, the present invention proposes a series of effective data processing solutions. At the same time, in order to solve the problem of small amount of data in traditional facial expression-based diagnosis methods and the lack of facial expressions when patients are not sick, AICG technology (artificial synthesis based on StyleGAN) is used for data augmentation, which effectively improves the robustness of the model. At the same time, the present invention uses a lightweight model architecture to improve the practicality, ease of deployment and scalability of the proposed diagnostic strategy, so as to better cope with the challenge of increasing PD patients in the aging society in the future. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 It is an overview of the contents of the present invention;

[0059] Figure 2 This is the STGCN++ model structure diagram;

[0060] Figure 3 Schematic diagram of the StyleGAN model training process and different features of hierarchical latent vector control;

[0061] Figure 4 Schematic diagram for calculating direction vector;

[0062] Figure 5 Schematic diagram of a neural network trained for expression classification;

[0063] Figure 6 Schematic diagram of feature extraction for 6 types using expression classifier as pre-training model;

[0064] Figure 7 Schematic diagram of using multiple feature fusion methods to fuse features and classify PD / non-PD. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0066] A method for intelligent Parkinson's disease assessment that integrates facial expressions and behavioral gait data. The method is divided into the following three steps: First, the behavioral gait video of Parkinson's disease patients is cleaned, and the YOLOv8 model is used to segment the patient from the video. HRNet is then used to extract skeletal key points, and STGCN++ is used to obtain features. Second, StyleGAN is used to generate pre-disease images that retain the patient's identity information to make up for the missing control group data in our collected data. After that, multiple deep learning models will be trained to classify the facial expression images of each person's six basic emotions (anger, disgust, fear, happiness, sadness, and surprise), and the best pre-trained model will be used to extract the features of the patient's facial expression. Third, the model with good generalization performance and few model parameters in the first two steps is used as a feature extractor, and the extracted gait features and facial expression features are subjected to multimodal feature fusion for Parkinson's disease prediction, and then the classifier is trained for PD / non-PD classification. Specific research content is as follows: Figure 1 As shown, the specific implementation of each step is as follows:

[0067] S1. Behavioral gait feature extraction based on skeleton key points:

[0068] In this step, we first preprocess the collected data. Since the PD patient behavior and gait video data we use does not specify a special shooting environment when shooting, nor does it restrict strict shooting conditions, the video dataset contains many interference factors. Therefore, we first use the YOLOv8 tracking and segmentation model to track the patients in the video, and use the mask to segment the video containing only the behavior and gait of PD patients. The YOLOv8 segmentation detection task mainly consists of segmentation-related losses, classification losses inherited from the target detection task, and rectangular box regression losses (CIoU loss and DFL):

[0069] S1.1, CIoU loss: The adversarial loss in the WGAN-GP model is used to replace the original adversarial loss in the StarGAN model to obtain more stable model training and higher comprehensive image quality. The adversarial loss function is as follows:

[0070]

[0071] Where IoU is the intersection over union ratio between the predicted bounding box and the true bounding box; ρ(b, b g ) is the Euclidean distance between the center point of the predicted bounding box and the center point of the true bounding box; c is the bounding box b around the predicted bounding box and the true bounding box b gThe diagonal length of the minimum rectangular box; α is an adjustment parameter used to balance the weight of IoU loss and center point distance loss; v is the aspect ratio loss, defined as:

[0072]

[0073] S1.2, DFL (Distribution Focal Loss): DFL does not directly predict the bounding box coordinates, but predicts a probability distribution, that is, the probability of different coordinate values, using the probability distribution of two adjacent discrete points and Instead of a single point prediction value, the loss function is calculated based on the true bounding box coordinate y i and i+1 The difference between the predicted value and the predicted value is used to optimize the model using logarithmic loss. The distribution-based prediction method allows the model to learn a smoother bounding box regression distribution, thereby improving the prediction accuracy of difficult samples. DFL is defined as:

[0074]

[0075] in and is the probability distribution of two adjacent discrete points predicted by the model, y i and i+1 is the regression target value of the true bounding box, and y is the bounding box regression value predicted by the model

[0076] S1.3, classification loss and segmentation loss: classification loss and segmentation loss are both based on BCEWithLogitsLoss. Segmentation loss is mainly the binary cross entropy loss calculated pixel by pixel between the predicted segmentation area and the ground truth segmentation area. BCEWithLogitsLoss is defined as:

[0077] l bn，c =-w bn，c [p c y bn，c ·log(σ(x bn，c ))+(1-y bn，c )·log(1-σ(x bn，c ))] (4)

[0078] Among them l bn，c is the bounding box classification loss for the b-th batch, n-th sample, c-th class (c>1 for multi-label binary classification, c=1 for single-label binary classification); p c is the true label of the cth class. bn，c is the true label of the cth class of the nth sample in the bth batch, and its value is 0 or 1; xbn，c is the prediction score of the b-th batch, n-th sample, and c-th class output by the model; logσ(x bn，c ) is to use the sigmoid function to convert x bn，c The prediction score is converted into a probability value; w bn，c It is a weight factor used to control the loss weight of different categories or different samples.

[0079] Finally, the four losses calculated above are weighted summed to obtain the total loss l all , defined as:

[0080] l all =w1l cloU +w2l DFL +w3l class +w4l seg (5)

[0081] Among them l CloU is IoU loss, l DFL is the DFL loss, l class is the classification loss, l seg For segmentation loss.

[0082] After obtaining data that only contains video clips of Parkinson's disease patients' steps after tracking segmentation extraction, based on the widespread application of deep learning skeleton key point detection technology in recent years and its advantage of accurately extracting spatial coordinate information of human joints, we decided to adopt a behavioral gait feature extraction method based on skeleton key points. The first part is to use HRNet to extract video clips into skeleton key point data. The loss function used by HRNet is MSE (Mean Squared Error Loss), which is defined as:

[0083]

[0084] Where n is the number of samples, y i is the true target value of the i-th sample, is the predicted value of the i-th sample. And OKS (Object Keypoint Similarity) is used as the evaluation indicator, and OKS is defined as:

[0085]

[0086] Where i represents the i-th key point, represents the Euclidean distance between the predicted coordinates and the true coordinates of the ith key point, s represents the scale of the image, usually the height or width of a human body, represents the scale-adaptive scaling factor of the key point, which is used to balance the importance of different key points, v i Indicates the visibility flag of the i-th key point, which is 1 if the key point is visible, otherwise 0, indicating the Kronecker delta function, such as v i >0 is 1, otherwise it is 0. The final OKS value is between 0 and 1. The closer to 1, the closer the predicted key point is to the real key point. The key points used here are the COCO17 standard.

[0087] The second part is to use the extracted skeleton key points to extract features through the excellent STGCN++ network. The model structure of STGCN++ is as follows: Figure 2 As shown in Figure 2. STGCN++ is an enhanced version of the original STGCN model.

[0088] S2. Synthesize facial expressions and extract features based on generative adversarial networks:

[0089] Due to the lack of control samples from PD patients before onset, it is challenging to train an effective discriminative model for feature extraction from the facial expression data we collected. To address this issue, we used StyleGAN to synthesize virtual facial expression images of PD patients in their pre-disease state. We then explored the use of different deep learning models for feature extraction and classification training on the enhanced facial expression data.

[0090] StyleGAN is a powerful and flexible image generation model based on generative adversarial networks, which excels in creativity, realism, and controllability. The generation process of StyleGAN is subject to fine-grained control of an intermediate "style" latent space. Various attributes of the image, such as expression, posture, skin color, etc., can be changed by adjusting this latent space vector. Therefore, if any input face is mapped to a latent vector in the latent space of the StyleGAN generative model through a special encoder, and the latent vector can be used to generate the corresponding face through a pre-trained StyleGAN generator, then with the rich expressive power of StyleGAN, by modifying the different dimensions of this latent vector, various editing operations can be performed on the input image, thereby achieving the generation of faces with different expressions while ensuring that the identity information remains unchanged. The StyleGAN training process is as follows: Figure 3 As shown in the figure on the left, the process of extracting facial expression features from PD patients is described in detail below:

[0091] S2.1. StyleGAN generates latent vectors: A combination of a pre-trained StyleGAN generator and an encoder network is used. During the training process, we optimize the encoder parameters to ensure that the reconstructed image generated by the generator is as close to the original input image as possible. This approach enables us to obtain a StyleGAN latent vector with similar identity information to the original input image. It is worth noting that during the training process, the parameters of the encoder network are fine-tuned through a carefully designed similarity loss function, which is a weighted combination of the VGG-16 perceptual loss and the per-pixel mean square error (MSE) loss. The loss function is defined as follows:

[0092]

[0093] Where c represents the target latent vector, I0 represents the input image, G represents the pre-trained StyleGAN generator, N represents the total number of image pixels, and λ mse is a weight hyperparameter. Perceptual loss L percept Use the pre-trained VGG network to measure the perceptual difference between the original image I0 and the synthesized image G(c). This perceptual loss is defined as follows:

[0094]

[0095] Among them C j (.) represents the feature map output of the jth convolutional layer of the VGG-16 network, k represents the total number of convolutional layers, N j represents the total number of pixels in the feature map of the jth convolutional layer, λ j is a weight hyperparameter.

[0096] S2.2. StyleGAN calculates the direction vector of facial expression: Another feature of StyleGAN is the use of progressive layers. A potential benefit of progressive layers is that if used properly, they can control different visual features of the image, such as Figure 3 As shown in the figure on the right. The lower the layer and resolution, the coarser the features it affects. In the latent space, each latent vector corresponds to a generated face image, such as Figure 4 As shown, by translating or superimposing along these expression direction vectors on the basis of the original latent vector (such as P1), a new latent vector (such as P'1) with different expressions but the same identity can be obtained. According to the method of step S2.1, latent vectors that share consistent identity information but show different facial expressions A and B (for example, neutral and happy) can be obtained.

[0097] Then, the transition direction vector from expression A to expression B is calculated. Specifically, the potential vector sets of expressions A and B are first obtained, which are denoted as label Aand label B , and assign them labels latent A =0 and latent B = 1. Then we construct the mapping function in can be considered as the normal vector from expression A to expression B, and the term (1-2label x ) is used to control the direction. When x = A, the item in f (1-2label A ) evaluates to 1, and the calculated direction is On the contrary, when x=B, the term in f (1-2label B ) evaluates to -1, and the calculated direction is Indicates the opposite direction from expression B to expression A. Next, a logistic regression model P = σ(f) is constructed to output the predicted class label probability, whose value is between [0,1], and the normal vector that can manipulate the transition from A to B is determined by optimizing the binary cross entropy (BCE) loss. this can be approximated as the required direction vector n AB .

[0098] Similarly, by adopting the above calculation algorithm, the direction vectors between multiple facial expression states can be obtained, so that different types of facial expressions can be synthesized while preserving the identity.

[0099] S2.3, multi-expression synthesis and deep feature extraction: According to the method described in step S2.2, multiple direction vectors with the ability to control different facial expressions can be successfully obtained. For any facial image, first apply the method mentioned in step S2.1 to extract its corresponding latent vector in the StyleGAN latent space. Subsequently, by performing linear calculations between the latent vector and the specific direction vector that controls the facial expression, the generator G of StyleGAN is used to synthesize a new image with the desired facial expression while retaining the identity information. The process of facial expression generation can be formalized as follows:

[0100]

[0101] Among them G enc and G dec denote the encoder and decoder of G respectively, I denotes a facial image with expression A, and I′ denotes an image of the same person with expression B. is the direction vector from expression A to B, and λ controls the degree of expression change.

[0102] By adopting the above facial expression synthesis method, facial expression images representing six basic emotions (anger, disgust, fear, sadness, happiness and surprise) are generated from a single neutral facial expression image of a Parkinson's disease patient, capturing the expression before the onset of the disease. These images will be used as a control group to construct an enhanced Parkinson's disease facial expression training dataset. Subsequently, various deep learning models (such as ResNet and MobileNetV3) will be explored on this expanded training dataset to train the discriminant model. The goal is to achieve facial expression classification and extract corresponding facial features, such as Figure 5 Finally, we will seek a balance between model size and facial expression classification performance, select a deep learning model that is both high-performance and suitable for deployment on mobile devices, and on this basis, select the best and lightweight model as the feature extractor and use six facial images with different expressions as input, and use the trained expression classifier as the pre-training model to fine-tune the model parameters to obtain the image features of facial expressions. The structure is shown in Figure 6 shown.

[0103] S3, multimodal feature fusion:

[0104] In this step, we use the appropriate gait video data processing method and feature extraction model found in the first two steps, as well as the facial expression feature extraction model, to obtain gait features and facial expression features. We then explored a variety of multimodal feature fusion methods, including: feature-level fusion (early fusion), decision-level fusion (late fusion), and hybrid-level fusion. Before multimodal fusion, in order to facilitate subsequent feature fusion, we use two fully connected layers to unify the dimensions of gait features and facial expression features to m dimensions. The fusion method is as follows: Figure 7 As shown, the specific method is as follows:

[0105] S3.1. Feature-level fusion: In the feature extraction stage, we directly concatenate the m-dimensional gait features and the m-dimensional facial expression features into a 2*m-dimensional high-dimensional feature vector, and then output a two-dimensional feature vector through a fully connected layer.

[0106] S3.2, Decision-level fusion: We fuse gait features and facial expression features by weighted averaging, etc., giving weights to the two modalities w1 and w2 respectively, and obtain an m-dimensional fusion feature vector. Then, a fully connected layer is used to output a two-dimensional feature vector.

[0107] S3.3, Hybrid-level fusion: We input the two modal features into the fully connected layer separately to obtain their respective scores. Then we concatenate the two scores into an m+1-dimensional fusion feature, and finally input it into the same fully connected layer for model training, output two two-dimensional features, and add them together to get the final result.

[0108] Finally, we use the transfer learning method. Specifically, the behavior gait feature extractor and the facial expression feature extractor are fine-tuned based on pre-trained models (such as ResNet and MobileNetV3). The parameters are frozen and not updated. Only the multimodal fusion model is trained, that is, the features of the two modalities are fused through the fully connected layer, and finally a vector consisting of two elements is output, representing the probability of PD and non-PD respectively.

[0109] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A Parkinson's disease intelligent assessment method integrating facial expression and behavioral gait data, characterized in that: The following steps are involved: S1. Behavioral gait feature extraction based on skeleton key points: The YOLOv8 tracking and segmentation model is used to track the patients in the video, and the mask is used to segment the video containing only the behavior and gait of the PD patients; the YOLOv8 segmentation detection task includes segmentation-related losses and classification losses inherited from the target detection task, and rectangular box regression losses, namely CIoUloss and DFL; S1.1, CIoU loss: The adversarial loss in the WGAN-GP model is used to replace the original adversarial loss in the StarGAN model. The adversarial loss function is as follows: Where IoU is the intersection over union ratio between the predicted bounding box and the true bounding box; ρ(b, b g ) is the Euclidean distance between the center point of the predicted bounding box and the center point of the true bounding box; c is the bounding box b around the predicted bounding box and the true bounding box b g The diagonal length of the minimum rectangular box; α is an adjustment parameter used to balance the weight of IoU loss and center point distance loss; v is the aspect ratio loss, defined as: S1.

2. DFL is defined as: in and is the probability distribution of two adjacent discrete points predicted by the model, y i and i+1 is the regression target value of the true bounding box, and y is the bounding box regression value predicted by the model; S1.3, classification loss and segmentation loss: classification loss and segmentation loss are both based on BCEWithLogitsLoss. Segmentation loss is mainly the binary cross entropy loss calculated pixel by pixel between the predicted segmentation area and the ground truth segmentation area. BCEWithLogitsLoss is defined as: l bn，c =-w bn，c [p c y bn，c ·log(σ(x bn，c ))+(1-y bn，c )·log(1-σ(x bn，c ))] (4) Among them l bn，c is the bounding box classification loss for the b-th batch, n-th sample, c-th class (c>1 for multi-label binary classification, c=1 for single-label binary classification); p c is the true label of the cth class; y bn，c is the true label of the cth class of the nth sample in the bth batch, and its value is 0 or 1; x bn，c is the prediction score of the b-th batch, n-th sample, and c-th class output by the model; logσ(x bn，c ) is to use the sigmoid function to convert x bn，c The prediction score is converted into a probability value; w bn，c Is a weight factor used to control the loss weight of different categories or different samples; Finally, the four losses calculated above are weighted summed to obtain the total loss l all , defined as: the all =w1l CloU +w2l DFL +w3l class +w4l seg (5) Among them l CloU is IoU loss, l DFL is the DFL loss, l class is the classification loss, l seg To split the loss; After obtaining data containing only video clips of Parkinson's disease patients after tracking and segmentation extraction, a behavioral gait feature extraction method based on skeleton key points was adopted; the first part is to use HRNet to extract video clips into skeleton key point data. The loss function used by HRNet is MSE, which is defined as: Where n is the number of samples, y i is the true target value of the i-th sample, is the predicted value of the i-th sample; and OKS is used as the evaluation indicator, which is defined as: Where i represents the i-th key point, represents the Euclidean distance between the predicted coordinates and the true coordinates of the i-th key point, s represents the scale of the image, usually the height or width of the human body, represents the scale-adaptive scaling factor of the key point, which is used to balance the importance of different key points, v i Represents the visibility mark of the i-th key point; The second part is to use the extracted skeleton key points to extract features through the excellent STGCN++ network; S2. Synthesize facial expressions and extract features based on generative adversarial networks: Use StyleGAN to synthesize virtual facial expression images of PD patients in their pre-disease state; S2.

1. StyleGAN generates latent vectors: A pre-trained StyleGAN generator is combined with an encoder network. During the training process, the parameters of the encoder network are fine-tuned through a carefully designed similarity loss function, which is a weighted combination of the VGG-16 perceptual loss and the per-pixel mean square error MSE loss. The loss function is defined as follows: where c represents the target latent vector, I0 represents the input image, G represents the pre-trained StyleGAN generator, N represents the total number of image pixels, and λ mse is a weight hyperparameter; the perceptual loss L percept A pre-trained VGG network is used to measure the perceptual difference between the original image I0 and the synthesized image G(c); this perceptual loss is defined as follows: Among them C j (·) represents the feature map output of the jth convolutional layer of the VGG-16 network, k represents the total number of convolutional layers, and N j represents the total number of pixels in the feature map of the jth convolutional layer, λ j is a weight hyperparameter; S2.2, StyleGAN calculates the facial expression direction vector: obtains latent vectors that share consistent identity information but show different facial expressions A and B; Then, the transition direction vector from expression A to expression B is calculated. Specifically, the potential vector sets of expressions A and B are first obtained, which are denoted as label A and label B , and assign them labels latent A =0 and latent B =1; then construct the mapping function in is considered as the normal vector from expression A to expression B, and the term (1-2label x ) is used to control the direction; Next, a logistic regression model P = σ(f) is constructed to output the predicted class label probability, whose value is between [0,1], and the normal vector that can manipulate the transition from A to B is determined by optimizing the binary cross entropy (BCE) loss. this is the desired direction vector n AB ; Similarly, by adopting the above calculation algorithm, the direction vectors between multiple facial expression states are obtained, so that different types of facial expressions can be synthesized while preserving the identity; S2.3, multi-expression synthesis and deep feature extraction: According to the method described in step S2.2, multiple direction vectors capable of controlling different facial expressions can be successfully obtained; the process of facial expression generation can be formalized as follows: Among them G enc and G dec denote the encoder and decoder of G respectively, I denotes a facial image with expression A, and I′ denotes an image of the same person with expression B. is the direction vector from expression A to B, and λ controls the degree of expression change; By adopting the above facial expression synthesis method, it is possible to generate facial expression images representing six basic emotions from a single neutral facial expression image of a Parkinson's disease patient, capturing the expression before the onset of the disease; These images will be used as a control group to construct an enhanced Parkinson's disease facial expression training dataset. Subsequently, a deep learning model will be used on this extended training dataset to train a discriminant model. Finally, the model will be used as a feature extractor and six facial images with different expressions will be used as input. The trained expression classifier will be used as a pre-training model to fine-tune the model parameters and obtain the image features of facial expressions. S3, multimodal feature fusion: Using the appropriate gait video data processing method and feature extraction model found in the first two steps, as well as the facial expression feature extraction model, gait features and facial expression features are obtained. In order to facilitate subsequent feature fusion, the dimensions of gait features and facial expression features are unified to m dimensions through two fully connected layers. The specific method is as follows: S3.1, Feature-level fusion: In the feature extraction stage, we directly concatenate the m-dimensional gait features and the m-dimensional facial expression features into a 2*m-dimensional high-dimensional feature vector, and then output a two-dimensional feature vector through a fully connected layer; S3.2, Decision-level fusion: We fuse gait features and facial expression features by weighted averaging and other methods, giving weights to the two modalities w1 and w2 respectively, and obtain an m-dimensional fusion feature vector; then output a two-dimensional feature vector through a fully connected layer; S3.3, Hybrid-level fusion: We input the two modal features into the fully connected layer separately to obtain their respective scores; The two scores are then concatenated into an m+1-dimensional fusion feature, which is then input into the same fully connected layer for model training, outputting two two-dimensional features that are then added together to obtain the final result. Finally, the transfer learning method is used to fuse the features of the two modalities through a fully connected layer, and finally a vector consisting of two elements is output, representing the probabilities of PD and non-PD respectively.

2. The method for intelligently assessing Parkinson's disease by integrating facial expression and behavioral gait data according to claim 1, characterized in that: In step S1.3, if the key point is visible, it is 1, otherwise it is 0, indicating the Kronecker delta function, such as v i >0 is 1, otherwise it is 0; the final OKS value is between 0 and 1, and the closer it is to 1, the closer the predicted key point is to the real key point; The key points used here are the standards of COCO17.

3. The Parkinson's disease intelligent assessment method integrating facial expression and behavior gait data according to claim 1, characterized in that: In step S2.2, another feature of StyleGAN is the use of progressive layers. The lower the layer and resolution, the coarser the features it affects. In the latent space, each latent vector corresponds to a generated face image. By translating or superimposing these expression direction vectors on the basis of the original latent vector, new latent vectors with different expressions but the same identity can be obtained.

4. The method for intelligently assessing Parkinson's disease by integrating facial expression and behavioral gait data according to claim 1, characterized in that: In step S2.2, when x = A, the term in f (1-2label A ) evaluates to 1, and the calculated direction is On the contrary, when x=B, the term in f (1-2label B ) evaluates to -1, and the calculated direction is Indicates the opposite direction from expression B to expression A.

5. The method for intelligently assessing Parkinson's disease by integrating facial expression and behavioral gait data according to claim 1, characterized in that: The six basic emotions include anger, disgust, fear, sadness, happiness and surprise.

Images