An intelligent Parkinson's disease assessment method integrating facial expression and behavioral gait data

Abstract

The present invention relates to the field of data processing technology, and more particularly to an intelligent Parkinson's disease assessment method that integrates facial expression and behavioral gait data. The method comprises three steps: first, data cleaning of a Parkinson's disease patient's behavioral gait video, segmenting the patient from the video using the YOLOv8 model, extracting skeletal key points using HRNet, and acquiring features using STGCN++; second, using StyleGAN to generate pre-disease images that retain the patient's identity information, training multiple deep learning models to classify facial expression images of each person's six basic emotions, and using the best pre-trained model to extract features of the patient's facial expression; and third, performing multimodal feature fusion of the extracted gait features and facial expression features for Parkinson's disease prediction. The method significantly improves diagnostic accuracy and reduces the problems of missed or misdiagnosed cases.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an intelligent Parkinson's disease assessment method that integrates facial expression and behavioral gait data. Background Art

[0002] Parkinson's disease (PD) is a chronic, progressive neurodegenerative disorder that is now second only to Alzheimer's in prevalence. It is often referred to as the "never-ending tremor." The average age of onset is around 60 years old, with most cases occurring in middle-aged and elderly individuals. Although the prevalence of PD is low, it is characterized by rapid progression and high disability, necessitating early screening for early intervention, treatment, and management. While there is no effective treatment to prevent or cure Parkinson's disease, levodopa and other related medications can effectively improve symptoms and enhance patients' quality of life.

[0003] One way to diagnose Parkinson's disease is with the help of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS) of the Movement Disorder Society (MDS). However, this assessment tool cannot capture subtle changes in motor function. In addition, the MDS-UPDRS assessment process relies on the patient's subjective self-report and the doctor's experience and judgment, and has objectivity and quantification deficiencies, which leads to a high misdiagnosis rate in Parkinson's disease screening. Another commonly used method for diagnosing Parkinson's disease is in vivo diagnosis, which mainly relies on professional medical imaging technology and diagnoses and differentiates patients based on the imaging characteristics of Parkinson's disease. This method requires patients to go to a professional medical institution for diagnosis, which is not convenient for elderly people with limited mobility. Secondly, the diagnostic costs of professional equipment are high, placing a financial burden on families. Compared with in vivo diagnosis, in vitro diagnostic methods are more convenient and less expensive. Among the commonly used in vitro biomarkers, diagnosis through voice signals may be affected by language type or even dialect, gait signals require wearing professional gait signal sensors, and dynamic note collection solutions currently lack unified standards. The collection of facial expressions as biomarkers requires patients to make multiple expressions when photographing. This method is relatively convenient and economical. However, in the in vitro diagnosis of Parkinson's disease, not all patients have certain unimodal symptoms. For example, the voices of some early patients are not much different from those when the patients are not sick, which leads to diagnostic errors in models trained with unimodal data. Therefore, unimodal diagnostic methods have limitations in the early diagnosis of Parkinson's disease.

[0004] Deep learning is currently performing increasingly well in fields such as computer vision. Due to its strong fitting capabilities, deep learning can be used to train data-driven diagnostic models with excellent generalization capabilities. These models only require minor adjustments to adapt to downstream tasks, eliminating the need for extensive time and financial resources to deploy applications. We have also discovered that using deep learning to extract facial expression features and gait video features can avoid the shortcomings of speech and handwriting signals, and that feature fusion techniques can compensate for the low generalization issues encountered in single-modality diagnosis. Therefore, the present invention proposes a research method for intelligent assisted diagnosis of Parkinson's disease that uses a deep learning model to extract facial expression images and gait video features for multimodal fusion. This method has the advantages of convenient data collection, lightweight models, good generalization performance, and ease of deployment and promotion. Summary of the Invention

[0005] In response to the above-mentioned deficiencies in the prior art, the present invention provides an intelligent Parkinson's disease assessment method that integrates facial expression and behavioral gait data to improve the reliability of diagnosis.

[0006] An intelligent Parkinson's disease assessment method integrating facial expression and behavioral gait data includes the following steps:

[0007] S1. Behavioral gait feature extraction based on skeleton key points:

[0008] The YOLOv8 tracking and segmentation model is used to track patients in the video and use masks to segment the video to only contain the PD patient's gait. The YOLOv8 segmentation detection task includes segmentation-related losses, classification losses inherited from the object detection task, and rectangular box regression losses, namely CIoU loss and DFL.

[0009] S1.1. CIoU loss: Adopt the adversarial loss in the WGAN-GP model to replace the original adversarial loss in the StarGAN model.

[0010] Loss, the adversarial loss function is as follows:

[0011]

[0012] Where IoU is the intersection over union ratio of the predicted bounding box and the true bounding box; ρ(b, b g ) is the Euclidean distance between the center point of the predicted bounding box and the center point of the true bounding box; c is the bounding box b around the predicted bounding box and the true bounding box b g The diagonal length of the minimum rectangular box; α is a tuning parameter used to balance the weights of IoU loss and center point distance loss; v is the aspect ratio loss, defined as:

[0013]

[0014] S1.2. DFL is defined as:

[0015]

[0016] in and is the probability distribution of two adjacent discrete points predicted by the model, y i and y i+1 is the regression target value of the true bounding box, and y is the bounding box regression value predicted by the model;

[0017] S1.3, classification loss and segmentation loss: Both classification loss and segmentation loss use BCEWithLogitsLoss. Segmentation loss is mainly a pixel-by-pixel calculation of binary cross entropy loss between the predicted segmentation area and the ground truth segmentation area. BCEWithLogitsLoss is defined as:

[0018] l bn，c =-w bn，c [p c y bn，c ·log(σ(x bn，c ))+(1-y bn，c )·log(1-σ(x bn，c ))] (4)

[0019] where l bn，c is the bounding box classification loss for the bth batch, nth sample, and cth class (c>1 for multi-label binary classification, c=1 for single-label binary classification); p c is the true label of class c; y bn，c is the true label of the cth class of the nth sample in the bth batch, with a value of 0 or 1; x bn，c is the prediction score of the bth batch, nth sample, and cth class output by the model; logσ(x bn，c ) is to use the sigmoid function to convert x bn，c The predicted score is converted into a probability value; w bn，c Is a weight factor used to control the loss weight of different categories or different samples;

[0020] Finally, the four losses calculated above are weighted summed to obtain the total loss l all , defined as:

[0021] l all =w1l CloU +w2l DFL +w3l class +w4l seg (5)

[0022] where l CloU is IoU loss, l DFL is the DFL loss, l class is the classification loss, l seg To split the loss;

[0023] After obtaining data containing only video clips of Parkinson's disease patients' steps through tracking segmentation extraction, a behavioral gait feature extraction method based on skeleton key points was adopted; the first part was to use HRNet to extract the video clips into skeleton key point data. The loss function used by HRNet was MSE, which is defined as:

[0024]

[0025] Where n is the number of samples, y i is the true target value of the i-th sample, is the predicted value of the i-th sample; and OKS is used as the evaluation indicator, which is defined as:

[0026]

[0027] Where i represents the i-th key point, represents the Euclidean distance between the predicted coordinates and the true coordinates of the i-th key point, s represents the scale of the image, usually the height or width of the human body, Represents the scale-adaptive scaling factor of the key point, which is used to balance the importance of different key points, v i Indicates the visibility mark of the i-th key point;

[0028] The second part is to use the extracted skeleton key points to perform feature extraction through the excellent STGCN++ network;

[0029] S2. Synthesize facial expressions and extract features based on generative adversarial networks:

[0030] Use StyleGAN to synthesize virtual facial expression images of PD patients in their pre-disease state;

[0031] S2.1. StyleGAN generates latent vectors: A pre-trained StyleGAN generator is combined with an encoder network. During training, the parameters of the encoder network are fine-tuned using a carefully designed similarity loss function, which is a weighted combination of the VGG-16 perceptual loss and the per-pixel mean squared error (MSE) loss. The loss function is defined as follows:

[0032]

[0033] where c represents the target latent vector, I0 represents the input image, G represents the pre-trained StyleGAN generator, N represents the total number of image pixels, and λ mse is a weight hyperparameter; the perceptual loss L percept Use the pre-trained VGG network to measure the perceptual difference between the original image I0 and the synthesized image G(c); this perceptual loss is defined as follows:

[0034]

[0035] Among them C j (·) represents the feature map output of the jth convolutional layer of the VGG-16 network, k represents the total number of convolutional layers, and N j represents the total number of pixels in the feature map of the jth convolutional layer, λ j is a weight hyperparameter;

[0036] S2.2, StyleGAN calculates facial expression direction vectors: obtains latent vectors that share consistent identity information but express different facial expressions A and B;

[0037] Then, the transition direction vector from expression A to expression B is calculated. Specifically, the potential vector sets of expressions A and B are first obtained and recorded as label A and label B , and assign them labels latent A =0 and latent B =1; then build the mapping function in is considered as the normal vector from expression A to expression B, and the term (1-2label x ) is used to control direction;

[0038] Next, a logistic regression model P = σ(f) is constructed to output the predicted class label probability, whose value is between [0,1], and the normal vector that can manipulate the transition from A to B is determined by optimizing the binary cross entropy (BCE) loss. this is the desired direction vector n AB ;

[0039] Similarly, by adopting the above-mentioned calculation algorithm, the direction vectors between multiple facial expression states are obtained, so that different types of facial expressions can be synthesized while preserving the identity;

[0040] S2.3. Multi-expression synthesis and deep feature extraction: According to the method described in step S2.2, multiple direction vectors capable of controlling different facial expressions can be successfully obtained. The process of facial expression generation can be formalized as follows:

[0041]

[0042] Among them G enc and G dec denote the encoder and decoder of G, I denotes a facial image with expression A, and I ′ represents an image of the same person with expression B, is the direction vector from expression A to B, and λ controls the degree of expression change;

[0043] By using the above facial expression synthesis method, facial expression images representing six basic emotions can be generated from a single neutral facial expression image of a Parkinson's disease patient, capturing the expressions before the onset of the disease. These images will serve as a control group to construct an enhanced Parkinson's disease facial expression training dataset. Subsequently, a deep learning model is used to train a discriminant model on this expanded training dataset. Finally, the model is used as a feature extractor and six facial images with different expressions are used as input. The trained expression classifier is used as a pre-trained model to fine-tune the model parameters to obtain image features of facial expressions.

[0044] S3. Multimodal feature fusion:

[0045] Using the appropriate gait video data processing method and feature extraction model found in the first two steps, as well as the facial expression feature extraction model, gait features and facial expression features are obtained. To facilitate subsequent feature fusion, the dimensions of gait features and facial expression features are unified to m dimensions through two fully connected layers. The specific method is as follows:

[0046] S3.1. Feature-level fusion: In the feature extraction stage, we directly concatenate the m-dimensional gait features and the m-dimensional facial expression features into a 2*m-dimensional high-dimensional feature vector, and then output a two-dimensional feature vector through a fully connected layer;

[0047] S3.2, Decision-level Fusion: We fuse gait features and facial expression features separately through weighted averaging, assigning weights to the two modalities w1 and w2 respectively, to obtain an m-dimensional fused feature vector. This is then passed through a fully connected layer to output a two-dimensional feature vector.

[0048] S3.3, Hybrid-level fusion: We input the two modal features into a fully connected layer separately to obtain their respective scores. These two scores are then concatenated into an m+1-dimensional fused feature. Finally, the two scores are input into the same fully connected layer for model training, outputting two 2D features that are summed to obtain the final result.

[0049] Finally, the transfer learning method is used to fuse the features of the two modalities through a fully connected layer, and finally a vector consisting of two elements is output, representing the probabilities of PD and non-PD respectively.

[0050] Furthermore, in step S1.3, if the keypoint is visible, it is 1, otherwise it is 0, indicating the Kronecker delta function, such as v i If the value is >0, it is 1, otherwise it is 0; the final OKS value is between 0 and 1, and the closer it is to 1, the closer the predicted key point is to the true key point; the key point used here is the COCO17 standard.

[0051] Furthermore, in step S2.2, another feature of StyleGAN is the use of progressive layers. The lower the layer and resolution, the coarser the features it affects. In the latent space, each latent vector corresponds to a generated face image. By translating or superimposing these expression direction vectors on the basis of the original latent vector, new latent vectors with different expressions but the same identity can be obtained.

[0052] Furthermore, in step S2.2, when x=A, the item in f (1-2label A ) evaluates to 1, the calculated direction is On the contrary, when x=B, the term in f (1-2label B ) evaluates to -1, and the calculated direction is Indicates the opposite direction from expression B to expression A.

[0053] Furthermore, the six basic emotions include anger, disgust, fear, sadness, happiness, and surprise.

[0054] The beneficial effects of the present invention are:

[0055] This paper proposes a new multimodal in vitro diagnostic scheme for PD based on facial expressions and behavioral gait. It is the first to propose a method for in vitro diagnosis of PD using daily mobile phone video data. By introducing gait features, compared with previous diagnostic methods that only use facial expressions, the accuracy of diagnosis is significantly improved, the number of missed diagnoses and misdiagnoses is reduced, and the excellent performance of multimodal in vitro diagnosis of PD is confirmed.

[0056] To reduce the difficulty of patient diagnosis and to meet the requirements of using mobile phones to capture gait video data in any environment, this paper proposes a series of effective data processing solutions. At the same time, to address the limited data volume of traditional facial expression-based diagnosis methods and the lack of facial expressions when the patient is not ill, AICG technology (artificial synthesis based on StyleGAN) is used for data augmentation, effectively improving the robustness of the model. At the same time, this paper uses a lightweight model architecture to improve the practicality, ease of deployment, and scalability of the proposed diagnostic strategy to better meet the challenges of the increasing number of PD patients in an aging society in the future. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 It is an overview of the contents of the present invention;

[0059] Figure 2 This is the STGCN++ model structure diagram;

[0060] Figure 3 Schematic diagram of the StyleGAN model training process and different features of hierarchical latent vector control;

[0061] Figure 4 Schematic diagram for calculating direction vector;

[0062] Figure 5 Schematic diagram of a neural network trained for expression classification;

[0063] Figure 6 Schematic diagram of feature extraction for 6 types using expression classifier as pre-training model;

[0064] Figure 7 Schematic diagram of feature fusion and PD / non-PD classification using multiple feature fusion methods. DETAILED DESCRIPTION

[0065] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0066] A method for intelligent Parkinson's disease assessment that integrates facial expressions and behavioral gait data. The method is divided into the following three steps: First, the behavioral gait video of Parkinson's disease patients is cleaned, and the patient is segmented from the video using the YOLOv8 model. The skeletal key points are extracted using HRNet, and features are obtained using STGCN++. Second, StyleGAN is used to generate pre-disease images that retain the patient's identity information to make up for the missing control group data in the data we collected. After that, multiple deep learning models will be trained to classify the facial expression images of the six basic emotions (anger, disgust, fear, happiness, sadness, and surprise) collected for each person, and the best pre-trained model will be used to extract the features of the patient's facial expression. Third, the model with good generalization performance and few model parameters in the first two steps is used as a feature extractor, and the extracted gait features and facial expression features are fused for multimodal features for Parkinson's disease prediction, and then the classifier is trained for PD / non-PD classification. Specific research content is as follows: Figure 1 As shown, the specific implementation of each step is as follows:

[0067] S1. Behavioral gait feature extraction based on skeleton key points:

[0068] In this step, we first preprocess the collected data. Since the PD patient gait video data we use was not captured in a specific environment or under strict shooting conditions, the video dataset contains many interference factors. Therefore, we first use the YOLOv8 tracking and segmentation model to track the patient in the video and use a mask to segment the video to only contain the PD patient's gait. The YOLOv8 segmentation detection task mainly consists of segmentation-related losses, a classification loss inherited from the object detection task, and a rectangular box regression loss (CIoU loss and DFL):

[0069] S1.1, CIoU loss: The adversarial loss in the WGAN-GP model is used to replace the original adversarial loss in the StarGAN model to obtain more stable model training and higher overall image quality. The adversarial loss function is as follows:

[0070]

[0071] Where IoU is the intersection over union ratio of the predicted bounding box and the true bounding box; ρ(b, b g ) is the Euclidean distance between the center point of the predicted bounding box and the center point of the true bounding box; c is the bounding box b around the predicted bounding box and the true bounding box b gThe diagonal length of the minimum rectangular box; α is a tuning parameter used to balance the weights of IoU loss and center point distance loss; v is the aspect ratio loss, defined as:

[0072]

[0073] S1.2, DFL (Distribution Focal Loss): DFL does not directly predict the bounding box coordinates, but predicts a probability distribution, that is, the probability of different coordinate values, using the probability distribution of two adjacent discrete points and Instead of a single point prediction value, the loss function is calculated based on the true bounding box coordinate y i and y i+1 The difference between the predicted value and the predicted value is calculated using logarithmic loss to optimize the model. The distribution-based prediction method allows the model to learn a smoother bounding box regression distribution, thereby improving the prediction accuracy of difficult samples. DFL is defined as:

[0074]

[0075] in and is the probability distribution of two adjacent discrete points predicted by the model, y i and y i+1 is the regression target value of the true bounding box, and y is the bounding box regression value predicted by the model

[0076] S1.3, classification loss and segmentation loss: Both classification loss and segmentation loss use BCEWithLogitsLoss. Segmentation loss is mainly a pixel-by-pixel calculation of binary cross entropy loss between the predicted segmentation area and the ground truth segmentation area. BCEWithLogitsLoss is defined as:

[0077] l bn，c =-w bn，c [p c y bn，c ·log(σ(x bn，c ))+(1-y bn，c )·log(1-σ(x bn，c ))] (4)

[0078] where l bn，c is the bounding box classification loss for the bth batch, nth sample, and cth class (c>1 for multi-label binary classification, c=1 for single-label binary classification); p c is the true label of class c. bn，c is the true label of the cth class of the nth sample in the bth batch, with a value of 0 or 1; xbn，c is the prediction score of the bth batch, nth sample, and cth class output by the model; logσ(x bn，c ) is to use the sigmoid function to convert x bn，c The predicted score is converted into a probability value; w bn，c It is a weight factor used to control the loss weight of different categories or different samples.

[0079] Finally, the four losses calculated above are weighted summed to obtain the total loss l all , defined as:

[0080] l all =w1l cloU +w2l DFL +w3l class +w4l seg (5)

[0081] where l CloU is IoU loss, l DFL is the DFL loss, l class is the classification loss, l seg For segmentation loss.

[0082] After obtaining video clips containing only the gait of Parkinson's disease patients through tracking and segmentation extraction, we decided to use a behavioral gait feature extraction method based on skeletal keypoints, based on the widespread application of deep learning skeletal keypoint detection technology in recent years and its advantage in accurately extracting spatial coordinate information of human joints. The first step is to use HRNet to extract skeletal keypoint data from the video clips. The loss function used by HRNet is MSE (Mean Squared Error Loss), which is defined as:

[0083]

[0084] Where n is the number of samples, y i is the true target value of the i-th sample, is the predicted value of the i-th sample. And OKS (Object Keypoint Similarity) is used as the evaluation indicator. OKS is defined as:

[0085]

[0086] Where i represents the i-th key point, represents the Euclidean distance between the predicted coordinates and the true coordinates of the i-th key point, s represents the scale of the image, usually the height or width of the human body, Represents the scale-adaptive scaling factor of the key point, which is used to balance the importance of different key points, v i Indicates the visibility flag of the i-th key point, which is 1 if the key point is visible and 0 otherwise, indicating the Kronecker delta function, such as v i If the predicted keypoints are closer to 1, the predicted keypoints are closer to the true keypoints. The keypoints used here are based on the COCO17 standard.

[0087] The second part is to use the extracted skeleton key points to extract features through the excellent STGCN++ network. The model structure of STGCN++ is as follows Figure 2 As shown in Figure 2. STGCN++ is an enhanced version of the original STGCN model.

[0088] S2. Synthesize facial expressions and extract features based on generative adversarial networks:

[0089] Due to the lack of pre-onset control samples from PD patients, training an effective discriminative model for feature extraction from our collected facial expression data is challenging. To address this issue, we used StyleGAN to synthesize virtual facial expression images of PD patients in their pre-onset state. We then explored the use of different deep learning models for feature extraction and classification training on the augmented facial expression data.

[0090] StyleGAN is a powerful and flexible image generation model based on generative adversarial networks, which excels in creativity, realism, and controllability. The generation process of StyleGAN is subject to fine-grained control of an intermediate "style" latent space. Various attributes of the image, such as expression, posture, skin color, etc., can be changed by adjusting this latent space vector. Therefore, if any input face is mapped to a latent vector in the latent space of the StyleGAN generative model through a special encoder, and the latent vector can be used to generate the corresponding face through a pre-trained StyleGAN generator, with the rich expressive power of StyleGAN, by modifying the different dimensions of this latent vector, various editing operations can be performed on the input image, thereby achieving the generation of faces with different expressions while ensuring that the identity information remains unchanged. The StyleGAN training process is as follows: Figure 3 As shown in the figure on the left, the following details the process of extracting facial expression features from PD patients:

[0091] S2.1. StyleGAN generates latent vectors: A combination of a pre-trained StyleGAN generator and an encoder network is used. During the training process, we optimize the encoder parameters to ensure that the reconstructed image generated by the generator is as close as possible to the original input image. This approach enables us to obtain a StyleGAN latent vector with similar identity information to the original input image. It is worth noting that during the training process, the parameters of the encoder network are fine-tuned through a carefully designed similarity loss function, which is a weighted combination of the VGG-16 perceptual loss and the per-pixel mean squared error (MSE) loss. The loss function is defined as follows:

[0092]

[0093] Where c represents the target latent vector, I0 represents the input image, G represents the pre-trained StyleGAN generator, N represents the total number of image pixels, and λ mse Is a weight hyperparameter. Perceptual loss L percept Use the pre-trained VGG network to measure the perceptual difference between the original image I0 and the synthesized image G(c). This perceptual loss is defined as follows:

[0094]

[0095] Among them C j (.) represents the feature map output of the jth convolutional layer of the VGG-16 network, k represents the total number of convolutional layers, N j represents the total number of pixels in the feature map of the jth convolutional layer, λ j is a weight hyperparameter.

[0096] S2.2. StyleGAN calculates the facial expression direction vector: Another feature of StyleGAN is the use of progressive layers. One potential benefit of progressive layers is that, if used properly, they can control different visual features of the image, such as Figure 3 As shown in the figure on the right. The lower the layer and resolution, the coarser the features it affects. In the latent space, each latent vector corresponds to a generated face image, such as Figure 4 As shown in Figure 2, by translating or superimposing the original latent vector (e.g., P1) along these expression direction vectors, a new latent vector (e.g., P'1) with different expressions but the same identity can be obtained. Following the method in step S2.1, latent vectors can be obtained that share consistent identity information but express different facial expressions A and B (e.g., neutral and happy).

[0097] Then, the transition direction vector from expression A to expression B is calculated. Specifically, the potential vector sets of expressions A and B are first obtained and recorded as label Aand label B , and assign them labels latent A =0 and latent B = 1. Then we construct the mapping function in It can be regarded as the normal vector from expression A to expression B, and the term (1-2label x ) is used to control the direction. When x=A, the item in f (1-2label A ) evaluates to 1, the calculated direction is On the contrary, when x=B, the term in f (1-2label B ) evaluates to -1, and the calculated direction is Indicates the opposite direction from expression B to expression A. Next, a logistic regression model P = σ(f) is constructed to output the predicted class label probability, whose value is between [0,1], and the normal vector that can manipulate the transition from A to B is determined by optimizing the binary cross entropy (BCE) loss. this The desired direction vector n can be approximated as AB .

[0098] Similarly, by adopting the above calculation algorithm, the direction vectors between multiple facial expression states can be obtained, so that different types of facial expressions can be synthesized while preserving the identity.

[0099] S2.3. Multi-expression synthesis and deep feature extraction: Based on the method described in step S2.2, multiple direction vectors capable of controlling different facial expressions can be successfully obtained. For any facial image, the method mentioned in step S2.1 is first applied to extract its corresponding latent vector in the StyleGAN latent space. Subsequently, by performing linear calculations between the latent vector and the specific direction vector controlling the facial expression, the StyleGAN generator G is used to synthesize a new image with the desired facial expression while preserving the identity information. The process of facial expression generation can be formalized as follows:

[0100]

[0101] Among them G enc and G dec denote the encoder and decoder of G respectively, I denotes a facial image with expression A, and I′ denotes an image of the same person with expression B. is the direction vector from expression A to B, and λ controls the degree of expression change.

[0102] By using the above-mentioned facial expression synthesis method, facial expression images representing six basic emotions (anger, disgust, fear, sadness, happiness, and surprise) are generated from a single neutral facial expression image of a Parkinson's disease patient, capturing the expression before the onset of the disease. These images will serve as a control group to construct an enhanced Parkinson's disease facial expression training dataset. Subsequently, various deep learning models (such as ResNet and MobileNetV3) will be explored on this expanded training dataset to train the discriminant model. The goal is to achieve facial expression classification and extract corresponding facial features, such as Figure 5 Finally, we will seek a balance between model size and facial expression classification performance, select a deep learning model that is both high-performance and suitable for deployment on mobile devices, and on this basis, select the best and lightweight model as the feature extractor and use six facial images with different expressions as input. The trained expression classifier is used as a pre-training model, and the model parameters are fine-tuned to obtain the image features of facial expressions. The structure is shown in the figure. Figure 6 shown.

[0103] S3. Multimodal feature fusion:

[0104] In this step, we use the appropriate gait video data processing method and feature extraction model found in the first two steps, as well as the facial expression feature extraction model, to obtain gait features and facial expression features. We then explored a variety of multimodal feature fusion methods, including: feature-level fusion (early fusion), decision-level fusion (late fusion), and hybrid-level fusion. Before multimodal fusion, in order to facilitate subsequent feature fusion, we use two fully connected layers to unify the dimensions of gait features and facial expression features to m dimensions. The fusion method is as follows: Figure 7 The specific method is as follows:

[0105] S3.1. Feature-level fusion: In the feature extraction stage, we directly concatenate the m-dimensional gait features and the m-dimensional facial expression features into a 2*m-dimensional high-dimensional feature vector, and then output a two-dimensional feature vector through a fully connected layer.

[0106] S3.2, Decision-Level Fusion: We fuse gait features and facial expression features using weighted averaging, assigning weights to the two modalities w1 and w2, respectively, to obtain an m-dimensional fused feature vector. This is then passed through a fully connected layer to output a two-dimensional feature vector.

[0107] S3.3, Hybrid-level Fusion: We input the two modal features into a fully connected layer to obtain their respective scores. These two scores are then concatenated into an m+1-dimensional fused feature. Finally, they are input into the same fully connected layer for model training, outputting two 2D features that are then summed to obtain the final result.

[0108] Finally, we use the transfer learning method. Specifically, the behavior gait feature extractor and facial expression feature extractor are fine-tuned based on pre-trained models (such as ResNet and MobileNetV3). The parameters are frozen and not updated. Only the multimodal fusion model is trained, that is, the features of the two modalities are fused through the fully connected layer, and finally a vector consisting of two elements is output, representing the probability of PD and non-PD respectively.

[0109] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A Parkinson's disease intelligent assessment method integrating facial expression and behavioral gait data, characterized in that: The following steps are involved: S1. Behavioral gait feature extraction based on skeleton key points: The YOLOv8 tracking and segmentation model is used to track patients in the video and use masks to segment the video to only contain the PD patient's gait. The YOLOv8 segmentation detection task includes segmentation-related losses, classification losses inherited from the object detection task, and rectangular box regression losses, namely CIoU loss and DFL. S1.

1. CIoU loss: The adversarial loss in the WGAN-GP model is used to replace the original adversarial loss in the StarGAN model. The adversarial loss function is as follows: Where IoU is the intersection over union ratio of the predicted bounding box and the true bounding box; ρ(b, b g ) is the Euclidean distance between the center point of the predicted bounding box and the center point of the true bounding box; c is the bounding box b around the predicted bounding box and the true bounding box b g The diagonal length of the minimum rectangular box; α is a tuning parameter used to balance the weights of IoU loss and center point distance loss; v is the aspect ratio loss, defined as: S1.

2. DFL is defined as: in and is the probability distribution of two adjacent discrete points predicted by the model, y i and y i+1 is the regression target value of the true bounding box, and y is the bounding box regression value predicted by the model; S1.3, classification loss and segmentation loss: Both classification loss and segmentation loss use BCEWithLogitsLoss. Segmentation loss is mainly the binary cross entropy loss calculated pixel by pixel between the predicted segmentation area and the ground truth segmentation area. BCEWithLogitsLoss is defined as: where l bn，c is the bounding box classification loss for the bth batch, nth sample, and cth class, where c>1 indicates multi-label binary classification and c=1 is for single-label binary classification; p c is the true label of class c; y bn,c is the true label of the cth class of the nth sample in the bth batch, with a value of 0 or 1; x bn，c is the prediction score of the bth batch, nth sample, and cth class output by the model; logσ(x bn，c ) is to use the sigmoid function to convert x bn，c The predicted score is converted into a probability value; w bn，c Is a weight factor used to control the loss weight of different categories or different samples; Finally, the four losses calculated above are weighted summed to obtain the total loss l all , defined as: the all =w1l CloU +w2l DFL +w3l class +w4l seg (5) where l CloU is IoU loss, l DFL is the DFL loss, l class is the classification loss, l seg To split the loss; After obtaining data containing only video clips of Parkinson's disease patients' steps through tracking segmentation extraction, a behavioral gait feature extraction method based on skeleton key points was adopted; the first part was to use HRNet to extract the video clips into skeleton key point data. The loss function used by HRNet was MSE, which is defined as: Where n is the number of samples, h i is the true target value of the i-th sample, is the predicted value of the i-th sample; and OKS is used as the evaluation indicator, which is defined as: Where i represents the i-th key point, represents the Euclidean distance between the predicted coordinates and the true coordinates of the i-th key point, s represents the scale of the image, usually the height or width of the human body, Represents the scale-adaptive scaling factor of the key point, which is used to balance the importance of different key points, v i Indicates the visibility mark of the i-th key point; The second part is to use the extracted skeleton key points to perform feature extraction through the excellent STGCN++ network; S2. Synthesize facial expressions and extract features based on generative adversarial networks: Use StyleGAN to synthesize virtual facial expression images of PD patients in their pre-disease state; S2.

1. StyleGAN generates latent vectors: A pre-trained StyleGAN generator is combined with an encoder network. During training, the parameters of the encoder network are fine-tuned using a carefully designed similarity loss function, which is a weighted combination of the VGG-16 perceptual loss and the per-pixel mean squared error (MSE) loss. The loss function is defined as follows: where c represents the target latent vector, I0 represents the input image, G represents the pre-trained StyleGAN generator, N represents the total number of image pixels, and λ mse is a weight hyperparameter; the perceptual loss L percept Use the pre-trained VGG network to measure the perceptual difference between the original image I0 and the synthesized image G(c); this perceptual loss is defined as follows: Among them C j (·) represents the feature map output of the jth convolutional layer of the VGG-16 network, k represents the total number of convolutional layers, and N j represents the total number of pixels in the feature map of the jth convolutional layer, λ j is a weight hyperparameter; S2.2, StyleGAN calculates facial expression direction vectors: obtains latent vectors that share consistent identity information but express different facial expressions A and B; Then, the transition direction vector from expression A to expression B is calculated. Specifically, the potential vector sets of expressions A and B are first obtained and recorded as label A and label B , and assign them labels latent A =0 and latent B =1; then build the mapping function in is considered as the normal vector from expression A to expression B, and the term (1-2label x ) is used to control direction; Next, a logistic regression model P = σ(f) is constructed to output the predicted class label probability, whose value is between [0,1], and the normal vector that can manipulate the transition from A to B is determined by optimizing the binary cross entropy (BCE) loss. this is the desired direction vector n AB ; Similarly, by adopting the above-mentioned calculation algorithm, the direction vectors between multiple facial expression states are obtained, so that different types of facial expressions can be synthesized while preserving the identity; S2.

3. Multi-expression synthesis and deep feature extraction: Based on the method described in step S2.2, multiple direction vectors capable of controlling different facial expressions are successfully obtained. The process of facial expression generation is formalized as follows: Among them G enc and G dec denote the encoder and decoder of G respectively, I denotes a facial image with expression A, and I′ denotes an image of the same person with expression B. is the direction vector from expression A to B, and λ controls the degree of expression change; By adopting the above facial expression synthesis method, it is possible to generate facial expression images representing six basic emotions from a single neutral facial expression image of a Parkinson's disease patient, capturing the expression before the onset of the disease; These images will serve as a control group to construct an enhanced Parkinson's disease facial expression training dataset. Subsequently, a deep learning model will be used on this expanded training dataset to train a discriminative model. Finally, the model will be used as a feature extractor with six facial images of people with different expressions as input. The trained expression classifier will be used as a pre-trained model to fine-tune the model parameters and obtain image features of facial expressions. S3. Multimodal feature fusion: Using the appropriate gait video data processing method and feature extraction model found in the first two steps, as well as the facial expression feature extraction model, gait features and facial expression features are obtained. To facilitate subsequent feature fusion, the dimensions of gait features and facial expression features are unified to m dimensions through two fully connected layers. The specific method is as follows: S3.

1. Feature-level fusion: In the feature extraction stage, we directly concatenate the m-dimensional gait features and the m-dimensional facial expression features into a 2*m-dimensional high-dimensional feature vector, and then output a two-dimensional feature vector through a fully connected layer; S3.2, Decision-level Fusion: We fuse gait features and facial expression features separately through weighted averaging, assigning weights to the two modalities w1 and w2 respectively, to obtain an m-dimensional fused feature vector. This is then passed through a fully connected layer to output a two-dimensional feature vector. S3.3, Hybrid-level fusion: We input the two modal features into the fully connected layer separately to obtain their respective scores; These two scores are then concatenated into an m+1 dimensional fusion feature, which is then fed into the same fully connected layer for model training. This outputs two 2D features, which are then summed to obtain the final result. Finally, the transfer learning method is used to fuse the features of the two modalities through a fully connected layer, and finally a vector consisting of two elements is output, representing the probabilities of PD and non-PD respectively.

2. The method for intelligently assessing Parkinson's disease by integrating facial expression and behavioral gait data according to claim 1, characterized in that: In step S1.3, if the keypoint is visible, it is 1, otherwise it is 0, indicating the Kronecker delta function, such as v i >0 is 1, otherwise it is 0; the final OKS value is between 0 and 1, and the closer it is to 1, the closer the predicted key point is to the true key point; The key points used here are the standards of COCO17.

3. The method for intelligent Parkinson's disease assessment based on the integration of facial expression and behavioral gait data according to claim 1, characterized in that: In step S2.2, another feature of StyleGAN is the use of progressive layers. The lower the layer and resolution, the coarser the features it affects. In the latent space, each latent vector corresponds to a generated face image. By translating or superimposing these expression direction vectors on the basis of the original latent vector, new latent vectors with different expressions but the same identity can be obtained.

4. The method for intelligently assessing Parkinson's disease by integrating facial expression and behavioral gait data according to claim 1, characterized in that: In step S2.2, when x=A, the item in f (1-2label A ) evaluates to 1, the calculated direction is On the contrary, when x=B, the term in f (1-2label B ) evaluates to -1, and the calculated direction is Indicates the opposite direction from expression B to expression A.

5. The method for intelligently assessing Parkinson's disease by integrating facial expression and behavioral gait data according to claim 1, characterized in that: The six basic emotions include anger, disgust, fear, sadness, happiness, and surprise.

Images