Parkinson depression micro-expression recognition method and system based on pathological association transfer learning

Abstract

The invention discloses a Parkinson depression micro-expression recognition method and system based on pathological association transfer learning, and belongs to the field of medical artificial intelligence. The method comprises the following steps: firstly, collecting a face video and clinical pathological data, extracting micro-expression spatial-temporal features and pathological features after preprocessing, fusing through an attention mechanism, combining migration, confrontation and a multi-task learning training model, and outputting a recognition result. The system comprises a data acquisition module, a data preprocessing module, a feature extraction module, a pathological association feature fusion module, a transfer learning and model training module and a micro-expression recognition module. The problems that feature extraction is difficult, over-fitting is caused and pathological information is not used are solved, and the Parkinson depression micro-expression recognition accuracy is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical artificial intelligence, and in particular to a Parkinson's depression micro-expression recognition method and system based on pathological correlation transfer learning. BACKGROUND

[0002] Parkinson's disease is a common neurodegenerative disease. In addition to motor symptoms, non-motor symptoms such as depression are also very common in Parkinson's patients, which seriously affects the quality of life of patients. Micro-expression refers to facial expressions that are extremely short in duration (usually between 1 / 25 and 1 / 5 seconds) and difficult to detect with the naked eye, and can reflect the true emotional state of individuals. By identifying the depression micro-expression of Parkinson's patients, important evidence can be provided for early diagnosis and intervention.

[0003] However, the existing micro-expression recognition technology has many defects when applied to Parkinson's depression diagnosis: on the one hand, micro-expression itself has a short duration and low motion intensity, and traditional feature extraction methods are difficult to effectively capture its subtle features; on the other hand, due to the difficulty of collecting samples of Parkinson's patients and the complexity of micro-expression data labeling, the related data set is small in size, which makes the deep learning model prone to overfitting and poor generalization ability. In addition, existing micro-expression recognition methods often only focus on facial expression features, ignoring the clinical and pathological information related to Parkinson's disease itself, and failing to fully utilize the potential relationship between the two to improve recognition accuracy. SUMMARY

[0004] Therefore, the present application provides a Parkinson's depression micro-expression recognition method and system based on pathological correlation transfer learning, which improves the accuracy and reliability of Parkinson's depression micro-expression recognition, and solves the problems of difficulty in extracting micro-expression features, model overfitting caused by small data set size, and insufficient use of pathological information in the prior art.

[0005] To achieve the above purpose, one of the present application provides the following technical scheme:

[0006] The Parkinson's depression micro-expression recognition method based on pathological correlation transfer learning comprises the following steps:

[0007] Data collection step: Collect facial video data of Parkinson's patients and normal people under specific stimuli, the design of specific stimuli aims to effectively arouse the emotional response of the subjects, such as watching carefully selected emotion-inducing videos, the video content covers various emotional scenes, including joy, sadness, anger, etc., to stimulate the subjects to produce corresponding micro expressions; carry out emotional arousal question and answer, the questions revolve around the life experience and emotional experience of the subjects, guide the subjects to recall and produce emotional fluctuations. At the same time, collect detailed clinical pathological data of Parkinson's patients, record the time span from the first appearance of Parkinson's disease related symptoms to the present; the severity of the disease is evaluated by using the internationally accepted scale, such as MDS-UPDRS, etc.; the drug use records the information of the names, doses, and use frequency of various anti-Parkinson drugs used by the patients;

[0008] Data preprocessing step: Denoising the collected facial video data, using advanced video denoising algorithms, such as deep learning based denoising network, to remove Gaussian noise, salt and pepper noise and other interference in the video; through image stabilization technology, using feature point matching and motion estimation, eliminate the picture jitter caused by the slight movement of the head. Use mature face detection algorithms, such as Haar cascade classifier or deep learning face detection model, to accurately detect the face in the video, and through key point positioning technology, such as deep learning based face key point detection network, extract 68 key points of the face, and then extract the face region of interest, such as the eye and mouth regions where the micro expression changes are obvious, and normalize them to a uniform size for subsequent processing. Standardize the clinical pathological data, convert the clinical data of different dimensions into data with uniform dimensions through data normalization formula, so that they can effectively participate in the calculation in the subsequent model;

[0009] Feature extraction step: Use spatiotemporal feature extraction network to process the preprocessed facial video data, the spatiotemporal feature extraction network is an improved 3D convolutional neural network, which introduces residual connection in the network to solve the gradient vanishing problem in deep network training, so that the network can learn more rich micro expression features; introduce attention module, automatically focus on the key features in the micro expression by calculating the importance weight of each feature channel, so as to extract the spatiotemporal dynamic features of the micro expression, including the time series of muscle contraction and relaxation in the time dimension, and the distribution information of different facial regions in the spatial dimension, such as muscle movement amplitude and coordination relationship. Use pathological feature extraction module to extract features from clinical pathological data, the pathological feature extraction module uses fully connected neural network, through the nonlinear transformation of multiple neurons, to excavate the deep features in the clinical pathological data, and get the pathological feature vector related to Parkinson's disease, such as the feature vector reflecting the disease progression stage, drug treatment effect, etc.;

[0010] Pathological correlation feature fusion step: a pathological correlation fusion model is constructed to fuse the extracted micro-expression spatio-temporal dynamic features and pathological feature vectors, different features are assigned weights through an attention mechanism, and features that have important influence on Parkinson's depression recognition are highlighted;

[0011] Dual-mode cross-attention fusion model core formula:

[0012]

[0013] wherein F S is the spatio-temporal feature, and F P is the pathological feature.

[0014] Weight distribution:

[0015]

[0016]

[0017] is a learnable attention matrix;

[0018] Fusion feature:

[0019]

[0020] For example, for patients with a long course of disease and obvious sadness features in micro-expression, higher weights are given to the corresponding course features and sadness micro-expression features;

[0021] Transfer learning and model training step: a deep learning model pre-trained on a large-scale general facial expression dataset is selected as a base model, such as ResNet, DenseNet and other models pre-trained on FER2013, CK+ and other datasets, and the general features learned by these models in general expression recognition are used. The fused features are input into the base model, and fine-tuning training is performed on the Parkinson's depression related dataset. In the training process, an adversarial training mechanism is adopted, a discriminator is introduced to distinguish the feature representation of Parkinson's depression patients and normal people, through the adversarial game between the generator and the discriminator, the model learns more discriminative features, and the sensitivity of the model to Parkinson's depression features is improved. At the same time, a multi-task learning method is used to train the model to recognize Parkinson's disease and Parkinson's depression, share part of the network layer parameters, use the correlation between the two tasks, and improve the generalization ability and feature learning ability of the model;

[0022] Micro-expression recognition step: after the facial video data to be recognized and the corresponding clinical pathology data are processed and feature extracted and fused according to the above steps, the data are input into the trained model, the model analyzes and judges the input features, and outputs the recognition result of Parkinson's depression, including whether Parkinson's depression is present and the prediction score of the depression degree, the depression degree score can be in the form of a 0-10 scale, 0 indicating no depression and 10 indicating severe depression.

[0023] Technical effect of the basic scheme: the scheme uses multiple steps in a synergistic manner, combines data acquisition with specific stimulation and clinical pathology data, pre-processing, feature extraction to capture the spatio-temporal dynamics and pathological characteristics of micro-expression respectively, pathological correlation fusion to highlight key features by means of attention mechanism, and the comprehensive use of various means such as transfer learning and adversarial training, which improves the generalization and discrimination ability of the model, and makes it possible to finally realize the accurate recognition of Parkinson's depression and degree and effectively overcome the defects of the prior art and significantly improve the recognition efficiency.

[0024] Further, the spatio-temporal feature extraction network is an improved 3D convolutional neural network, which introduces residual connection and attention module in the network to enhance the extraction ability of micro-expression fine features and the attention to important features.

[0025] Beneficial effect: for the improved 3D convolutional neural network with residual connection and attention module as the spatio-temporal feature extraction network, the 3D convolution can capture the spatial form and temporal change of micro-expression at the same time, the residual connection solves the gradient vanishing problem during the training of deep network, and the network can learn complex features deeply; the attention module can automatically focus on the key facial area for depression recognition, enhance the extraction of fine micro-expression, and improve the effectiveness of features.

[0026] Further, the pathological feature extraction module uses a fully connected neural network to perform nonlinear transformation on the clinical pathology data to extract deep pathological features.

[0027] Beneficial effect: the fully connected neural network is used in the pathological feature extraction module, which can perform nonlinear transformation on the clinical pathology data to mine the deep correlation between the disease course, medication, etc. and depression, and convert discrete pathological indicators into a unified feature vector, providing valuable pathological basis for subsequent fusion.

[0028] Further, in the transfer learning and model training step, a multi-task learning method is used to simultaneously train the model to recognize Parkinson's disease and Parkinson's depression, share part of the network layer parameters, and improve the generalization ability and feature learning ability of the model.

[0029] Beneficial effect: Using multi-task learning to train Parkinson's disease and its depression recognition at the same time, sharing parameters can make the model learn the features of the two related tasks at the same time, improve the understanding of the association between disease and depression through the complementary information between tasks, enhance the generalization and feature learning ability of the model, and alleviate the problem of small data volume.

[0030] Further, in the data collection step, the collected facial video data is also labeled, and the labeling content includes the start time and end time of the micro-expression, the micro-expression type (happy, sad, angry), and the corresponding Parkinson's depression state.

[0031] Beneficial effect: Labeling in the data collection step clearly defines the time, type and corresponding depression state of micro-expression, providing precise supervision information for model training, allowing the model to learn the corresponding relationship between micro-expression and depression, improving recognition accuracy, and making the data more usable and consistent.

[0032] Further, the disease severity score adopts the MDS-UPDRS scale.

[0033] Beneficial effect: From the disease severity score using the MDS-UPDRS scale, the MDS-UPDRS scale is an internationally recognized and standardized Parkinson's disease comprehensive evaluation scale that can comprehensively and accurately reflect the severity of Parkinson's disease. Including it in the claims can ensure that the collected clinical pathological data is uniform and authoritative, reduce data errors caused by non-uniform scoring standards, make the extracted pathological features more reliable and comparable, and thus provide high-quality basic data for subsequent feature fusion and model training, improving the accuracy of Parkinson's depression micro-expression recognition.

[0034] Further, the large-scale general facial expression dataset includes FER2013 and CK+.

[0035] Beneficial effect: FER2013 and CK+ are widely used and have high recognition in the field of expression recognition, containing rich expression samples and various expression types. Clearly defining these datasets can ensure the quality of pre-training of the base model in transfer learning, and the general expression features learned by the base model on these large-scale data are more representative and have generalization ability. Using these datasets for model fine-tuning for Parkinson's depression micro-expression recognition can effectively utilize existing knowledge, solve the problem of small scale of Parkinson's depression related datasets, enhance the learning ability and generalization performance of the model, and improve the stability of the recognition effect.

[0036] Based on the above method, the present application also provides a Parkinson's depression micro-expression recognition system based on pathological association transfer learning, comprising:

[0037] The data acquisition module is responsible for collecting facial video data of Parkinson's patients and normal people under specific stimulation and clinical pathological data of Parkinson's patients, and the facial video data can be collected through a camera device, and the clinical pathological data can be acquired through a hospital information system.

[0038] The data preprocessing module is responsible for denoising, stabilization, face detection, key point positioning, region of interest extraction and normalization processing of the collected facial video data, and standardization processing of the clinical pathological data, and the above functions are realized by using corresponding algorithm libraries and software modules.

[0039] The feature extraction module includes a spatio-temporal feature extraction submodule and a pathological feature extraction submodule, the spatio-temporal feature extraction submodule extracts the spatio-temporal dynamic features of micro-expression by using a spatio-temporal feature extraction network, and the pathological feature extraction submodule extracts the pathological feature vector of the clinical pathological data by using a pathological feature extraction module, and the feature extraction function is realized by constructing a corresponding neural network model and training.

[0040] The pathological correlation feature fusion module constructs a pathological correlation fusion model, fuses the spatio-temporal dynamic features of micro-expression and the pathological feature vector by using an attention mechanism, and realizes the construction of the fusion model and the calculation of the attention mechanism by writing program codes.

[0041] The transfer learning and model training module selects a deep learning model pre-trained on a large-scale general facial expression dataset as a base model, fine-tunes the fused features for training, and trains by using an adversarial training mechanism and a multi-task learning method, and realizes the training process of the model by using a deep learning framework such as TensorFlow, PyTorch, etc.

[0042] The micro-expression recognition module inputs the processed facial video data and clinical pathological data into the trained model, outputs the recognition result of Parkinson's depression, and predicts and displays the recognition result by calling the trained model.

[0043] Further, the spatio-temporal feature extraction network used by the spatio-temporal feature extraction submodule is an improved 3D convolutional neural network, which contains a residual connection and an attention module.

[0044] Further, the pathological feature extraction module used by the pathological feature extraction submodule is a fully connected neural network.

[0045] The beneficial effects of the present application are as follows:

[0046] The present application fuses pathological information and micro-expression features, fully utilizes the pathological relationship between Parkinson's disease and depression, and improves the accuracy and reliability of recognition compared with the recognition method based on micro-expression only. For example, by analyzing the course of disease, severity of disease and other information in the clinical pathological data, combined with micro-expression features, it can more comprehensively judge whether the patient has depressive symptoms, and avoid the misjudgment that may occur only according to micro-expression.

[0047] The improved spatiotemporal feature extraction network and attention mechanism effectively enhance the extraction ability of micro-expression subtle features, can capture the micro-expression changes that are difficult to detect by traditional methods, and improve the precision of micro-expression recognition. For example, in the face of some weak and short micro-expression, the method of the present application can more accurately identify its type and meaning, and provide a stronger basis for the diagnosis of Parkinson's depression.

[0048] Using transfer learning and adversarial training mechanism, the overfitting problem caused by small scale of Parkinson's depression related data set is effectively solved, the generalization ability of the model is improved, and the model can be better applied to actual clinical scene. For example, on the Parkinson's patient data collected by different hospitals and different devices, the model of the present application can still maintain high recognition accuracy, and has good universality and adaptability.

[0049] The adoption of multi-task learning mode further improves the learning ability of the model to the features related to Parkinson's disease and Parkinson's depression, and through sharing network layer parameters, the model can more efficiently utilize the information in the data, and improve the overall performance. For example, in the training process, the model can simultaneously learn the motor symptom related features of Parkinson's disease and the micro-expression features of depression, and through the mutual promotion between tasks, better understand the internal relationship between the two, thereby improving the recognition effect.

[0050] Other advantages, objects and features of the present application will be set forth in part in the following specification, and in part will become apparent to those skilled in the art from the examination of the following text, or can be learned from the practice of the present application. The objects and other advantages of the present application can be realized and obtained by the following specification. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to make the purpose, technical scheme and advantages of the present application more clear, the preferred detailed description of the present application will be combined with the drawings as follows, in which:

[0052] Figure 1 The schematic diagram of the Parkinson's depression micro-expression recognition system based on pathological correlation transfer learning of the present application;

[0053] Figure 2A flowchart of the Parkinson's disease depression micro-expression recognition method based on pathological correlation transfer learning of the present application. DETAILED DESCRIPTION

[0054] The advantages and effects of the present application can be easily understood by those skilled in the art from the disclosure of the embodiments. The present application can also be implemented or applied in other different embodiments, and various modifications or changes can be made to the details based on different views and applications without departing from the spirit of the present application. It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0055] The accompanying drawings are only used for exemplary illustration, and the representations are only schematic diagrams, not physical diagrams, and cannot be understood as limitations of the present application; in order to better illustrate the embodiments of the present application, some components in the drawings are omitted, enlarged or reduced, and do not represent the actual product size; it is understandable for those skilled in the art that some well-known structures and their descriptions in the drawings can be omitted.

[0056] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it should be understood that if the terms "upper", "lower", "left", "right", "front", "back" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationship in the drawings are only used for exemplary illustration, and cannot be understood as limitations of the present application, for those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0057] As Figure 1 shown, the present application also provides a Parkinson's disease depression micro-expression recognition system based on pathological correlation transfer learning, comprising:

[0058] The data acquisition module is responsible for collecting facial video data of Parkinson's patients and normal people under specific stimulation and clinical pathological data of Parkinson's patients, the facial video data can be collected by a camera device, and the clinical pathological data can be obtained through a hospital information system;

[0059] The data preprocessing module performs denoising, stabilization, face detection, key point positioning, region of interest extraction and normalization processing on the collected facial video data, and standardization processing on the clinical pathological data, and uses corresponding algorithm library and software module to realize the above functions;

[0060] The feature extraction module includes a spatio-temporal feature extraction submodule and a pathological feature extraction submodule. The spatio-temporal feature extraction submodule extracts the spatio-temporal dynamic features of the micro-expression by using a spatio-temporal feature extraction network. The pathological feature extraction submodule extracts the pathological feature vector of the clinical pathological data by using a pathological feature extraction module. The feature extraction function is realized by constructing a corresponding neural network model and training.

[0061] The pathological correlation feature fusion module constructs a pathological correlation fusion model. The micro-expression spatio-temporal dynamic features and the pathological feature vector are fused by using an attention mechanism. The construction of the fusion model and the calculation of the attention mechanism are realized by writing program codes.

[0062] The transfer learning and model training module selects a deep learning model pre-trained on a large-scale general facial expression dataset as a base model. The fused features are input for fine-tuning training. The training is performed by using an adversarial training mechanism and a multi-task learning method. The training process of the model is realized by using a deep learning framework such as TensorFlow or PyTorch.

[0063] The micro-expression recognition module inputs the facial video data to be recognized and the clinical pathological data into the trained model after processing. The recognition result of Parkinson's depression is output. The trained model is called to perform prediction and the recognition result is displayed.

[0064] Further, the spatio-temporal feature extraction network used by the spatio-temporal feature extraction submodule is an improved 3D convolutional neural network containing residual connections and attention modules. The spatio-temporal feature extraction submodule uses an improved 3D convolutional neural network containing residual connections and attention modules. The 3D convolution can capture the spatial distribution and temporal dynamic change of the micro-expression. The residual connection solves the gradient vanishing problem of the deep network, making the network easier to learn complex features. The attention module can automatically focus on the facial regions (such as the eyes and mouth) that are critical for depression recognition, enhancing the ability to extract subtle micro-expression features and improving the effectiveness of feature representation.

[0065] Further, the pathological feature extraction module used by the pathological feature extraction submodule is a fully connected neural network. The fully connected network can perform nonlinear transformation on the clinical pathological data, excavate the hidden deep correlations in the data (such as the potential relationship between the disease course and the depression level), convert discrete pathological indicators into a unified high-dimensional feature vector, and effectively fuse the micro-expression spatio-temporal features, providing more comprehensive pathological basis for subsequent recognition and improving the overall recognition accuracy of the system.

[0066] The Parkinson's depression micro-expression recognition method using the above system is as shown in FIG. 1. The following steps are taken: Figure 2

[0067] 1. Data acquisition ​

[0068] 100 Parkinson's patients and 50 age and gender matched normal controls were recruited in the Department of Neurology, the First Affiliated Hospital of Zhejiang University. Micro-expression was effectively aroused by using stratified stimulation strategy:

[0069] Video stimuli: Select clinically verified segments from MAHNOB-HCI emotional video library (e.g. sad scene: movie "The Pursuit of Happyness" unemployment clip, 45 seconds; angry scene: traffic accident conflict clip, 30 seconds), control video brightness 50-80 cd / m², frame rate 30 fps, avoid visual interference.

[0070] Emotion arousal questionnaire: Design structured question set, such as "please describe the feeling of the last time you couldn't complete daily activities independently due to physical discomfort" (related to course depression), "is there a significant emotional fluctuation after taking medicine?" (related to drug influence), audio record emotional tone change throughout the process to assist labeling.

[0071] At the same time, high-definition camera was used to collect their facial video data at a frame rate of 30 frames / second. For Parkinson's patients, detailed clinical data was collected, including disease duration (average duration of 5 years, range of 1-10 years), MDS-UPDRS score (average score of 30 points, range of 10-50 points) and current anti-Parkinson's disease drug information.

[0072] 2. Data preprocessing

[0073] For the collected facial video data, a deep learning-based denoising network was used to remove noise, and feature point matching and motion estimation algorithm was used for image stabilization processing. A deep learning-based face detection model was used to detect the face, and 68 key points were extracted by a facial key point detection network. The eye and mouth regions of interest were extracted and normalized to 128x128 pixels in size. For clinical pathological data, z-score standardization method was used for standardization processing. The specific steps include:

[0074] Video preprocessing algorithm stack:

[0075] Denoising: Improved DnCNN network (added attention gate module) was used, with a Gaussian noise removal rate of 92% (σ=15-25), and a PSNR improvement of 1.8 dB compared with traditional BM3D algorithm;

[0076] Stabilization: Based on KLT feature point tracking + RANSAC motion estimation algorithm, the correction error of head shaking (displacement ≤5 pixels) was ≤0.5 pixels, ensuring the stability of eye and mouth ROI;

[0077] Face processing: Use MTCNN face detection algorithm (accuracy 99.2%) to locate the face, and combine Dlib 68-point key point model to extract ROI (eye: 36-47 points, mouth: 48-67 points), normalized to 128x128x3 channels (RGB format)

[0078] Pathological data standardization, using piecewise normalization strategy:

[0079] Continuous data such as disease duration and drug dosage: z-score standardization (μ is the mean and σ is the standard deviation)

[0080] MDS-UPDRS score and other classification data: mapped to the [0, 1] interval (such as "0-10 points → 0.2, 11-20 points → 0.5"), retaining the clinical significance weight.

[0081] 3. Feature extraction

[0082] 1) Spatio-temporal feature extraction network, based on C3D network architecture optimization, the core is:

[0083] Residual attention module: Insert "channel attention + spatial attention" double branch structure after the 3rd and 4th convolution layers, channel attention generates a 1x1xC weight vector (C is the number of channels) through Sigmoid activation, and spatial attention focuses on the eye (weight ratio 35%) and mouth (weight ratio 40%) regions through 7x7 convolution;

[0084] Network parameters: 5 3D convolution layers (kernel size 3x3x3, stride 1x1x1), output channel number 64→128→256→512→512, get 512-dimensional spatio-temporal feature vector through global average pooling.

[0085] Effect: The feature capture accuracy of micro-expressions such as "mouth corner droop (sadness)" and "eyelid droop (tiredness)" is improved by 23% compared with traditional 3D-CNN.

[0086] 2) Pathological feature extraction network (deep fully connected network) uses a fully connected network with residual connection:

[0087] Structure: Input layer (12-dimensional pathological features: disease duration, MDS-UPDRS sub-items, etc.) → Hidden layer 1 (128 neurons, ReLU) → Residual connection → Hidden layer 2 (64 neurons, LeakyReLU) → Output layer (32-dimensional pathological feature vector)

[0088] Regularization: Add Dropout (rate=0.3) and L2 regularization (λ=0.001) to avoid overfitting caused by sparse pathological data.

[0089] ​4. Pathological correlation feature fusion

[0090] Dual-modal cross-attention fusion model, the core formula is as follows:

[0091]

[0092] F S is the spatiotemporal feature, F P is the pathological feature.

[0093] Weight distribution:

[0094]

[0095]

[0096] is a learnable attention matrix.

[0097] Fusion feature:

[0098]

[0099] Example: For the sample of "8-year course + MDS-UPDRS score 15 + sad micro-expression", the model allocates Ws=0.42 (sad micro-expression feature) and Wp=0.58 (course and non-motor symptom feature), which is 35% higher than the information utilization rate of the traditional splicing fusion method.

[0100] 5. Transfer learning and model training

[0101] Pre-trained model: ResNet50-3D model pre-trained on FER2013 (35887 expression images) and CK+ (981 sequences) is selected, and the first 3 layers of convolution parameters are frozen (general expression features are retained);

[0102] Fine-tuning strategy: "small batch gradient descent" (batch_size=16, learning rate 0.0001, Adam optimizer) is used for the fusion feature layer, fine-tuned on the Parkinson's depression dataset (200 patients, 800 videos), and the convergence speed is 2.3 times faster than random initialization.

[0103] Discriminator design: 3-layer fully connected network (input 544-dimensional fusion feature→128→64→2 classification), loss function:

[0104] L adv =-E[logD(F fusion )]-E[log(1-D(F fake ))];

[0105] Generator optimization: By backpropagating the adversarial loss, the generator is forced to learn "Parkinson's depression-specific features" (such as "smile micro-expression caused by drug side effects"), which are 18% more discriminative than those without adversarial training.

[0106] Main task: Parkinson's depression recognition (output depression probability + 0-10 scale score);

[0107] Auxiliary task: Parkinson's disease motor symptom recognition (based on MDS-UPDRS III score);

[0108] Loss function: L total = 0.7L depression + 0.3L motor ) (weights determined by cross-validation), share the first 4 layers of convolutional parameters, and the model generalization ability is improved by 25% in small sample scenarios.

[0109] 6. Micro-expression recognition

[0110] After processing the newly collected facial video data and clinical pathology data of 10 Parkinson's patients and 5 normal control subjects according to the above steps, the trained model is input. The model outputs the recognition result. For Parkinson's patients, 3 of them are accurately identified as having depression symptoms, with depression severity scores of 6, 7 and 8 respectively, which has high consistency with the diagnosis results of clinicians.

[0111] The specific embodiment clearly selects samples, standardizes data collection and preprocessing procedures, extracts micro-expression spatiotemporal dynamic features and pathological features using improved 3D convolutional neural network and fully connected neural network respectively, realizes feature fusion using attention mechanism, optimizes the model using transfer learning, adversarial training and multi-task learning, and finally achieves recognition results highly consistent with clinical diagnosis on newly collected samples. The application process and effectiveness of the technical solution of the present application in practical scenarios are fully demonstrated, and an operable example for the implementation of the method is provided.

[0112] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and all should be covered in the scope of the claims of the present application.

Claims

1. A method for recognizing micro-expressions in Parkinson's disease depression based on pathological association transfer learning, characterized in that, Includes the following steps: Data collection steps: Collect facial video data of Parkinson's patients and normal individuals under specific stimuli, including watching emotion-inducing videos and engaging in emotion-evoking question-and-answer sessions; collect clinicopathological data of Parkinson's patients, including disease course, disease severity score, and medication status. Data preprocessing steps: Denoising and stabilizing the collected facial video data to eliminate noise interference and image jitter caused by minor head movements; extracting regions of interest on the face using face detection and key point localization technology and normalizing them to a uniform size; standardizing the clinical pathology data to give it a uniform dimension. Feature extraction steps: The preprocessed facial video data is processed using a spatiotemporal feature extraction network to extract the spatiotemporal dynamic features of micro-expressions, including the temporal changes and spatial distribution information of facial muscle movements; the pathological feature extraction module is used to extract features from the clinical pathological data to obtain pathological feature vectors related to Parkinson's disease. Pathological feature fusion steps: Construct a pathological feature fusion model, fuse the extracted micro-expression spatiotemporal dynamic features and pathological feature vectors, and assign weights to different features through an attention mechanism to highlight features that have an important impact on the identification of Parkinson's depression; Core formula of the bimodal cross-attention fusion model: Among them, F S As a spatiotemporal feature, F P These are pathological features. Weighting: A learnable attention matrix; Fusion characteristics: Transfer learning and model training steps: Select a deep learning model pre-trained on a large-scale general facial expression dataset as the base model, input the fused features into the base model, and fine-tune the training on a Parkinson's depression-related dataset; During the training process, an adversarial training mechanism is adopted to introduce a discriminator to distinguish the feature representations of Parkinson's depression patients and normal people, so as to enable the model to learn more discriminative features. Micro-expression recognition steps: After processing, extracting and fusing the facial video data to be recognized and the corresponding clinical pathology data according to the above steps, the data is input into the trained model, and the recognition result of Parkinson's depression is output, including whether the person has Parkinson's depression and the predicted score of the degree of depression.

2. The method for recognizing micro-expressions in Parkinson's disease depression based on pathological association transfer learning according to claim 1, characterized in that, The spatiotemporal feature extraction network is an improved 3D convolutional neural network that introduces residual connections and attention modules to enhance the ability to extract subtle micro-expression features and focus on important features.

3. The method for recognizing micro-expressions in Parkinson's disease depression based on pathological association transfer learning according to claim 1, characterized in that, The pathological feature extraction module uses a fully connected neural network to perform nonlinear transformations on clinical pathological data and extract deep-level pathological features.

4. The method for recognizing micro-expressions in Parkinson's disease depression based on pathological association transfer learning according to claim 1, characterized in that, In the transfer learning and model training steps, a multi-task learning approach is adopted, simultaneously training the model to identify Parkinson's disease and Parkinson's depression. The discriminator is designed as a 3-layer fully connected network (input 544-dimensional fusion features → 128 → 64 → 2-class classification), and its loss function is: L adv =-E[logD(F fusion )]-E[log(1-D(F fake ))]。 5. The method for recognizing micro-expressions in Parkinson's disease depression based on pathological association transfer learning according to claim 1, characterized in that, The data collection process also includes labeling the collected facial video data. The labeling includes the start and end times of micro-expressions, the type of micro-expression (happy, sad, angry), and the corresponding Parkinson's depression state.

6. The method for recognizing micro-expressions in Parkinson's disease depression based on pathological association transfer learning according to claim 1, characterized in that, The severity of the condition was scored using the MDS-UPDRS scale.

7. The method for recognizing micro-expressions in Parkinson's disease depression based on pathological association transfer learning according to claim 1, characterized in that, The large-scale general facial expression dataset includes FER2013 and CK+.

8. A micro-expression recognition system for Parkinson's disease depression based on pathological association transfer learning, characterized in that, include: Data acquisition module: used to collect facial video data of Parkinson's patients and normal people under specific stimuli, as well as clinical and pathological data of Parkinson's patients; Data preprocessing module: performs noise reduction, stabilization, face detection, key point localization, region of interest extraction and normalization on the collected facial video data, and standardizes the clinical pathology data; Feature extraction module: includes a spatiotemporal feature extraction submodule and a pathological feature extraction submodule. The spatiotemporal feature extraction submodule uses a spatiotemporal feature extraction network to extract the spatiotemporal dynamic features of micro-expressions, and the pathological feature extraction submodule uses a pathological feature extraction module to extract pathological feature vectors from clinical pathological data. Pathological correlation feature fusion module: Constructs a pathological correlation fusion model and uses an attention mechanism to fuse micro-expression spatiotemporal dynamic features and pathological feature vectors; Transfer learning and model training module: A deep learning model pre-trained on a large-scale general facial expression dataset is selected as the base model. The fused feature input is fine-tuned and trained, and an adversarial training mechanism and a multi-task learning method are used for training. Micro-expression recognition module: After processing the facial video data and clinical pathology data to be recognized, it inputs them into the trained model and outputs the recognition results of Parkinson's depression.

9. The Parkinson's Depression Micro-expression Recognition System based on Pathological Association Transfer Learning according to claim 8, characterized in that, The spatiotemporal feature extraction submodule uses an improved 3D convolutional neural network, which includes residual connections and attention modules.

10. The Parkinson's Depression Micro-expression Recognition System based on Pathological Association Transfer Learning according to claim 8, characterized in that, The pathological feature extraction submodule uses a fully connected neural network.

Images