Parkinson's disease auxiliary diagnosis method and system based on multi-modal dynamic facial expression

Abstract

The invention discloses a Parkinson's disease auxiliary diagnosis method and system based on multi-modal dynamic facial expressions, and the method comprises the steps: obtaining videos, simulating different facial expressions, of a to-be-diagnosed patient, and extracting a facial image sequence; inputting the facial image sequences of different expressions into a visual feature extraction module based on a ViT network, and extracting visual features; inputting texts describing different facial expressions into a text processing module based on a CLIP model, and extracting text features; calculating cosine similarity between the two modal features to fuse the features, obtaining multi-dimensional dynamic facial expression intensity features, extracting statistical index values of the features, and obtaining optimized expression intensity features; and inputting the optimized expression intensity features into an LSTM-based classification network, and outputting a final auxiliary diagnosis result. Through cross-modal fusion of visual and text features, the defects of pure visual features are overcome, the recognition precision of different emotion categories is improved, and an accurate reference basis is provided for auxiliary diagnosis of Parkinson's disease.

Description

Technical Field

[0001] The present invention relates to the field of auxiliary diagnosis technology, and in particular to a Parkinson's disease auxiliary diagnosis method and system based on multimodal dynamic facial expressions. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, while research on Parkinson's disease (PD) auxiliary diagnosis based on static facial expressions has achieved some success, less attention has been paid to dynamic facial expression features. In fact, facial expression disorders in Parkinson's disease patients are not only manifested as a reduction or absence of facial expressions, but also include dynamic abnormalities such as facial muscle stiffness. Compared with static image analysis, dynamic expression analysis can more comprehensively capture the temporal characteristics of facial movement, such as the amplitude, speed, and duration of expression changes, thereby more sensitively reflecting abnormalities in facial movement in Parkinson's disease patients. Therefore, in-depth exploration and analysis of dynamic facial features is of great significance for accurately depicting the patient's expression disorder status and improving the accuracy of auxiliary diagnosis. It can also provide a more valuable reference basis for clinical evaluation.

[0004] At present, although there are auxiliary diagnosis schemes for Parkinson's disease patients based on facial expression features or comprehensive analysis of static and dynamic facial expression features, the existing schemes are still limited to the characteristics of a single modality, namely facial visual features. The information expression ability of a single modality has certain limitations, which will affect the accuracy of expression analysis and thus affect the results of auxiliary diagnosis. Summary of the Invention

[0005] To address the problem that the above-mentioned existing technologies do not pay enough attention to the dynamic facial features of Parkinson's patients and rely solely on visual features for analysis, the present invention provides a Parkinson's disease auxiliary diagnosis method and system based on multimodal dynamic facial expressions. In the dynamic facial expression analysis task, the extracted visual features are cross-modally fused with the expression description text to make up for the shortcomings of pure visual features, improve the recognition accuracy of different emotion categories, and provide an accurate reference basis for Parkinson's disease auxiliary diagnosis.

[0006] In a first aspect, the present invention provides a method for auxiliary diagnosis of Parkinson's disease based on multimodal dynamic facial expressions.

[0007] A Parkinson's disease auxiliary diagnosis method based on multimodal dynamic facial expressions, comprising: Obtain videos of patients imitating different facial expressions and extract facial image sequences; The facial image sequences with different expressions are input into the visual feature extraction module based on the ViT network to extract visual features; The text describing different facial expressions is input into the text processing module based on the CLIP model to extract text features; Based on the visual features and text features of different facial expressions, the cosine similarity between the two modal features is calculated to fuse the features and obtain multi-dimensional dynamic facial expression intensity features; Extract the statistical index value of the multi-dimensional dynamic facial label intensity feature to obtain the optimized expression intensity feature; The optimized expression intensity features are input into the LSTM-based classification network to output the final auxiliary diagnosis results.

[0008] In a second aspect, the present invention provides a Parkinson's disease auxiliary diagnosis system based on multimodal dynamic facial expressions.

[0009] A Parkinson's disease auxiliary diagnosis system based on multimodal dynamic facial expressions, comprising: An image data acquisition module is used to acquire videos of patients to be diagnosed imitating different facial expressions and extract facial image sequences; A visual feature extraction module is used to input facial image sequences with different expressions into the visual feature extraction module based on the ViT network to extract visual features; A text feature extraction module is used to input text describing different facial expressions into a text processing module based on the CLIP model to extract text features; The feature fusion module is used to calculate the cosine similarity between the visual features and text features of different facial expressions to fuse the features and obtain multi-dimensional dynamic facial expression intensity features; A feature processing module is used to extract the statistical index value of the multi-dimensional dynamic facial label intensity feature to obtain the optimized expression intensity feature; The auxiliary diagnosis module is used to input the optimized expression intensity features into the LSTM-based classification network and output the final auxiliary diagnosis results.

[0010] In a third aspect, the present invention further provides an electronic device comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned Parkinson's disease auxiliary diagnosis method based on multimodal dynamic facial expressions when executing the executable instructions stored in the memory.

[0011] In a fourth aspect, the present invention further provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned Parkinson's disease auxiliary diagnosis method based on multimodal dynamic facial expressions.

[0012] In a fifth aspect, the present invention also provides a computer program product, which includes executable instructions, and the executable instructions are stored in a computer-readable storage medium; wherein, when the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-mentioned Parkinson's disease auxiliary diagnosis method based on multimodal dynamic facial expressions is implemented.

[0013] One or more of the above technical solutions have the following beneficial effects: The present invention proposes a method and system for auxiliary diagnosis of Parkinson's disease based on multimodal dynamic facial expressions. For the dynamic facial expression analysis task, a visual feature extraction module based on Vision Transformer is designed to extract discriminative visual dynamic features from videos of patients imitating different facial expressions; a text processing module based on the CLIP model is constructed to extract semantic features from text descriptions related to facial expressions; the extracted visual features are cross-modally fused with the expression description text, that is, by calculating the cosine similarity between the visual and text features, the fusion of the two modal features is achieved, thereby obtaining the intensity representation of the facial dynamic expression, and the corresponding statistical indicators of the intensity representation are extracted in the data processing module. The auxiliary diagnosis of Parkinson's disease is realized through the LSTM-based classification module. Through the above-mentioned cross-modal feature fusion, the present invention makes up for the shortcomings of pure visual features, improves the recognition accuracy of different emotion categories, and provides an accurate reference basis for auxiliary diagnosis of Parkinson's disease.

[0014] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0016] Figure 1 4 is a flowchart of a method for assisting diagnosis of Parkinson's disease based on multimodal dynamic facial expressions according to an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a ViT network in an embodiment of the present invention; Figure 3 Schematic diagram of the structure of the timing modeling module in an embodiment of the present invention; Figure 4 This is a schematic diagram of the network structure of a text editor in an embodiment of the present invention; Figure 5 Schematic diagram of a feature fusion module in an embodiment of the present invention; Figure 6Schematic diagram of the structure of the LSTM-based classification network in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] It should be noted that the following detailed descriptions are exemplary only and are intended to describe specific embodiments and provide further explanation of the present invention, and are not intended to limit the exemplary embodiments according to the present invention. Unless otherwise indicated, all technical and scientific terms used herein have the same meanings as those commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0018] Example 1 This embodiment provides a method for assisting diagnosis of Parkinson's disease based on multimodal dynamic facial expressions, which mainly proposes an auxiliary diagnosis network that integrates multimodal dynamic facial expression information, such as Figure 1 As shown in the figure, it consists of five parts: a visual processing module based on VisionTransformer, a text processing module based on CLIP, a feature fusion module, a data processing module, and a diagnostic classification module based on LSTM. This network can be used to perform emotion recognition on videos of patients imitating different facial expressions, providing an accurate reference for auxiliary diagnosis of Parkinson's disease. The method includes the following steps: Step S1: Obtain a video of a patient imitating different facial expressions and extract a facial image sequence. In this embodiment, a video of a patient imitating four different facial expressions (neutral, happy, surprised, and angry) is obtained and converted into an image sequence through frame-by-frame processing. The facial region in each frame is accurately extracted using the Face++ algorithm, thereby constructing a continuous facial image sequence.

[0019] Step S2: input the facial image sequences with different expressions into the visual feature extraction module based on the ViT network to extract visual features.

[0020] In this example, a visual feature extraction module based on Vision Transformer (ViT) is proposed to extract effective visual features from videos of Parkinson's disease patients imitating different facial expressions. The module mainly consists of three parts: a Vision Transformer (ViT) network, a Multi-Head Self-Attention (MHSA) network, and a temporal modeling module. Visual feature extraction using this module includes: Step S2.1: Input the facial image sequence into the ViT network for preliminary visual feature encoding. As an image processing model based on the Transformer encoder, ViT is better at capturing global dependencies than traditional convolutional neural networks (CNNs) and overcomes the limitations of local perception and sequential modeling. The ViT network structure used is as follows: Figure 2 shown.

[0021] Specifically, facial expression images are fed into the Vision Transformer (ViT) network. The input image is first divided into multiple fixed-size, non-overlapping image patches. Each patch is then flattened into a one-dimensional vector and converted into an embedding vector via a linear mapping, which serves as the basis for the model's input features. Furthermore, to incorporate spatial position information, these embedding vectors are concatenated with the class label and a positional encoding is added. This positional encoding provides each patch with its position within the original image, thus compensating for the lack of built-in spatial awareness in the Transformer encoder and enabling ViT to model the spatial relationships between patches.

[0022] The positionally encoded vectors are then fed into a multi-layer Transformer encoder to model the global dependencies between image patches, thereby extracting rich image features at multiple levels and achieving an understanding of the overall image semantics. Furthermore, to improve numerical stability and mitigate potential floating-point precision issues, a LayerNormalization layer is introduced into the ViT network. Finally, the feature vectors output by the multi-layer Transformer encoder are converted to feature dimensions suitable for subsequent classification tasks through a Layer Mapping layer, further optimizing their representation capabilities.

[0023] Step S2.2: To enhance the model's ability to model complex facial features and further improve its ability to capture global dependencies, the initial features output by ViT are fed into a Multi-Head Self-Attention (MHSA) network with a residual structure. This MHSA network weights the similarities between features, highlighting the correlations between key regions in the image, thereby extracting richer contextual information. The introduction of the residual structure not only helps alleviate the vanishing gradient problem in deep networks but also promotes the effective transfer of feature information. The combination of ViT and MHSA significantly improves the representation of facial expression features, helping to enhance the model's performance and accuracy in Parkinson's disease recognition tasks.

[0024] Step S2.3: Input the features of the fused context information extracted from each frame into the temporal modeling module to reconstruct the temporal dynamic features of the original video. Figure 3 As shown in the figure, the temporal modeling module is a Transformer-based temporal model. The module first combines facial features with a learnable classification vector (i.e., class mark, which is a vector that assists the operation of the Transformer encoder) to provide a more discriminative input for the subsequent Transformer encoder. Then, by introducing temporal coding, the time information between frames is embedded in the feature sequence, so that the model has the ability to model the temporal structure; then, through the multi-layer Transformer encoder, this feature sequence that combines time and visual information is further deeply modeled and analyzed to explore the potential pattern of facial expression evolution over time and extract deep features; finally, the output deep features are adjusted to the dimension required by the subsequent modules through the Layer Mapping layer, thereby completing the extraction of visual features.

[0025] In the above way, the vision processing module based on Vision Transformer first samples the facial expression video M Frames are used to construct a video frame sequence, and then an image encoder consisting of a ViT network and an MHSA network is constructed to extract image features in the video frame sequence. Finally, the features extracted from the same video are input into the temporal model to obtain visual features. This process is expressed as shown in formula (1): (1) in, Represents the ViT network, Indicates the MHSA network, represents the timing model, represents a sampled image in the input video frame sequence, .

[0026] Step S3: Input the text describing different facial expressions into a text processing module based on the CLIP model to extract text features.

[0027] In this embodiment, in order to improve the network's ability to understand different facial expressions, in addition to extracting visual features of facial expressions, a text processing module based on CLIP (Contrastive Language Image Pre-training) is used to analyze the text describing different facial expressions to extract text features, including: Step S3.1: Input text data describing different facial expressions into a text processing module based on the CLIP model for preliminary processing, including: Before extracting text features, large language models such as ChatGPT are used to summarize text describing different facial expressions. For example, a "happy" expression is described as: a smiling mouth, raised cheeks, wrinkled eyes, and arched eyebrows; a "neutral" expression is described as: relaxed facial muscles, a straight mouth, a smooth forehead, and unremarkable eyebrows; an "angry" expression is described as: furrowed eyebrows, narrow eyes, tightened lips, and flared nostrils; and a "surprised" expression is described as: widened eyes, an open mouth, raised eyebrows, and a frozen expression.

[0028] Based on the acquired text data, the module first tokenizes the input text describing different facial expressions, converting each segment into a fixed-length sequence of word vectors. This allows the model to more effectively understand and process language information. This tokenization process not only ensures a reasonable division of words but also unifies the text representation, laying the foundation for subsequent feature extraction and modeling. The module then constructs corresponding contextual prompt vectors (Tokenized Prompts) for each segment. These prompt vectors are used in the CLIP module by converting a set of text prompts into the sequences required by the CLIP module. For each segment, the corresponding sequences are concatenated into a batch tensor to obtain the contextual prompt vector for that segment. This vector serves as input for the module's text encoding process. These prompts introduce contextual information, helping the model more accurately grasp the semantic relationships and underlying intent within the text, thereby enhancing the expressive power of the language feature representation.

[0029] Step S3.2: After the initial processing, the word vector sequence and the context prompt vector are input into the text encoder. The overall structure of the encoder is as follows: Figure 4As shown in Figure 2, during the encoding process, position encoding is first added to the contextual prompt vector to explicitly introduce word order information, thereby overcoming the problem of lack of built-in order perception in the Transformer encoder. After the encoded vector undergoes dimension mapping, the prompt vector is input into the multi-layer Transformer encoder to extract its deep semantic features. After feature extraction, the model fuses the deep representation of the contextual prompt with the original word vector sequence to generate a complete text feature representation. The fusion process is shown in formula (2): (2) in, Represents the deep features extracted by the Transformer encoder, represents a sequence of word vectors, represents the projection matrix to map the output to a space convenient for subsequent calculations, and B represents the batch size.

[0030] In this way, the text information describing different facial expressions is used as input in the CLIP-based text processing module. The corresponding word vector sequence and context prompt vector are first calculated, and then the two are input into the text encoder to obtain the final text features. , the process can be expressed as formula (3): (3) in, T represents a sequence of word vectors, represents the contextual prompt vector, Represents a text encoder.

[0031] Step S4: Based on the visual features and text features of different facial expressions, the cosine similarity between the two modal features is calculated to fuse the features and obtain a multi-dimensional dynamic facial expression intensity feature.

[0032] In this embodiment, a feature fusion module is set to combine visual features With text features Fusion is performed to obtain dynamic facial expression intensity features. The schematic diagram of this module is as follows Figure 5 As shown, H Indicates the number of expression categories that the network can analyze. In order to effectively conduct comparative learning of visual and text features, the visual features and text features are fused by calculating cosine similarity to obtain facial expression intensity feature data. The calculation formula of cosine similarity is shown in formula (4): (4) in, Representing text features G The row vectors in , represents the L2 norm, represents the normalization parameter.

[0033] By computing visual features F and text features G The cosine similarity of H results, assuming the label of the original video input is h , then the training optimization goal of this module is to maximize while minimizing the rest Furthermore, during the model training process, the feature fusion module is trained using contrast loss, as shown in formula (5): (5) in, Representing text features G The label is h The row vector of Represents visual features, represents the normalization parameter, H The number of results obtained by calculating the cosine similarity.

[0034] The Vision Transformer-based visual feature extraction module, the CLIP-based text feature extraction module, and the data processing module together form a multimodal dynamic facial expression analysis network. This network extracts visual features from facial expression videos and text features describing the relevant expressions. The network then fuses these features by calculating cosine similarity to obtain expression intensity features for the input facial video. In this example, four facial expressions—neutral, happy, surprised, and angry—are selected for analysis, and the corresponding expression intensity features are extracted.

[0035] Step S5: extracting statistical index values ​​of the multi-dimensional dynamic facial label strength feature to obtain optimized expression strength features.

[0036] Specifically, considering that after a facial expression video is input, the network processes the visual information and expression text information of the video and outputs a four-dimensional vector to represent the expression intensity characteristics of the video under the four facial expression categories of neutral, happy, surprised and angry. Therefore, after each patient to be diagnosed imitates the four facial expression videos of neutral, happy, surprised and angry and inputs them into the multimodal dynamic facial expression analysis network, a total of 16-dimensional facial expression intensity features are obtained. , the meaning of each dimension feature is shown in Table 1.

[0037] Table 1 The corresponding meanings of expression intensity features

[0038] In order to enhance the model's ability to perceive the expression intensity information that matches the input expression label, this embodiment uses the expression intensity feature I The data is input to the data processing module for data processing to extract statistical indicators. In the data processing module, first, the multi-dimensional dynamic facial label intensity features are grouped according to the different facial expression videos input. I Divided into four groups, ,in Afterwards, the data processing module calculates some statistical indicators of each set of feature data, including key values ​​(Highlight Value, hv ),average value( Mean ), Standard Deviation, std ), Z-Score Percentage Difference, pd ),scope( Range ), Difference from Minimum, ) and Difference from Maximum, ). On this basis, the original expression intensity feature I Combined with all statistical features (i.e. statistical indicators) extracted by the above data processing module, the optimized expression intensity feature is obtained. .

[0039] The calculation method of the above statistical indicators is shown in formula (6): (6) in, Mean It provides the central tendency of expression intensity characteristics, reflects the overall level of facial expression intensity, and can reveal the average value of expression intensity displayed by patients in different expression imitations; std The method measures the degree of data dispersion and quantifies the fluctuation range of expression intensity of different expression categories in facial videos, thereby reflecting the variation and instability of facial expressions. Z-Score Reflected by standardization hv Relative difference from the rest of the expression; pd A quantitative measure that reflects the difference in expression intensity, used to compare the relative differences between different expression categories; Range Indicates the range of the data, describes the overall variation range of each group of expression intensity characteristics, and helps evaluate the diversity and richness of expression; and It reflects hvThe extreme deviations in this data set provide information on abnormal fluctuations in the expression imitation process.

[0040] By extracting statistical indicators of facial expression intensity, we can not only comprehensively characterize its changing trends and distribution characteristics, but also significantly enrich the feature expressions that subsequent classification models rely on. The introduction of these statistical features enables the network to more effectively capture the dynamic changes in Parkinson's patients when imitating different facial expressions, thereby improving the model's ability to perceive and identify temporal changes in expressions. The design of this module provides more precise, quantitative, and physiologically relevant feature support for auxiliary diagnosis, helping to enhance the accuracy and stability of the classification model in Parkinson's disease diagnosis tasks.

[0041] Step S6: Input the optimized expression intensity feature into the LSTM-based classification network and output the final auxiliary diagnosis result.

[0042] Finally, the features after the data processing module The data is sent to a classification network based on LSTM (Long Short-Term Memory Network) to complete the final diagnosis classification task. The structure of the classification network based on LSTM is as follows: Figure 6 As shown, the classification network only processes one-dimensional vectors The processing includes multiple linear layers and LSTM networks. The LSTM network consists of a series of recurrent units, each of which includes three core gating structures: the forget gate, the input gate, and the output gate. LSTM dynamically regulates the transmission process of feature data through the gating mechanism, effectively improving the network's ability to model data.

[0043] Compared to other classification networks, the LSTM network effectively mitigates the vanishing gradient problem when processing expression intensity features and enhances feature representation, thereby improving diagnostic prediction accuracy. Furthermore, this module incorporates a residual structure based on the LSTM network to accelerate network convergence and prevent overfitting. The specific structure and parameters of the different layers of the LSTM-based diagnostic classification network are shown in Table 2.

[0044] Table 2 Classification network architecture

[0045] Furthermore, to enhance the performance of the multimodal dynamic facial expression analysis network in extracting facial expression intensity features, the network was pre-trained to improve its feature expression capability and generalization performance. The network was trained on the existing DFEW (Dynamic Facial Expression in-the-Wild) dataset, a large-scale facial expression dataset containing 16,372 facial expression video clips from various movies, covering a wide range of complex emotional responses.

[0046] When pre-training the multimodal dynamic facial expression analysis network, video clips representing four basic facial expressions (neutral, happy, surprised, and angry) were selected from the DFEW dataset as the training dataset. The specific composition of the training dataset used in the pre-training phase is shown in Table 3.

[0047] Table 3 Composition of training dataset

[0048] When pre-training the multimodal dynamic facial expression analysis network, the batch size was set to 16 and a total of 50 rounds of training were performed. The learning rate of the image encoder in the Vision Transformer-based visual processing module was set to , the learning rate of the temporal network is set to , the learning rate of the text encoder in the CLIP-based text processing module is .

[0049] Furthermore, the dynamic facial expression intensity feature data is analyzed. The multimodal dynamic facial expression analysis network proposed in this embodiment extracts expression intensity features from facial expression videos to provide effective feature representation. The facial expression intensity features have excellent effectiveness in analyzing expression disorders in Parkinson's patients and can effectively support subsequent auxiliary diagnosis and classification tasks. In the multimodal dynamic facial expression analysis network, the network predicts the category label to which the expression video belongs by calculating the maximum value in the output expression intensity vector. This process is shown in formula (7): (7) in, represents the expression intensity vector extracted from the facial video, pred Indicates the predicted result.

[0050] In addition, for the network model proposed in this embodiment, the evaluation indicators used include accuracy, precision, recall, and F1-Score. Among them, accuracy is the most intuitive performance evaluation indicator, indicating the proportion of correctly predicted samples to the total number of samples, which can reflect the classification effect of the model in the overall sample; precision is a measure of the proportion of samples predicted as positive by the model that are actually positive, and this indicator mainly focuses on the proportion of false positive samples; recall measures the proportion of samples that the model can correctly identify among all samples that are actually positive, and is commonly used in medical scenarios; F1-Score, as the harmonic average of precision and recall, comprehensively considers the trade-off between the two and can provide a more comprehensive evaluation in the case of class imbalance.

[0051] Through the above scheme, this embodiment uses a multimodal dynamic facial expression analysis network to analyze facial expression videos. By introducing the joint representation of text information and visual data, the model's ability to understand emotional expression is improved. The fusion of text features not only supplements the deficiencies of visual features, but also provides richer contextual information, enabling the model to more accurately capture changes in expression features, thereby further improving classification performance and the model's generalization ability.

[0052] Example 2 This embodiment provides a Parkinson's disease auxiliary diagnosis system based on multimodal dynamic facial expressions, including: An image data acquisition module is used to acquire videos of patients to be diagnosed imitating different facial expressions and extract facial image sequences; A visual feature extraction module is used to input facial image sequences with different expressions into the visual feature extraction module based on the ViT network to extract visual features; A text feature extraction module is used to input text describing different facial expressions into a text processing module based on the CLIP model to extract text features; The feature fusion module is used to calculate the cosine similarity between the visual features and text features of different facial expressions to fuse the features and obtain multi-dimensional dynamic facial expression intensity features; A feature processing module is used to extract the statistical index value of the multi-dimensional dynamic facial label intensity feature to obtain the optimized expression intensity feature; The auxiliary diagnosis module is used to input the optimized expression intensity features into the LSTM-based classification network and output the final auxiliary diagnosis results.

[0053] Example 3 This embodiment provides an electronic device, including: a memory for storing executable instructions; and a processor for implementing the above method provided in this embodiment when executing the executable instructions stored in the memory.

[0054] Example 4 This embodiment further provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by a processor, the processor will be caused to execute the above method provided in this embodiment.

[0055] Example 5 This embodiment provides a computer program product including executable instructions, which are computer instructions stored in a computer-readable storage medium. When a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the electronic device performs the method provided in this embodiment.

[0056] The steps involved in the above embodiments 2 to 5 correspond to those in embodiment 1. For detailed implementation, please refer to the relevant description of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media that includes one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and cause the processor to perform any method of the present invention.

[0057] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0058] The above description is only a preferred embodiment of the present invention. Although the specific implementation of the present invention is described in conjunction with the accompanying drawings, it does not limit the scope of protection of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A Parkinson's disease auxiliary diagnosis method based on multimodal dynamic facial expressions, characterized in that: include: Obtain videos of patients imitating different facial expressions and extract facial image sequences; The facial image sequences with different expressions are input into the visual feature extraction module based on the ViT network to extract visual features; The text describing different facial expressions is input into the text processing module based on the CLIP model to extract text features; Based on the visual features and text features of different facial expressions, the cosine similarity between the two modal features is calculated to fuse the features and obtain multi-dimensional dynamic facial expression intensity features; Extract the statistical index value of the multi-dimensional dynamic facial label intensity feature to obtain the optimized expression intensity feature; The optimized expression intensity features are input into the LSTM-based classification network to output the final auxiliary diagnosis results.

2. The Parkinson's disease auxiliary diagnosis method based on multimodal dynamic facial expressions according to claim 1, characterized in that: The ViT network-based visual feature extraction module includes a ViT network, a multi-head self-attention mechanism network, and a temporal modeling module; Visual feature extraction is performed using a visual feature extraction module based on the ViT network, including: The facial image sequence is input into the ViT network for preliminary visual feature encoding, that is, the input image is divided into multiple image blocks, each image block is flattened into a one-dimensional vector, and converted into an embedding vector through a linear mapping. The embedding vector is concatenated with the class label and a positional encoding is added. The vector embedded in the positional encoding is then input into a multi-layer Transformer encoder to extract the initial features. The initial features output by the ViT network are input into a multi-head self-attention mechanism network with a residual structure. The similarity between features is calculated by weight to extract features that integrate contextual information. The features of the fused context information extracted from each frame of the image are input into the temporal modeling module to reconstruct the temporal dynamic features in the original video and extract visual features.

3. The Parkinson's disease auxiliary diagnosis method based on multimodal dynamic facial expressions according to claim 1, characterized in that: The text feature extraction is performed using the text processing module based on the CLIP model, including: Perform word segmentation on the input text data describing different facial expressions, convert each text segment into a fixed-length word vector sequence, and construct a corresponding context prompt vector for each text segment; The word vector sequence and the context prompt vector are input into the text encoder together. Position encoding is first added to the context prompt vector. The encoded vector is dimensionally mapped and then input into the multi-layer Transformer encoder to extract deep semantic features. Finally, the deep semantic features are fused with the original word vector sequence to generate text features.

4. The Parkinson's disease auxiliary diagnosis method based on multimodal dynamic facial expressions according to claim 1, characterized in that: The calculation formula of the cosine similarity is: ； in, Representing text features G The row vectors in , represents the L2 norm, represents the normalization parameter.

5. The Parkinson's disease auxiliary diagnosis method based on multimodal dynamic facial expressions according to claim 1, characterized in that: During the feature fusion training process, contrast loss is used for training, which can be expressed as: ； in, Representing text features G The label is h The row vector of Represents visual features, represents the normalization parameter, H The number of results obtained by calculating the cosine similarity.

6. The Parkinson's disease auxiliary diagnosis method based on multimodal dynamic facial expressions according to claim 1, characterized in that: According to the different facial expression videos, the multi-dimensional dynamic facial label strength features are grouped, and the statistical index values ​​of each group of features are calculated; among them, the statistical indicators include key value, mean, standard deviation, Z-Score, percentage difference, range, minimum difference and maximum difference.

7. A Parkinson's disease auxiliary diagnosis system based on multimodal dynamic facial expressions, characterized in that: include: An image data acquisition module is used to acquire videos of patients to be diagnosed imitating different facial expressions and extract facial image sequences; A visual feature extraction module is used to input facial image sequences with different expressions into the visual feature extraction module based on the ViT network to extract visual features; A text feature extraction module is used to input text describing different facial expressions into a text processing module based on the CLIP model to extract text features; The feature fusion module is used to calculate the cosine similarity between the visual features and text features of different facial expressions to fuse the features and obtain multi-dimensional dynamic facial expression intensity features; A feature processing module is used to extract the statistical index value of the multi-dimensional dynamic facial label intensity feature to obtain the optimized expression intensity feature; The auxiliary diagnosis module is used to input the optimized expression intensity features into the LSTM-based classification network and output the final auxiliary diagnosis results.

8. An electronic device, characterized in that: include: a memory for storing executable instructions; The processor is configured to implement the Parkinson's disease auxiliary diagnosis method based on multimodal dynamic facial expressions as described in any one of claims 1 to 6 when executing the executable instructions stored in the memory.

9. A computer-readable storage medium, characterized in that Executable instructions are stored, which are used to cause a processor to execute the executable instructions to implement the Parkinson's disease auxiliary diagnosis method based on multimodal dynamic facial expressions according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes executable instructions stored in a computer-readable storage medium; When the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the Parkinson's disease auxiliary diagnosis method based on multimodal dynamic facial expressions according to any one of claims 1 to 6 is implemented.

Images