A method and system for facial paralysis dynamic grading based on multi-modal data and graph convolution network

Abstract

This invention discloses a dynamic grading method and system for facial paralysis based on multimodal data and graph convolutional networks, belonging to the field of medical image processing and AI-assisted diagnosis. Addressing the problems of traditional facial paralysis grading being highly subjective, lacking dynamic assessment, and insufficient utilization of multimodal information, this invention simultaneously collects dynamic facial videos, infrared thermal images, and traditional Chinese medicine clinical texts from patients; extracts facial key point motion trajectories, regional temperature asymmetry features, and RoBERTa text features; constructs a heterogeneous graph and performs cross-modal feature alignment and adaptive fusion through a graph convolutional network; jointly outputs static HB grading labels and dynamic functional scores through multi-task learning; introduces an asymmetric cost matrix and uses Bayesian optimization to search for the optimal classification threshold to reduce the risk of missed diagnoses of severe cases. This invention also discloses the corresponding grading system. Experiments show that the grading accuracy reaches 91.6%, the Kappa coefficient reaches 0.85, and the dynamic scoring MAE is 4.3. After deployment at the grassroots level, the diagnostic accuracy increases from 68% to 85%, significantly improving the objectivity of assessment and clinical consistency, and can empower the diagnosis and treatment of facial paralysis in grassroots medical institutions.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing and artificial intelligence-assisted diagnosis technology. Specifically, it relates to a method and system for dynamic grading of facial paralysis based on multimodal data such as dynamic video, infrared thermal imaging and clinical text, using graph convolutional networks for cross-modal feature fusion, and through multi-task learning and cost-sensitive threshold optimization. Background Technology

[0002] Facial paralysis (FP) is a common neurological disorder. Currently, the clinical assessment of the severity of facial paralysis mainly relies on the House-Brackmann (HB) grading system. However, this method has the following technical limitations: 1. High subjectivity and poor consistency: The HB classification relies on the doctor's personal experience, and the consistency coefficient (Kappa) between different assessors is only 0.6-0.7, which leads to about 30% of patients being misclassified, affecting the choice of treatment plan. 2. Static assessment, lack of dynamic information: Existing methods are based on static facial features and cannot quantify key dynamic parameters such as the speed of eye closing and the range of mouth movements, resulting in a lag in monitoring the rehabilitation process and making it difficult to guide timely adjustments to the treatment plan. 3. Single-modal information, lack of complementarity: Most existing methods only utilize visible light images or videos, ignoring the facial blood circulation information reflected by infrared thermal imaging and the syndrome information contained in traditional Chinese medicine clinical texts (tongue appearance, pulse appearance), which are of great value for comprehensively assessing the severity of facial paralysis. 4. Ignoring the cost of asymmetric misdiagnosis: Conventional classification models use a uniform probability threshold of 0.5, failing to consider that the clinical cost of misdiagnosing a severe case as a mild case is far higher than that of misdiagnosing a mild case as a severe case, leading to high-risk missed diagnoses. In recent years, deep learning-based facial paralysis assessment methods have emerged (such as video CNN and late-stage fusion), but most are still single-modal or simple feature splicing, failing to achieve deep alignment and complementary fusion of multimodal features. Therefore, developing an objective, dynamic, multimodal fusion-based intelligent facial paralysis grading method that fully considers the cost of clinical misdiagnosis has significant clinical value and social benefits. Summary of the Invention

[0003] Purpose of the invention To address the shortcomings of existing technologies, this invention provides a dynamic grading method and system for facial paralysis based on multimodal data and graph convolutional networks. The aim is to achieve an objective, dynamic, and accurate quantitative assessment of the severity of facial paralysis, improve the accuracy of grading and clinical consistency, and reduce the risk of missed diagnosis of severe cases.

[0004] Technical solution In a first aspect, the present invention provides a dynamic classification method for facial paralysis based on multimodal data and graph convolutional networks, comprising the following steps: Step S1: Multimodal data acquisition and preprocessing Simultaneously, dynamic videos of patients performing standardized facial movements, facial infrared thermal images, and clinical text data containing information from the four diagnostic methods of Traditional Chinese Medicine are collected; facial key point detection and tracking are performed on the dynamic videos to extract dynamic motion trajectories; facial region segmentation is performed on the infrared thermal images to calculate the temperature and left-right asymmetry index of each region; entity recognition and feature encoding are performed on the clinical text data to extract TCM syndrome feature vectors. Step S2: Construction of cross-modal feature maps The video features, infrared features, and text features extracted in step S1 are used as three types of nodes in the heterogeneous graph neural network to construct a heterogeneous graph containing intramodal self-connected edges and intermodal fully connected edges. Step S3: Cross-modal feature fusion based on graph convolutional networks The heterogeneous graph is input into a multi-layer graph convolutional network for neighborhood aggregation and feature update, outputting a unified cross-modal fusion feature vector, and adaptively weighted fusion of video features, infrared features and text features is performed through a cross-modal attention mechanism. Step S4: Multi-task joint learning and hierarchical classification The cross-modal fusion feature vector is input into a multi-task learning module, which includes a classification head for outputting static grading labels (HB I-VI levels) for facial paralysis and a regression head for outputting quantified dynamic functional scores (0-100 points), and performs joint optimization based on the weighted sum of classification loss and regression loss. Step S5: Optimization of Asymmetric Misdiagnosis Costs An asymmetric misdiagnosis cost matrix is ​​constructed, in which the cost of misdiagnosing a severe case as a mild case is higher than the cost of misdiagnosing a mild case as a severe case. Based on the cost matrix, the optimal classification decision threshold for each grade category is searched using a Bayesian optimization method. Step S6: Results Output and Interpretability Analysis Output static HB classification and dynamic score, and calculate the contribution of each input feature to the classification result based on the SHAP method to generate a feature importance ranking map. Secondly, the present invention provides a system for implementing the above method, comprising: 1. Data acquisition module: high-definition camera, infrared thermal imager, and clinical text input interface; 2. Feature extraction module: video feature extraction unit, infrared feature extraction unit, and text feature extraction unit; 3. Graph Construction and Fusion Module: Constructs heterogeneous graphs based on graph convolutional networks and aligns and fuses multimodal features; 4. Multi-task rating module: Outputs static HB rating labels and dynamic function scores; 5. Threshold optimization and decision-making module: Stores the asymmetric misdiagnosis cost matrix and the optimal classification threshold; 6. Interpretive Analysis Module: Generates interpretations of feature contribution based on the SHAP method; 7. Human-computer interaction and visualization module: used to guide patients to perform standardized actions and display the grading results.

[0005] Beneficial effects 1. High accuracy and high consistency: Through multimodal graph convolution fusion, the grading accuracy reaches 91.6%, which is 23.6% higher than the traditional HB grading method; the Kappa coefficient of consistency with expert diagnosis reaches 0.85, which is significantly better than existing technologies. 2. Dynamic quantitative assessment: It can quantitatively analyze the temporal characteristics of facial muscle movements, such as speed (accuracy 0.1mm) and amplitude, and output continuous dynamic scores (MAE=4.3), providing objective basis for monitoring the rehabilitation process. 3. Multimodal complementarity: For the first time, infrared thermal imaging (temperature asymmetry in 6 regions) and TCM clinical text (32-dimensional syndrome features extracted by RoBERTa) were incorporated into the facial paralysis assessment model. The complementarity between the modes was verified by the Bhattacharyya distance (dynamic 0.78, infrared 0.62, text 0.55). 4. Reduce high-risk missed diagnoses: Through asymmetric cost-sensitive learning (severe → mild cost 5.0, mild → severe cost 1.0), the total misdiagnosis cost is reduced by 34%, effectively reducing the risk of severe facial paralysis being misdiagnosed as mild. 5. High interpretability: Combined with the SHAP method, it can intuitively show the contribution of different modal features (such as "mouth corner displacement difference" and "forehead temperature difference") to the judgment results, enhancing clinical credibility. 6. Empowering primary healthcare: After deployment in county-level hospitals, the diagnostic accuracy rate increased from 68% to 85%, the misdiagnosis rate decreased by 32%, the average length of hospital stay was shortened by 1.5 days, and the full recovery rate increased by 40%. Attached Figure Description

[0006] Figure 1 This is an overall flowchart of the dynamic grading method for facial paralysis provided in this embodiment of the invention. Figure 2 This is a diagram of a cross-modal feature fusion network structure based on graph convolutional networks provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of multimodal data acquisition and preprocessing provided in an embodiment of the present invention. Figure 4 This is a system hardware and software architecture diagram provided in an embodiment of the present invention. Figure 5 This is an example of a SHAP feature importance waterfall plot provided in an embodiment of the present invention. Detailed Implementation

[0007] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. These embodiments are implemented based on the technical solutions of this invention, but the scope of protection of this invention is not limited to the following embodiments. Example 1: A dynamic grading method for facial paralysis based on multimodal data and graph convolutional networks like Figure 1 As shown, the method provided in this embodiment specifically includes the following steps: Step S1: Multimodal data acquisition and preprocessing Data from 1,824 patients with facial paralysis were collected at the Affiliated Hospital of Liaoning University of Traditional Chinese Medicine. Among them, 912 cases had complete multimodal data for primary analysis, and the rest were used for external validation. S1.1 Dynamic Video Acquisition and Feature Extraction The system guides patients to complete eight standardized facial movements: closing eyes, opening eyes, puffing out cheeks, showing teeth, raising eyebrows, wrinkling the nose, smiling, and pursing lips. Dynamic video is simultaneously captured using a 1080P / 30fps high-definition camera. The Dlib library is used to extract 68 2D facial key points from each frame, and the trajectory of these key points is tracked. The displacement amplitude, instantaneous velocity, and acceleration of each key point are calculated, and the dynamic asymmetry index of symmetrical areas (such as the left and right corners of the mouth and the left and right eye fissures) is also calculated. Specifically, the asymmetry index is calculated using the following formula: For each action, 68 temporal features were extracted. After dimensionality reduction through principal component analysis (preserving 95% variance), a 32-dimensional dynamic feature vector X was obtained. v . S1.2 Infrared Thermal Imaging Acquisition and Feature Extraction Facial thermal images of the patient were acquired using a FLIR T540 infrared thermal imager. Image segmentation algorithms were employed to divide the face into six regions of interest (ROIs): bilateral forehead, cheekbones, and perioral region. The average temperature and left-right temperature difference of each region were calculated to generate a 6-dimensional infrared feature vector X. i Clinical data shows that the temperature on the affected side of patients with severe facial paralysis is on average 0.6 ± 0.3℃ lower than that on the healthy side. S1.3 Clinical Text Acquisition and Feature Extraction Clinical text data from patients was collected, including tongue and pulse examinations, and descriptions of TCM syndromes (such as "qi and blood stagnation" and "meridian obstruction"). An improved RoBERTa model was used for feature encoding, which was pre-trained using 150,000 TCM medical records and ancient texts for domain-adaptive processing. After entity recognition and feature extraction, a 32-dimensional text feature vector X was obtained. t . S1.4 Multimodal Feature Standardization The three features were Z-score standardized to have zero mean and unit variance. The complementarity between the modalities was calculated using the Bhattacharyya distance: 0.78 for dynamic features, 0.62 for infrared features, and 0.55 for text features, indicating good complementarity among the three modalities. Step S2: Construction of cross-modal feature maps like Figure 2 As shown on the left, construct a heterogeneous graph. Node set These correspond to feature nodes for video, infrared, and text modalities, respectively. Each modality contains multiple nodes (e.g., 32 feature dimensions for the video modality). Edge set This includes: self-connected edges within the same modality based on k-nearest neighbors (k=10); and cross-modal edges constructed using a fully connected approach between different modalities. The initial node feature matrix H... (0) =[X v ;X i ;X t ]∈R 70 . Step S3: Cross-modal feature fusion based on graph convolutional networks The constructed heterogeneous graph is input into a two-layer graph convolutional network (GCN). The inter-layer propagation rule of GCN is as follows: Among them, A ~ =A+I N To add the adjacency matrix after adding self-loops, D ~ For A ~ The degree matrix, W (l) Let be the learnable weight matrix of the l-th layer, and σ be the ReLU activation function. After two layers of GCN, node features h with uniform dimensions are obtained. v ,h i ,h t ∈R 64 . Subsequently, a cross-modal attention mechanism is introduced to adaptively fuse node features from the three modalities: Where g is the global context vector (learnable parameters) and a is the attention parameter vector. The final fused feature h is obtained. fused ∈R 64 The entire GCN converged architecture is as follows: Figure 2 As shown. Step S4: Multi-task joint learning and hierarchical classification fusing feature h fused Input to both task headers simultaneously: Classification head: A fully connected layer followed by a Softmax function, used to predict static HB grading (6 classes from I to VI). The loss function is cross-entropy loss L CE . Regression head: A fully connected layer, used to predict the dynamic function score on a scale of 0 - 100. The loss function is mean squared error loss L MSE . The total loss function is: L = L CE + 0.5L MSE The model is trained end-to-end using the Adam optimizer (learning rate 0.001) and stratified 5-fold cross-validation, for a total of 200 epochs with a batch size of 32. Step S5: Asymmetric misdiagnosis cost optimization Construct a 6×6 asymmetric cost matrix C, where: The diagonal elements C ii = 0 (correct classification); The off-diagonal elements satisfy: The cost of misjudging a high grade (severe) as a low grade (mild) is greater than the reverse misjudgment, and the cost is proportional to the absolute value of the grade difference. Specifically, set C ij = ∣i − j∣×w ij , where w ij = 5.0 when i > j (severe misjudged as mild), w ij = 1.0 when i < j (mild misjudged as severe). For example, the cost of misjudging grade V as grade II is ∣5 − 2∣×5.0 = 15 Use the TPE sampler in the Optuna framework to search for the optimal classification threshold vector τ ∈ [0.05, 0.35] on the validation set 6 , with the goal of minimizing the expected misdiagnosis cost on the validation set: After about 300 evaluations, it converges, and the optimal thresholds for each level are obtained: The thresholds for grades V and VI are reduced from the default 0.17 to 0.09 and 0.08 respectively, effectively reducing the missed diagnosis rate of severe cases. Step S6: Result output and interpretability analysis The final grading results and dynamic scores are presented through the front-end interface. At the same time, use TreeSHAP to calculate the Shapley values of each input feature (original 70 dimensions), and generate a waterfall chart of feature importance rankings. In the typical output of severe facial paralysis cases, the top 5 features with the greatest contribution are: mouth corner displacement difference (+0.24), perioral infrared temperature difference (+0.18), eye closure speed (+0.12), zygomatic displacement ratio (+0.09), and the text feature "qi and blood stagnation" (+0.07). Example 2: A dynamic grading system for facial paralysis based on multimodal data and graph convolutional networks like Figure 4 As shown, the system provided in this embodiment includes the following modules: 1. Data Acquisition Module: Includes a 1080P high-definition camera, a FLIR T540 infrared thermal imager, a clinical text input interface, and a set of action guidance instructions. Used for synchronous acquisition of dynamic video, infrared thermal images, and TCM clinical text. 2. Data Preprocessing and Feature Extraction Module: Video feature extraction unit: Based on Dlib's 68-point keypoint tracking, it calculates displacement, velocity, acceleration and asymmetry index, and outputs 32-dimensional dynamic features. Infrared feature extraction unit: Calculates the average temperature and temperature difference of 6 regions based on ROI segmentation, and outputs 6-dimensional infrared features. Text feature extraction unit: Based on the improved RoBERTa model (pre-trained with 150,000 TCM corpora), it outputs 32-dimensional text features. 3. Graph Construction and GCN Cross-Modal Fusion Module: Constructs a heterogeneous graph (modal internal edges with k=10 + fully connected intermodal edges), performs two layers of GCN feature propagation and attention fusion, and outputs 64-dimensional cross-modal fusion features. 4. Multi-task classification module: pre-trained classification head (6-class HB classification) and regression head (0-100 dynamic score), output static grade and dynamic score. 5. Cost-Sensitive Calibration Module: Stores the asymmetric cost matrix and the optimal threshold vector obtained by Optuna optimization, used to calibrate the original classification probability. 6. Interpretability Module: TreeSHAP computation unit, which generates global and local feature contribution interpretations (waterfall plot, feature ranking). 7. Human-computer interaction interface: Built on Django + Vue.js, it is used for patient action guidance, data collection quality feedback, result visualization and remote collaboration.

[0008] Experimental verification Five-fold cross-validation was performed on 912 complete multimodal datasets, and the results compared with existing methods are shown in the table below: Model accuracy Macro F1 Kappa Dynamic MAE Total cost Primary physician 68.4% — 0.62 — 1243 SVM + Handcrafted Features 72.1% 0.68 0.65 12.7 892 CNN video 85.2% 0.81 0.78 8.4 567 Late fusion 88.6% 0.84 0.81 6.9 478 This invention 91.6% 0.87 0.85 4.3 312 In a deployment test involving 215 patients in three county-level hospitals, the diagnostic accuracy increased from 68% to 85%, and the misdiagnosis rate decreased by 32%. The complete recovery rate of the system-assisted group increased by 40% (45%→63%) compared with the historical control group, and the average length of hospital stay was shortened by 1.5 days (12.3→10.8 days, p=0.02).

[0009] Other implementation methods The method and system described in this invention are not only applicable to the dynamic grading of facial paralysis, but can also be applied to the assessment of other motor dysfunction diseases, such as facial motor assessment of Parkinson's disease and facial function assessment of post-stroke sequelae, only requiring corresponding adjustments to the standardized action set and grading criteria. The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A dynamic grading method for facial paralysis based on multimodal data and graph convolutional networks, characterized in that, Includes the following steps: Step S1: Multimodal Data Acquisition and Preprocessing: Simultaneously acquire dynamic videos of patients performing standardized facial movements, facial infrared thermal images, and clinical text data containing information from the four diagnostic methods of Traditional Chinese Medicine (TCM); perform facial key point detection and tracking on the dynamic videos to extract dynamic motion trajectories; perform facial region segmentation on the infrared thermal images to calculate the temperature and left-right asymmetry index of each region; perform entity recognition and feature encoding on the clinical text data to extract TCM syndrome feature vectors; Step S2: Cross-modal Feature Map Construction: Use the video features, infrared features, and text features extracted in Step S1 as three types of nodes in a heterogeneous graph neural network to construct a heterogeneous graph containing intramodal self-connected edges and intermodal fully connected edges; Step S3: Cross-modal Feature Fusion Based on Graph Convolutional Networks: Input the heterogeneous graph into a multi-layer graph convolutional network. The process involves neighborhood aggregation and feature updating to output a unified cross-modal fusion feature vector. An adaptive weighted fusion of video, infrared, and text features is performed using a cross-modal attention mechanism. Step S4: Multi-task joint learning and grading: The cross-modal fusion feature vector is input into a multi-task learning module. This module includes a classification head for outputting static grading labels for facial paralysis and a regression head for outputting quantified dynamic functional scores. Joint optimization is performed based on the weighted sum of classification and regression losses. Step S5: Asymmetric misdiagnosis cost optimization: An asymmetric misdiagnosis cost matrix is ​​constructed, where the cost of misdiagnosing a severe case as a mild case is higher than the cost of misdiagnosing a mild case as a severe case. Based on this cost matrix, the optimal classification decision threshold for each grading category is searched using a Bayesian optimization method.

2. The method according to claim 1, characterized in that, In step S1, 68 2D key points of the face are extracted using the Dlib library, and the displacement, velocity and acceleration of the key points over time are calculated. An asymmetry index is calculated based on facial symmetry as the dynamic motion trajectory. The standardized facial movements include at least one of the following: closing eyes, opening eyes, puffing out cheeks, showing teeth, raising eyebrows, wrinkling nose, smiling, and pouting.

3. The method according to claim 1, characterized in that, In step S1, the improved RoBERTa model is used to encode the clinical text data. The improved RoBERTa model is pre-trained using domain-adaptive corpora of traditional Chinese medicine medical records and ancient texts.

4. The method according to claim 1, characterized in that, In step S1, when segmenting the facial region of the infrared thermal image, six regions of interest are divided into bilateral forehead, cheekbone, and perioral area. The average temperature of each region and the temperature difference between the left and right sides are calculated to form a 6-dimensional infrared feature vector.

5. The method according to claim 1, characterized in that, The inter-layer propagation rule of the graph convolutional network in step S3 is as follows: Among them, H (0) A is the initial feature of the spliced ​​multimodal system. ~ To add the adjacency matrix after adding self-loops, D ~ For A ~ The degree matrix, W (l) Let σ be the learnable weight matrix, and σ be the non-linear activation function.

6. The method according to claim 1, characterized in that, The adaptive weighted fusion calculation formula for the cross-modal attention mechanism in step S3 is as follows: Where g is the global context vector, a is the attention parameter vector, and h v ,h i ,h t These are the video, infrared, and text modal features updated by the graph convolutional network, respectively, h. fused This is the fused feature vector.

7. The method according to claim 1, characterized in that, In step S4, the loss function for the classification head is the cross-entropy loss L. CE The loss function for regressing the head is the mean squared error loss L. MSE The total loss function is L=L CE +λL MSE , where λ is the weighting coefficient for balancing the two tasks, with a value of 0.

5.

8. The method according to claim 1, characterized in that, In the said step S5, the element C of the asymmetric misdiagnosis cost matrix ij satisfies that when i > j, that is, a severe case is misjudged as a mild case, C ij = |i - j| × w heavy ; when i < j, that is, a mild case is misjudged as a severe case, C ij = |i - j| × w light ; and w heavy > w light ; preferably, w heavy = 5.0, w light = 1.

0.

9. The method according to claim 1, characterized in that, In step S5, Bayesian optimization is performed using the TPE sampler from the Optuna framework to minimize the expected misdiagnosis cost on the validation set. With the objective of [0.05, 0.35], the optimal classification threshold vector τ∈[0.05, 0.35] is obtained through searching. 6 .

10. The method according to claim 1, characterized in that, It also includes step S6: Result output and interpretability analysis: Output static HB classification and dynamic score, and calculate the Shapley value of each input feature based on the TreeSHAP method to generate a feature importance ranking map or waterfall plot.

11. A dynamic grading system for facial paralysis based on multimodal data and graph convolutional networks, used to execute the method according to any one of claims 1 to 10, characterized in that, include: The data acquisition module includes a high-definition camera, an infrared thermal imager, and a clinical text input interface, used to simultaneously acquire dynamic video, infrared thermal images, and clinical text data; the feature extraction module includes a video feature extraction unit, an infrared feature extraction unit, and a text feature extraction unit, used to extract dynamic motion trajectories, facial temperature asymmetry, and TCM syndrome feature vectors, respectively; the graph construction and fusion module constructs heterogeneous graphs based on graph convolutional networks and performs spatial alignment and fusion of multimodal features, outputting cross-modal fused feature vectors. Multi-task grading module: Based on the cross-modal fusion feature vector, outputs static HB grading labels and dynamic functional scores; Threshold optimization and decision module: Stores a pre-generated asymmetric misdiagnosis cost matrix and optimal classification decision threshold, used to calibrate the output of the multi-task grading module; Human-computer interaction and visualization module: Used to guide patients to perform standardized actions, provide real-time feedback on acquisition quality, and display grading results, dynamic scores, and key criterion features in a visual manner.

12. The system according to claim 11, characterized in that, It also includes an interpretability analysis module, which is used to calculate and display the contribution of each modal feature to the classification results based on the SHAP method, and generate feature importance ranking or waterfall plot.

13. The system according to claim 11, characterized in that, The high-definition camera in the data acquisition module is 1080P / 30fps, and the infrared thermal imager is a FLIR T540 or a device with equivalent parameters; the standardized facial movements include eight movements: closing the eyes, opening the eyes, puffing out the cheeks, showing teeth, raising the eyebrows, wrinkling the nose, smiling, and pouting.

14. The system according to claim 11, characterized in that, The human-computer interaction and visualization module is built on the Django+Vue.js framework and supports deployment in clinic terminals and access to remote cloud platforms.

Images