Facial paralysis rehabilitation evaluation and treatment method and system based on multi-modal data fusion

Abstract

The invention relates to the technical field of treatment, and relates to a facial paralysis rehabilitation evaluation and treatment method and system based on multi-modal data fusion, and the method comprises the steps: collecting an electromyographic signal, a facial video of a static expression of a patient, and a facial video of a dynamic expression of the patient, and obtaining video data and electromyographic data; detecting face key feature points of the video data by using a pre-constructed face detection model, and generating a face mark graph and feature mark data; inputting the feature mark data and the myoelectricity data into a pre-constructed data fusion model, and outputting comprehensive data; calculating facial paralysis score data according to the facial paralysis scale and the comprehensive data; collecting moxibustion acupoint data in the moxibustion acupoint database, inputting the facial paralysis score data and the moxibustion acupoint data into a pre-constructed moxibustion point location optimization model, and outputting a moxibustion point location scheme. By integrating the face video data and the electromyographic signals, accurate evaluation of the facial nerve function and the muscle activity of the patient is achieved.

Description

Technical Field

[0001] The present application relates to the field of therapeutic technology, and in particular to a facial paralysis rehabilitation assessment and treatment method and system based on multimodal data fusion. Background Art

[0002] Facial paralysis is a neurological disease that affects facial muscle movement, often leading to partial or complete loss of facial expression muscles, which has a significant impact on the patient's life and psychology. Existing facial paralysis assessment methods are generally subjective vision (photos and videos) and electromyography. Subjective vision, for example, a Chinese patent with a publication number of CN118710644B, involves the field of image processing technology, specifically a method and system for detecting the efficacy of acupuncture for patients with facial paralysis based on image processing; the method includes: obtaining an optical flow difference feature value corresponding to a feature image in a facial video, a feature sequence, and a preamble subsequence corresponding to a feature image in the feature sequence, and obtaining an optimization factor corresponding to the feature image according to the preamble subsequence corresponding to the feature image in the feature sequence and the optical flow difference feature value corresponding to each feature image in the preamble subsequence; using the optimization factor corresponding to the feature image, optimizing the optical flow difference feature value corresponding to the feature image in the feature sequence, and obtaining the optimized optical flow difference feature value corresponding to the feature image in the feature sequence; obtaining the efficacy evaluation result of the facial paralysis patient after acupuncture treatment according to the optimized optical flow difference feature value; the invention can improve the accuracy of obtaining the efficacy evaluation result of the facial paralysis patient after acupuncture treatment. However, the accuracy of subjective vision is questionable. Due to the particularity of facial paralysis, electromyography mainly evaluates the electrical activity of the muscles, but does not directly reflect the functional state of the muscles or the naturalness of facial expressions. The recovery of electromyography signals does not necessarily mean complete recovery of facial function or aesthetics. In summary, a single evaluation method cannot bring about an accurate treatment plan suitable for the patient. Summary of the invention

[0003] In order to solve the above problems, an embodiment of the present invention provides a facial paralysis rehabilitation assessment and treatment method based on multimodal data fusion, the method comprising:

[0004] Collect electromyographic signals, facial videos of patients with static expressions, and facial videos of patients with dynamic expressions to obtain video data and electromyographic data;

[0005] Use the pre-built facial detection model to detect key facial feature points on the video data and generate facial markers and feature marker data;

[0006] Input the feature-marked data and electromyographic data into a pre-built data fusion model and output the integrated data;

[0007] The facial paralysis score data was calculated based on the facial paralysis scale and comprehensive data;

[0008] The moxibustion point data in the moxibustion point database are collected, the facial paralysis score data and the moxibustion point data are input into the pre-built moxibustion point optimization model, and the moxibustion point plan is output.

[0009] Furthermore, the face detection model includes a feature extraction model and a Kalman filter smoothing;

[0010] The training method of the feature extraction model includes:

[0011] All video data are used as input of a feature extraction model, the feature extraction model takes the key point coordinates and key point heat map corresponding to each group of video data as output, takes the actual key point coordinates and key point heat map corresponding to each group of video data as prediction targets, and takes minimizing the sum of the first prediction accuracies of all predicted key point coordinates and key point heat maps as a training target; the feature extraction model is trained until the sum of the first prediction accuracies reaches convergence and the training is stopped; the feature extraction model is a convolutional neural network model.

[0012] Furthermore, the calculation method of the Kalman filter smoothing includes:

[0013] Initialize the state vector and covariance matrix according to the key point coordinates and video data to obtain the initial state vector X o and the initial covariance matrix P 0 :

[0014] From the initial state vector X o and the initial covariance matrix P 0 Initially, recursive calculations are performed according to the prediction formula and update formula of the Kalman filter to obtain a set of key facial feature points.

[0015] Furthermore, the data fusion model includes an input layer, a fusion layer and an output layer; the fusion layer includes a modality alignment unit and a feature fusion unit;

[0016] Modal Alignment Unit:

[0017] Let the time frame of the feature label data be T v , the time frame of the EMG data is T e ;

[0018] If T v <T e , then the electromyographic signal is downsampled;

[0019] If T v >T e , then interpolate the video frame;

[0020] Feature fusion unit;

[0021] The feature-labeled data is used as the query and the electromyographic data as the key and value; the attention mechanism algorithm is used to obtain the comprehensive data F.

[0022] Furthermore, the calculation method of facial paralysis score data includes:

[0023] The score obtained from the feature-labeled data in the facial paralysis scale is set as the geometric feature score S geo , calculate the comprehensive data and S geo The Pearson correlation coefficient S sync ;

[0024] Pearson correlation coefficient S sync The calculation formula includes:

[0025]

[0026] In the formula, b is the comprehensive data, c is the geometric feature score; i is the group number of the comprehensive data and geometric feature score; n is the total number;

[0027] Facial paralysis scoring data total The calculation formula includes:

[0028] S total =w 1 ·S geo +w 2 ·S sync ;

[0029] In the formula, w 1 For S geo The preset weight, w 2 For S sync The preset weights.

[0030] The training method of the moxibustion point optimization model includes:

[0031] All facial paralysis scoring data and moxibustion acupoint data are used as inputs of a moxibustion point optimization model; the moxibustion point optimization model uses the moxibustion point scheme corresponding to each group of facial paralysis scoring data and moxibustion acupoint data as output, uses the actual moxibustion point scheme corresponding to each group of facial paralysis scoring data and moxibustion acupoint data as a prediction target, and uses minimizing the sum of the second prediction accuracies of all predicted moxibustion point schemes as a training target; the moxibustion point optimization model is trained until the sum of the second prediction accuracies converges, and the training is stopped; the moxibustion point optimization model is a convolutional neural network model.

[0032] Facial paralysis rehabilitation assessment and treatment system based on multimodal data fusion, the system includes:

[0033] A collection module, the collection module is used to collect electromyographic signals, facial videos of patients with static expressions, and facial videos of patients with dynamic expressions, to obtain video data and electromyographic data;

[0034] A marking module, the marking module is used to detect key facial feature points on the video data using a pre-built facial detection model, and generate a facial marking map and feature marking data;

[0035] A data fusion module, wherein the data fusion module is used to input the feature tag data and the electromyographic data into a pre-built data fusion model and output comprehensive data;

[0036] A comprehensive evaluation module, which is used to calculate facial paralysis scoring data based on the facial paralysis scale and comprehensive data;

[0037] A pre-treatment module is used to collect moxibustion point data in a moxibustion point database, input facial paralysis score data and moxibustion point data into a pre-built moxibustion point optimization model, and output a moxibustion point plan.

[0038] The technical effects and advantages of the facial paralysis rehabilitation assessment and treatment method and system based on multimodal data fusion provided by the present invention are as follows:

[0039] The present invention integrates facial video data, electromyographic signals, facial paralysis scoring scales and moxibustion acupoint data to achieve accurate assessment of patients' facial nerve function and muscle activity, and designs the most suitable treatment plan for each patient based on these data; Kalman filtering is used to smooth and eliminate the influence of unnaturalness or twitching of patients with facial paralysis, thereby improving the accuracy of facial recognition; multimodal data (video, electromyographic signals) fusion is used, combined with facial paralysis scales and moxibustion acupoint databases, to accurately assess patients' facial nerve function and muscle activity, provide a comprehensive facial paralysis severity score, and help doctors identify the condition more accurately; through the moxibustion point optimization model, moxibustion points are personalized recommended according to facial paralysis scoring data, and the most appropriate moxibustion points are selected for treatment, thereby increasing the pertinence and effectiveness of the treatment; data fusion models and machine learning algorithms are used to provide data-driven decision support, making treatment decisions more scientific and accurate, and reducing the risks of misdiagnosis and improper treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a connection diagram of the facial paralysis rehabilitation assessment and treatment system based on multimodal data fusion in Example 1;

[0041] Figure 2 This is a schematic diagram of the facial detection model connection in Example 1;

[0042] Figure 3 This is a flow chart of the facial paralysis rehabilitation assessment and treatment method based on multimodal data fusion in Example 2. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Embodiment 1:

[0045] See also Figure 1 As shown, the facial paralysis rehabilitation assessment and treatment method based on multimodal data fusion described in this embodiment includes:

[0046] Collect electromyographic signals, facial videos of patients with static expressions, and facial videos of patients with dynamic expressions to obtain video data and electromyographic data;

[0047] Use the pre-built facial detection model to detect key facial feature points on the video data and generate facial markers and feature marker data;

[0048] Input the feature-marked data and electromyographic data into a pre-built data fusion model and output the integrated data;

[0049] The facial paralysis score data was calculated based on the facial paralysis scale and comprehensive data;

[0050] The facial paralysis score data and the moxibustion point data in the moxibustion point database are input into the pre-built moxibustion point optimization model, and the moxibustion point plan is output.

[0051] Sensors are placed on the frontalis, orbicularis oculi, and perioral muscles to collect electromyographic signals during different facial expressions, thereby obtaining electromyographic data and recording time frames at the same time. Facial videos are collected using a combination of static and dynamic methods, and video frames (time frames) are recorded at the same time. Facial expressions include static rest, closing eyes, raising eyebrows, frowning, raising nose, pouting lips, showing teeth, and puffing cheeks. The collection of facial videos is synchronized with the time of collecting electromyographic signals.

[0052] like Figure 2 As shown, the face detection model includes a feature extraction model and a Kalman filter smoothing;

[0053] The training methods for extracting feature models include:

[0054] All video data are used as input of the feature extraction model, and the feature extraction model takes the key point coordinates and key point heat map corresponding to each group of video data as output, the actual key point coordinates and key point heat map corresponding to each group of video data as prediction targets, and minimizing the sum of the first prediction accuracies of all predicted key point coordinates and key point heat maps as training target.

[0055] Among them, the calculation formula for the first prediction accuracy is: JE g =(JY g -JR g ) 2 , where g is the number of each set of video data, JE g is the first prediction accuracy, JY g is the predicted key point coordinates and key point heat map corresponding to the g-th group of video data, JR g The actual key point coordinates and key point heat map corresponding to the g-th group of video data; the feature extraction model is trained until the sum of the first prediction accuracy reaches convergence and the training is stopped; the feature extraction model is a convolutional neural network model.

[0056] The feature extraction model is used to extract the features of a single frame image in the video data, generate a key point heat map, each heat map corresponds to a key point, and then extract the key point candidate coordinates from the key point heat map; the convolutional neural network (CNN) is combined with other methods to improve the performance and robustness of facial feature point detection.

[0057] The calculation method of Kalman filter smoothing includes:

[0058] Initialize the state vector and covariance matrix according to the key point coordinates and video data to obtain the initial state vector X o and the initial covariance matrix P 0 :

[0059]

[0060] Where x and y are the coordinates of the key point (horizontal and vertical coordinates); v is the velocity.

[0061] From the initial state vector X o and the initial covariance matrix P 0 First, recursive calculation is performed according to the prediction formula and update formula of Kalman filter to obtain the key point trajectory, that is, the set of key feature points of the face;

[0062] The calculation method of the prediction formula includes:

[0063]

[0064] In the formula, is the current frame state predicted based on the previous frame state; A is the state transfer matrix, which is used to describe the motion relationship of the system from the previous frame to the current frame; t is the frame number, indicating the tth frame; is the state covariance of the current frame predicted based on the state covariance of the previous frame; Q is the process noise covariance matrix, which describes the uncertainty of the feature extraction model and is usually set to a diagonal matrix. The larger the value, the less reliable the feature extraction model.

[0065] The calculation method of the update formula includes:

[0066]

[0067] In the formula, K t is the Kalman gain, which represents the observed value (key point coordinates) of the current frame to the predicted value The degree of correction; t is the observation value of the current frame; H is the observation matrix, which maps the state to the observation value space, and only the key point coordinates are observed, and the speed is not observed; R is the observed noise covariance matrix, which describes the uncertainty of the observation value; is the correction term, the observed value is combined with the predicted value, and the corrected state is output; I is the unit matrix; X t For status update; P t is the covariance update.

[0068] Kalman filtering can effectively remove random noise from facial key point position data and provide smooth key point dynamic trajectories, which is especially important for facial movement analysis of patients with facial paralysis because their facial movements may be unnatural or twitchy.

[0069] The feature marking data includes key point coordinates, key point heat maps and a set of facial key feature points; when the facial key feature points are marked on a frame of video data according to the key point coordinates, the picture is now a facial marking map.

[0070] The data fusion model includes input layer, fusion layer and output layer; the fusion layer includes modality alignment unit and feature fusion unit;

[0071] Modal Alignment Unit:

[0072] Let the time frame of the feature label data be T v , the time frame of the EMG data is T e ;

[0073] If T v <T e , then the electromyographic signal is downsampled;

[0074] If T v >T e , the video frames are interpolated.

[0075] Exemplary:

[0076] If the video frame rate is 30fps, the video has 30 time frames in 1 second; if the EMG signal sampling rate is 1kHz, the EMG signal has 1000 time frames in 1 second; in summary, T v <T e , down-sample the EMG signal.

[0077] The modality alignment unit is designed to support the alignment of asynchronous modalities and be compatible with different sampling rates.

[0078] Feature fusion unit;

[0079] Take the feature-labeled data as the query and the electromyographic data as the key and value; use the attention mechanism algorithm to obtain the comprehensive data F;

[0080] The calculation method of the attention mechanism algorithm includes:

[0081] F = softmax(W Q F v (W k F E ) T )W v F E ;

[0082] Where W Q , W k , W v is the weight matrix of query, key and value; softmax() is the normalized exponential function.

[0083] The feature fusion unit can effectively capture the dynamic correlation between video and electromyographic signals, making the calculation of facial paralysis score more accurate and more in line with the patient's actual nerve damage.

[0084] The facial paralysis scale is an authoritative score rating scale based on the activity data of key facial feature points obtained through a large amount of actual measurement. The facial paralysis scale includes the eyebrow ptosis scale, eyelid closure scale, mouth corner ptosis scale and philtrum deviation scale.

[0085] Eyebrow ptosis rating scale:

[0086] score state 0 points Eyebrow droop angle 0° is normal 1 point Eyebrow droop angle 0-5° 2 points Eyebrow droop angle 5-10° 3 points Eyebrow droop angle 10-15° 4 points Eyebrow droop angle 15-20° 5 points Eyebrow droop angle more than 20°

[0087] Eyelid closure scale:

[0088] score state 0 points Normal closure of upper and lower eyelids 1 point The upper and lower eyelids can barely close, but the speed is slow and the closure is not tight 2 points The upper and lower eyelids cannot close completely, and the white of the eye is exposed to 1 / 5 of the palpebral fissure 3 points The upper and lower eyelids cannot close completely, and the white of the eye is exposed to 2 / 5 of the palpebral fissure 4 points The upper and lower eyelids cannot close completely, and the white of the eye is exposed to 3 / 5 of the palpebral fissure 5 points The upper and lower eyelids cannot close completely, and the white of the eye is exposed to 4 / 5 of the palpebral fissure

[0089] Mouth corner ptosis grading scale:

[0090] score state 0 points The angle of the mouth corner is 0°, which is normal 1 point Mouth corner droop angle 0-5° 2 points The angle of mouth corner droop is 5-10° 3 points The angle of mouth corner droop is 10-15° 4 points The angle of mouth corner droop is 15-20° 5 points The angle of the mouth corner is more than 20°

[0091] Philtrum deviation grading scale:

[0092] score state 0 points The philtrum deflection angle is 0°, which is normal. 1 point The angle of philtrum deviation is 0-10° 2 points The angle of philtrum deviation is 10-20° 3 points The angle of the philtrum is 20-30° 4 points The angle of the philtrum is 30-40° 5 points The philtrum is deviated by more than 40°

[0093] The calculation method of facial paralysis score data includes:

[0094] The score obtained from the feature labeling data in the facial paralysis scale is set as the geometric feature score S geo , calculate the comprehensive data and S geo The Pearson correlation coefficient S sync , the Pearson correlation coefficient range is [-1.1], a Pearson correlation coefficient of -1 means a completely negative correlation, a Pearson correlation coefficient of 1 means a completely positive correlation, and a Pearson correlation coefficient of 0 means S sync With S geo There is no linear relationship between them.

[0095] Pearson correlation coefficient S sync The calculation formula includes:

[0096]

[0097] Where b is the comprehensive data, c is the geometric feature score, i is the group number of the comprehensive data and the geometric feature score, and n is the total number.

[0098] The Pearson correlation coefficient was used to calculate the correlation between the activity of key facial feature points and the electromyographic data, and the coordination of the patient's muscle control ability and facial expressions was evaluated, which helps to understand the extent of the impact of facial paralysis and its impact on daily expression ability. The Pearson correlation coefficient was used to calculate the correlation between the activity of key facial feature points and the electromyographic data, and the coordination of the patient's muscle control ability and facial expressions was evaluated, which helps to understand the extent of the impact of facial paralysis and its impact on daily expression ability.

[0099] Facial paralysis scoring data total The calculation formula includes:

[0100] S total =w 1 ·S geo +w 2 ·S sync ;

[0101] In the formula, w 1 For S geo The weight, w 2 For S sync The weight of .

[0102] Combining the comprehensive data with the facial paralysis scale score can provide personalized treatment recommendations for individual facial paralysis conditions. For example, exercises or treatments to strengthen the orbicularis oculi muscle may be recommended for patients with poor closure ability.

[0103] The Moxibustion Point Database is a dataset containing various moxibustion points and their related information (such as location, function, indications, etc.). The Moxibustion Point Database is constructed based on traditional Chinese medicine knowledge and modern medical research to support moxibustion-based treatment and research.

[0104] The training methods of the moxibustion point optimization model include:

[0105] All facial paralysis score data and moxibustion acupoint data are used as the input of the moxibustion point optimization model. The moxibustion point optimization model takes the moxibustion point plan corresponding to each group of facial paralysis score data and moxibustion acupoint data as output, and the actual moxibustion point plan corresponding to each group of facial paralysis score data and moxibustion acupoint data as the prediction target. Minimizing the sum of the second prediction accuracies of all predicted moxibustion point plans is used as the training goal.

[0106] Among them, the calculation formula for the second prediction accuracy is: Fs R =(Fg R -Fy R ) 2 , where R is the number of each group of facial paralysis score data and moxibustion acupoint data, Fs R is the second prediction accuracy, Fg R is the predicted moxibustion point scheme corresponding to the facial paralysis score data and moxibustion acupoint data of group R, Fy R The actual moxibustion point scheme corresponding to the facial paralysis score data and the moxibustion acupoint data of the R group is provided; the moxibustion point optimization model is trained until the sum of the second prediction accuracy reaches convergence and the training is stopped; the moxibustion point optimization model is a convolutional neural network model.

[0107] The moxibustion point plan is a set of specific moxibustion treatment points obtained after processing with an optimization algorithm based on facial paralysis score data and moxibustion point database information. These point plans are specially selected to maximize the treatment effect, usually taking into account the patient's specific facial paralysis symptoms and detailed information on muscle activity; these points can be selected specifically to improve facial function in specific areas or alleviate specific symptoms, and can generally be used in conjunction with facial moxibustion devices.

[0108] Embodiment 2:

[0109] like Figure 3As shown, based on the same inventive concept as the facial paralysis rehabilitation assessment and treatment method based on multimodal data fusion in the aforementioned embodiment, the present application provides a facial paralysis rehabilitation assessment and treatment system based on multimodal data fusion, and the method and system embodiments in the embodiments of the present application are based on the same inventive concept. The system includes:

[0110] A collection module is used to collect electromyographic signals, facial videos of patients with static expressions, and facial videos of patients with dynamic expressions, to obtain video data and electromyographic data;

[0111] A labeling module, used to detect key facial feature points on the video data using a pre-built facial detection model, and generate a facial labeling map and feature labeling data;

[0112] A data fusion module is used to input the feature labeling data and the electromyographic data into a pre-built data fusion model and output the integrated data;

[0113] A comprehensive evaluation module is used to calculate facial paralysis score data based on the facial paralysis scale and comprehensive data;

[0114] The pre-treatment module is used to collect moxibustion point data in the moxibustion point database, input the facial paralysis score data and the moxibustion point data into the pre-built moxibustion point optimization model, and output the moxibustion point plan.

[0115] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0116] What has been described above is only a preferred specific implementation manner of the embodiments of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can make equivalent substitutions or changes according to the technical scheme and concept of the present application within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A method for the rehabilitation assessment and treatment of facial paralysis based on multimodal data fusion, characterized in that: Methods include: Collect electromyographic signals, facial videos of patients with static expressions, and facial videos of patients with dynamic expressions to obtain video data and electromyographic data; Use the pre-built facial detection model to detect key facial feature points on the video data and generate facial markers and feature marker data; Input the feature-marked data and electromyographic data into a pre-built data fusion model and output the integrated data; The facial paralysis score data was calculated based on the facial paralysis scale and comprehensive data; The moxibustion point data in the moxibustion point database are collected, the facial paralysis score data and the moxibustion point data are input into the pre-built moxibustion point optimization model, and the moxibustion point plan is output.

2. The facial paralysis rehabilitation assessment and treatment method based on multimodal data fusion according to claim 1, characterized in that: The face detection model includes a feature extraction model and a Kalman filter smoothing; The training method of the feature extraction model includes: All video data are used as input of a feature extraction model, the feature extraction model takes the key point coordinates and key point heat map corresponding to each group of video data as output, takes the actual key point coordinates and key point heat map corresponding to each group of video data as prediction targets, and takes minimizing the sum of the first prediction accuracies of all predicted key point coordinates and key point heat maps as a training target; the feature extraction model is trained until the sum of the first prediction accuracies reaches convergence and the training is stopped; the feature extraction model is a convolutional neural network model.

3. The facial paralysis rehabilitation assessment and treatment method based on multimodal data fusion according to claim 2 is characterized in that: The calculation method of the Kalman filter smoothing includes: Initialize the state vector and covariance matrix according to the key point coordinates and video data to obtain the initial state vector X o And the initial covariance matrix P0: From the initial state vector X o Starting with the initial covariance matrix P0, recursive calculations are performed according to the prediction formula and update formula of the Kalman filter to obtain a set of facial key feature points.

4. The facial paralysis rehabilitation assessment and treatment method based on multimodal data fusion according to claim 1, characterized in that: The data fusion model includes an input layer, a fusion layer and an output layer; the fusion layer includes a modality alignment unit and a feature fusion unit; Modal Alignment Unit: Let the time frame of the feature label data be T v , the time frame of the EMG data is T e ; If T v <T e , then the electromyographic signal is downsampled; If T v >T e , then interpolate the video frame; Feature fusion unit; The feature-labeled data is used as the query and the electromyographic data as the key and value; the attention mechanism algorithm is used to obtain the comprehensive data F.

5. The facial paralysis rehabilitation assessment and treatment method based on multimodal data fusion according to claim 1, characterized in that: The calculation method of facial paralysis score data includes: The score obtained from the feature-labeled data in the facial paralysis scale is set as the geometric feature score S geo , calculate the comprehensive data and S geo The Pearson correlation coefficient S sync ; Pearson correlation coefficient S sync The calculation formula includes: In the formula, b is the comprehensive data, c is the geometric feature score; i is the group number of the comprehensive data and geometric feature score; n is the total number; Facial paralysis scoring data total The calculation formula includes: S total =w1·S geo +w2·S sync ; Where w1 is S geo The preset weight of S sync The preset weights.

6. The facial paralysis rehabilitation assessment and treatment method based on multimodal data fusion according to claim 1, characterized in that: The training method of the moxibustion point optimization model includes: All facial paralysis scoring data and moxibustion acupoint data are used as inputs of a moxibustion point optimization model; the moxibustion point optimization model uses the moxibustion point scheme corresponding to each group of facial paralysis scoring data and moxibustion acupoint data as output, uses the actual moxibustion point scheme corresponding to each group of facial paralysis scoring data and moxibustion acupoint data as a prediction target, and uses minimizing the sum of the second prediction accuracies of all predicted moxibustion point schemes as a training target; the moxibustion point optimization model is trained until the sum of the second prediction accuracies converges, and the training is stopped; the moxibustion point optimization model is a convolutional neural network model.

7. The facial paralysis rehabilitation assessment and treatment system based on multimodal data fusion is characterized by: A collection module, the collection module is used to collect electromyographic signals, facial videos of patients with static expressions, and facial videos of patients with dynamic expressions, to obtain video data and electromyographic data; A marking module, the marking module is used to detect key facial feature points on the video data using a pre-built facial detection model, and generate a facial marking map and feature marking data; A data fusion module, wherein the data fusion module is used to input the feature tag data and the electromyographic data into a pre-built data fusion model and output comprehensive data; A comprehensive evaluation module, which is used to calculate facial paralysis scoring data based on the facial paralysis scale and comprehensive data; A pre-treatment module is used to collect moxibustion point data in a moxibustion point database, input facial paralysis score data and moxibustion point data into a pre-built moxibustion point optimization model, and output a moxibustion point plan.

Images