Multi-modal facial nerve evaluation nerve physical examination method and system
By employing a multimodal assessment method, utilizing a head positioning bracket and YOLO-V8 key point recognition technology, combined with electromyography and video signal processing, the problems of subjective interference and incompleteness in facial nerve function assessment were solved, achieving efficient and reliable facial nerve assessment.
Patent Information
- Application Number
- CN202511769249.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-02-27
AI Technical Summary
Existing methods for assessing facial nerve function are susceptible to interference from doctors' subjective factors or are incomplete, failing to simultaneously capture the dynamic coordination of facial muscles.
A multimodal assessment method was adopted, in which the patient's head was fixed by a head positioning bracket, a demonstration video of the examination task was played, electromyographic signals and high-definition video signals were collected, key points were identified using YOLO-V8, and data processing was performed by combining convolutional neural networks and multilayer perceptron neural networks to output evaluation results.
It enables reliable assessment of facial nerve function, reduces assessment errors, improves assessment efficiency, meets privacy protection requirements, is compatible with common hardware, and is easy to promote in clinical practice.
Smart Images
Figure CN121570130A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of biomedical engineering, and in particular to a multimodal facial nerve assessment method and system. Background Technology
[0002] Currently, facial nerve function assessment mainly relies on two types of methods: subjective assessment using physicians (such as the House-Brackmann scale) and sensor-based detection, such as surface electromyography (EMG). However, both have significant limitations. Subjective scales are based on visual observation, making the results susceptible to the influence of the physician's clinical experience, and they are difficult to quantify subtle muscle movements such as localized twitching and asymmetrical contractions. Surface EMG can only collect single electromyographic signals and cannot simultaneously capture the dynamic coordination of facial muscles. Current methods for clinically assessing facial nerve function still need improvement. Summary of the Invention
[0003] The technical problem to be solved by the present invention is to provide a multimodal facial nerve assessment and neurological examination method and system to address the above-mentioned deficiencies of the prior art, aiming to solve the problem that the facial nerve function assessment results in the prior art are easily affected by the subjective factors of doctors or are incomplete.
[0004] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows: Firstly, a multimodal facial nerve assessment and neurological examination method is proposed, including the following steps: The patient's head is fixed in place by a head positioning bracket; a demonstration video of the examination task is played for the patient, and the patient is then guided to perform the examination task. The next examination task is performed only after the previous one is completed. The examination tasks include closing the eyes, opening the eyes, showing teeth, smiling, and frowning. Electrode pads were attached around the patient's eyes and mouth to collect electromyographic signals as the patient performed the examination; high-definition video of the patient's face was captured to collect video signals as the patient performed the examination. The electromyographic and video signals are preprocessed before being spliced together. The concatenated data is input into a trained convolutional neural network model based on a multilayer perceptron, and the output is an evaluation result. The evaluation result includes a task completion label, a task completion score, and an anomaly event label.
[0005] In one embodiment, when the patient's head is fixed by the head positioning bracket, the patient is guided to place their chin and forehead against the head positioning bracket, and the contact position between the chin and forehead and the head positioning bracket is set as the positioning point. Based on YOLO-V8 key point recognition technology, the key points and positioning points of eyebrows, eyes, nose, lips, and facial contours in the high-definition video of the patient's face are identified. The two-dimensional coordinates of the key points and positioning points are obtained, and the first coordinate matrix with the dimension of (number of key points + number of positioning points) × 2 × (T*f2) is obtained, where T is the time for the patient to perform a single examination task and f2 is the sampling frequency of the video signal.
[0006] In one implementation, the starting frame coordinates in the first coordinate matrix are rotated so that the positioning point coordinates are located on the vertical center line of the screen, and the distance D between the positioning points is recorded at this time; rotation and scaling are performed on all coordinates of the first coordinate matrix at subsequent time points so that the positioning point coordinates are always located on the vertical center line of the screen and the distance between the positioning points is always D, thereby obtaining the second coordinate matrix.
[0007] In one embodiment, the video signal preprocessing step includes compressing the second coordinate matrix into a matrix of dimension C×(T*f2) using a convolutional neural network model, where C is the number of electromyographic signal channels, T is the time for the patient to perform a single examination task, and f2 is the video signal sampling frequency.
[0008] In one embodiment, the electromyographic signal is a matrix of dimension C×(T*f1), where C is the number of electromyographic signal channels, T is the time for the patient to perform a single examination task, and f1 is the sampling frequency of the electromyographic signal; the preprocessing step of the electromyographic signal includes filtering and downsampling to obtain a matrix of dimension C×(T*f2), where C is the number of electromyographic signal channels, T is the time for the patient to perform a single examination task, and f2 is the sampling frequency of the video signal.
[0009] In one embodiment, the electromyographic signals and video signals are spliced together to form a matrix with dimensions (C*2)×(T*f2), where C is the number of electromyographic signal channels, T is the time for the patient to perform a single examination task, and f2 is the sampling frequency of the video signal.
[0010] In one implementation, the steps of training a convolutional neural network—based on a multilayer perceptron neural network model—include: High-definition facial videos of normal individuals and patients performing examination tasks are collected, along with simultaneous electromyography (EMG) and video signals. Neurologists review these videos and score and label them based on dimensions such as task completion, task completion rating, and abnormal event labeling. The EMG signals, video signals, and the neurologists' scores and labels are compiled into a training set to train a convolutional neural network model based on a multilayer perceptron neural network.
[0011] Secondly, a multimodal facial nerve assessment and neurological examination system is proposed, including: The task guidance module includes a head positioning submodule and an audiovisual task prompting submodule. The head positioning bracket module is used to fix the patient's head with the head positioning bracket. The audiovisual task prompting submodule is used to play a demonstration video of the examination task to the patient and then guide the patient to perform the examination task. After one examination task is completed, the next task is performed. The examination tasks include closing the eyes, opening the eyes, showing teeth, smiling, and frowning. The multimodal signal acquisition module is used to attach electrode pads around the patient's eyes and mouth to acquire electromyographic signals when the patient performs examination tasks; and to capture high-definition video of the patient's face to acquire video signals when the patient performs examination tasks. The signal processing module preprocesses the electromyographic signals and video signals, and then splices them together; the spliced data is input into a trained convolutional neural network-based multilayer perceptron neural network model, and the evaluation results are output; the evaluation results include task completion labels, task completion scores, and abnormal event labels.
[0012] In one embodiment, the head positioning submodule further includes a method for guiding the patient to place their chin and forehead against the head positioning bracket when the patient's head is fixed by the head positioning bracket, and setting the contact position between the chin and forehead and the head positioning bracket as a positioning point; the multimodal signal acquisition module further includes a method for identifying key points of eyebrows, eyes, nose, lips, and facial contours, as well as positioning points, in the high-definition video of the patient's face based on YOLO-V8 key point recognition technology, and obtaining the two-dimensional coordinates of the key points and positioning points.
[0013] In one embodiment, the signal processing module includes a training submodule and a prediction submodule.
[0014] The beneficial effects of this invention are: 1. This invention utilizes electromyography (EMG) signals to directly quantify the electrophysiological activity of muscles and uses video keypoint data to accurately calculate the amplitude and symmetry of movement, avoiding inconsistencies caused by subjective scales. By fusing the multimodal features of EMG and video signals, it simultaneously quantifies muscle electrical activity and spatial motion characteristics, significantly improving the sensitivity of recognizing subtle patient movements, such as localized twitching and asymmetrical contractions that are difficult for the human eye to detect, reducing errors in assessing patient dynamic coordination, and making facial nerve assessment results more reliable.
[0015] 2. The original high-definition video data is enormous. This invention, through "positioning bracket + key point compression," does not process all pixels of each frame. Instead, it compresses each frame into data containing only the coordinates of a few key points (i.e., a second coordinate matrix) using key point detection (such as YOLO-V8). This reduces video data processing volume by over 90%, significantly improving the efficiency of physical examination and diagnosis. Furthermore, by fixing the head with a head positioning bracket and calibrating the coordinates of the key points and positioning point coordinate matrices through rotation and scaling, the influence of factors such as the patient's face shape, size, and posture on the assessment results is effectively eliminated. This makes the subtle displacements calculated from the 2D video sequence theoretically feasible at the algorithmic level.
[0016] 3. This invention uses YOLO-V8 key point recognition technology to convert high-definition videos of patients' faces into key point animations, thus meeting GDPR (General Data Protection Regulation) medical privacy requirements.
[0017] 4. The method and system proposed in this invention are compatible with ordinary cameras and commercial electromyography devices, have strong hardware reusability, and are easy to promote and apply in clinical practice.
[0018] 5. The multimodal facial nerve assessment and neurological examination method and system proposed in this invention is a standardized process, which makes the assessment results comparable and solves the problem that current clinical assessments rely on doctors' subjective scales and the assessment results are not comparable between different medical institutions.
[0019] 6. This invention guides patients to complete actions through standardized video demonstrations and voice prompts, replacing doctors' verbal instructions and demonstrations, ensuring the consistency of action timing and amplitude, and achieving rapid assessment through a dual-model architecture of CNN-MLP, significantly saving clinical examination time and improving the efficiency of the examination process. Attached Figure Description
[0020] The invention will now be further described with reference to the accompanying drawings.
[0021] Figure 1 This is a framework diagram of the multimodal facial nerve assessment and neurological examination system according to an embodiment of the present invention. Detailed Implementation
[0022] This invention provides a multimodal facial nerve assessment method, characterized by comprising the following steps: S1. Fix the patient's head with a head positioning bracket; play a demonstration video of the examination task to the patient, and then guide the patient to perform the examination task. After one examination task is completed, the next one is performed. The examination tasks include closing the eyes, opening the eyes, showing teeth, smiling, and frowning. Specifically, the shape of the head positioning bracket is not fixed. In this embodiment, a head positioning bracket that allows the patient's chin and forehead to rest on the bracket is used, similar to an optometry head frame. When fixing the patient's head, the bracket must first be adjusted to fit the patient's size. The video playback device is placed 1 meter away from the patient, with the specific position adjusted according to the testing environment, ensuring that the patient can fully see the video content and fully hear the voice prompts. The examination task is first demonstrated by the video playback task, then the patient is prompted to perform the examination task, maintaining it for T seconds before the task is indicated to be over. One task is completed before proceeding to the next task, until all examination tasks are completed. The examination tasks used in this embodiment are set with reference to the House-Brackmann grading assessment scale. In actual use, the content of the examination tasks can be adjusted as needed, such as adding items like cheek puffing and whistling.
[0023] In one embodiment, when the patient's head is fixed by the head positioning bracket, the patient is guided to place their chin and forehead against the head positioning bracket, and the contact position between the chin and forehead and the head positioning bracket is set as the positioning point. Specifically, highly reflective positioning spheres are set at the positioning points (forehead and chin positions) on the side of the head positioning bracket facing the video acquisition device to assist in subsequent algorithm calibration and recognition.
[0024] S2. Place the electrode pads around the patient's eyes and mouth to collect electromyographic signals when the patient performs the examination task. Specifically, first wipe the skin of the target area with an alcohol swab, repeating the wiping 2-3 times. Wait for the alcohol to completely evaporate before attaching the electrodes. The electrodes around the eyes should avoid the eyelid margins and the eyeball, and the electrodes around the mouth should avoid the lip mucosa.
[0025] In one embodiment, the electromyographic signal is a matrix of dimension C×(T*f1), where C is the number of electromyographic signal channels (corresponding to the number of electrodes), T is the time for the patient to perform a single examination task, and f1 is the sampling frequency of the electromyographic signal; each row in the matrix corresponds to the continuous sampling data of one channel, and each column corresponds to the sampling value of all channels at the same time.
[0026] Capture high-definition videos of the patient's face and collect video signals when the patient performs examination tasks; In one implementation, based on YOLO-V8 keypoint recognition technology, keypoints and positioning points of eyebrows, eyes, nose, lips, and facial contours in a high-definition video of a patient's face are identified. Two-dimensional coordinates of the keypoints and positioning points are obtained, resulting in a first coordinate matrix with dimensions (number of keypoints + number of positioning points) × 2 (two-dimensional coordinates) × (T*f2) (total number of sampled frames). In this matrix, T represents the time taken for the patient to perform a single examination task, and f2 represents the video signal sampling frequency. The first dimension of the first coordinate matrix corresponds to one keypoint or positioning point; the second dimension corresponds to the two-dimensional coordinates of the keypoint or positioning point; and the third dimension corresponds to the coordinate data of each frame's time node. In other words, each dimension combination uniquely corresponds to the two-dimensional position of one keypoint or positioning point at a given moment.
[0027] Specifically, YOLO-V8's keypoint recognition technology, through a unified network architecture, eliminates the need for a dedicated keypoint detection network. While performing target detection, it can simultaneously output the coordinates of keypoints and localization points. It possesses enhanced feature extraction capabilities, combining adaptive anchor box matching and weighted loss function optimization to consistently output results even with varying target poses, occlusion, or complex backgrounds. This technology is existing and will not be elaborated further. In this embodiment, eyebrows, eyes, nose, lips, and facial contours are set as keypoints, while the forehead and chin are set as localization points. In actual use, the position and number of keypoints / localization points can be adjusted as needed.
[0028] In one implementation, the starting frame coordinates in the first coordinate matrix are rotated. This first coordinate matrix is a coordinate matrix with dimensions (number of keypoints + number of positioning points) × 2 (two-dimensional coordinates) so that the positioning point coordinates are located on the vertical center line of the image, and the distance D between the positioning points is recorded at this point. Then, all coordinates at subsequent time points in the first coordinate matrix are rotated and scaled to ensure that the positioning point coordinates are always located on the vertical center line of the image, and the distance between the positioning points is always D, thus obtaining the second coordinate matrix. Coordinate calibration can eliminate the influence of factors such as the patient's face shape, size, and posture on the evaluation results.
[0029] S3, preprocess the electromyographic signals and video signals, and then splice them together; In one implementation, the preprocessing of the electromyographic (EMG) signal includes filtering and downsampling to obtain a matrix of dimension C×(T*f2), where C is the number of EMG signal channels, T is the time for the patient to perform a single examination task, and f2 is the video signal sampling frequency. Specifically, filtering may include notch filtering to remove power frequency interference and avoid the influence of power grid noise on signal quality; it may also include bandpass filtering to filter out low-frequency motion artifacts (such as slight facial muscle movements) and high-frequency electronic noise, retaining only the effective frequency components of the EMG signal. Based on the video signal sampling frequency f2, a downsampling coefficient is calculated, and the filtered signal is decimated according to the calculated coefficient, compressing the data length of each channel from T*f1 to T*f2, so that the EMG signal sampling frequency is consistent with the video signal f2, facilitating subsequent synchronous analysis.
[0030] In one embodiment, the video signal preprocessing step includes compressing the second coordinate matrix into a matrix of dimension C×(T*f2) using a convolutional neural network model, where C is the number of electromyographic signal channels, T is the time the patient performs the examination task, and f2 is the video signal sampling frequency. Specifically, the two-dimensional coordinate matrix of (number of key points + number of positioning points)×2 is expanded into a temporal sequence (length T*f2) along the time dimension. After being input into the convolutional neural network, spatiotemporal features at different scales are extracted through multiple convolutional kernels. Pooling operations are used to reduce dimensionality and retain key information. Then, after mapping and transformation by a fully connected layer, a matrix of dimension C×(T*f2) with the same dimension as the preprocessed electromyographic signal is finally output, making the video coordinate data and the electromyographic signal completely matched in dimension and temporal length, laying the foundation for their splicing. This convolutional neural network model is mainly used to compress video signals and is independent of the convolutional neural network-based multilayer perceptron neural network model. In one embodiment, the electromyographic signals and video signals are spliced together into a matrix of dimension (C*2)×(T*f2), where C is the number of electromyographic signal channels, T is the time for the patient to perform the examination task, and f2 is the sampling frequency of the video signal.
[0031] S4. Input the concatenated data into the trained convolutional neural network (CNN) - based on the multilayer perceptron neural network (MLP) model, and output the evaluation results; the evaluation results include task completion label, task completion score, and abnormal event label.
[0032] In one implementation, the steps for training a convolutional neural network (CNN) based on a multilayer perceptron neural network model include: acquiring high-definition facial videos of normal individuals and patients performing examination tasks, and simultaneously acquiring electromyographic (EMG) signals and video signals; having a neurologist review the videos and score and label them based on dimensions such as task completion, task completion rating, and abnormal event labeling; and compiling the EMG signals, video signals, and the neurologist's scoring and labeling results into a training set to train the CNN-based EMG neural network model. In this embodiment, 10-fold cross-validation is used to verify the model's performance.
[0033] Specifically, task completion marking refers to the professional neurologist marking the patient's actual performance during a specific examination task, with the marking result ranging from 0 to 1, rounded to one decimal place. Task completion scoring refers to the professional neurologist's quantitative assessment and scoring of the quality of the patient's completion of the examination task. The scoring criteria consider multiple aspects such as accuracy, coordination, strength, and speed of movement, with the marking result ranging from 0 to 1, rounded to one decimal place. Abnormal event marking involves the professional neurologist recording and marking any events or phenomena that deviate from normal conditions during the patient's performance of the examination task. For example, significant tremors, increased muscle tone, and other abnormal events are marked, along with the time period of the abnormal episode.
[0034] like Figure 1 As shown, this embodiment of the invention also proposes a multimodal facial nerve assessment and neurological examination system, comprising: The task guidance module includes a head positioning submodule and an audiovisual task prompting submodule. The head positioning bracket submodule is used to fix the patient's head using a head positioning bracket. In one embodiment, the head positioning submodule further includes a feature to guide the patient to place their chin and forehead against the head positioning bracket when fixing the patient's head, setting the contact points between the chin / forehead and the head positioning bracket as positioning points. A highly reflective positioning sphere is provided at the positioning point (forehead and chin position) on the side of the head positioning bracket facing the video acquisition device to assist subsequent algorithm calibration and recognition. The audiovisual task prompting submodule plays a demonstration video of the examination task to the patient through the video demonstration engine of the audiovisual prompting system, and then guides the patient to perform the examination task and make action responses through audiovisual instructions. One examination task is completed before the next is performed. The examination tasks include closing eyes, opening eyes, showing teeth, smiling, and frowning. The multimodal signal acquisition module is used to attach electrode pads to the patient's eye and mouth area and acquire electromyographic signals when the patient performs the examination task through an electromyographic sensor; it also acquires video signals when the patient performs the examination task by capturing high-definition video of the patient's face through a high-definition camera; in one embodiment, based on YOLO-V8 key point recognition technology, it identifies key points and positioning points of eyebrows, eyes, nose, lips, and facial contours in the high-definition video of the patient's face, obtains the two-dimensional coordinates of the key points and positioning points, and obtains a first coordinate matrix with dimensions of (number of key points + number of positioning points) × 2 (two-dimensional coordinates) × (T*f2) (total number of sampling frames).
[0035] The signal processing module is used for spatial calibration of key points. Specifically, it rotates the starting frame coordinates in the first coordinate matrix, which is a coordinate matrix with dimensions (number of key points + number of positioning points) × 2 (two-dimensional coordinates), so that the positioning point coordinates are located on the vertical center line of the image, and records the distance D between the positioning points at this time. It then performs rotation and scaling on all coordinates of the first coordinate matrix at subsequent time points, ensuring that the positioning point coordinates are always located on the vertical center line of the image, and the distance between the positioning points is always D, thus obtaining the second coordinate matrix. The electromyography (EMG) signal and video signal are preprocessed. The EMG signal preprocessing includes filtering and downsampling. Both the EMG and video signals are organized into matrices of dimension C × (T*f2) and then concatenated. The concatenated data is input into a trained convolutional neural network (CNN) based multilayer perceptron neural network model. The CNN (CNN fusion model) performs feature extraction, and the multilayer perceptron neural network (MLP classifier) performs task evaluation, outputting the evaluation results. The evaluation results include task completion labels, task completion scores, and abnormal event labels.
[0036] In one implementation, the signal processing module includes a training submodule and a prediction submodule. The training submodule acquires high-definition facial videos of normal individuals and patients performing examination tasks, and simultaneously acquires electromyography (EMG) signals and video signals. A neurologist reviews the video and scores and labels it based on dimensions such as task completion, task completion rating, and abnormal event labeling. The EMG signals, video signals, and the neurologist's scores and labels are compiled into a training set for training a convolutional neural network—a multilayer perceptron-based neural network model. The prediction submodule preprocesses the data acquired by the multimodal signal acquisition module, extracts features, and outputs evaluation results.
[0037] It should be fully understood that the user information involved in this application (including but not limited to user physiological information, user personal information, etc.) is information and data authorized by the user or fully authorized by all parties. The use of user information shall comply with privacy policies and practices that are generally considered to meet or exceed industry or government requirements for maintaining user privacy. The collection, use and processing of related data shall comply with relevant laws, regulations and standards, and provide corresponding operation access points for users to choose to authorize or refuse.
[0038] The above descriptions of the embodiments are merely illustrative of the concept of this application and should not be construed as limiting the scope of protection of this application. It should be understood that the application of this invention is not limited to the examples described above. Those skilled in the art can make improvements or modifications based on the above descriptions, and all such improvements and modifications should fall within the scope of protection of the appended claims.
Claims
1. A multimodal facial nerve assessment and neurological examination method, characterized in that, Includes the following steps: The patient's head is fixed in place by a head positioning bracket; a demonstration video of the examination task is played for the patient, and the patient is then guided to perform the examination task. The next examination task is performed only after the previous one is completed. The examination tasks include closing the eyes, opening the eyes, showing teeth, smiling, and frowning. Electrode pads were attached around the patient's eyes and mouth to collect electromyographic signals as the patient performed the examination. Capture high-definition videos of the patient's face and collect video signals when the patient performs examination tasks; The electromyographic and video signals are preprocessed before being spliced together. The concatenated data is input into a trained convolutional neural network model based on a multilayer perceptron, and the output is an evaluation result. The evaluation result includes a task completion label, a task completion score, and an anomaly event label.
2. The multimodal facial nerve assessment and neurological examination method as described in claim 1, characterized in that: When fixing the patient's head with a head positioning bracket, guide the patient to place their chin and forehead against the head positioning bracket, and set the contact point between the chin and forehead and the head positioning bracket as the positioning point. Based on YOLO-V8 key point recognition technology, the key points and positioning points of eyebrows, eyes, nose, lips, and facial contours in the high-definition video of the patient's face are identified. The two-dimensional coordinates of the key points and positioning points are obtained, and the first coordinate matrix with the dimension of (number of key points + number of positioning points) × 2 × (T*f2) is obtained, where T is the time for the patient to perform a single examination task and f2 is the sampling frequency of the video signal.
3. The multimodal facial nerve assessment and neurological examination method as described in claim 2, characterized in that: Rotate the starting frame coordinates in the first coordinate matrix so that the positioning point coordinates are located on the vertical center line of the screen, and record the distance D between the positioning points at this time; rotate and scale all coordinates of the first coordinate matrix at subsequent time points so that the positioning point coordinates are always located on the vertical center line of the screen and the distance between the positioning points is always D, thereby obtaining the second coordinate matrix.
4. The multimodal facial nerve assessment and neurological examination method as described in claim 3, characterized in that: The video signal preprocessing steps include compressing the second coordinate matrix into a matrix of dimension C×(T*f2) using a convolutional neural network model, where C is the number of electromyographic signal channels, T is the time for the patient to perform a single examination task, and f2 is the video signal sampling frequency.
5. The multimodal facial nerve assessment and neurological examination method as described in claim 1, characterized in that: The electromyographic signal is a matrix of dimension C×(T*f1), where C is the number of electromyographic signal channels, T is the time for the patient to perform a single examination task, and f1 is the sampling frequency of the electromyographic signal. The preprocessing steps for the electromyographic signal include filtering and downsampling to obtain a matrix of dimension C×(T*f2), where C is the number of electromyographic signal channels, T is the time for the patient to perform a single examination task, and f2 is the sampling frequency of the video signal.
6. The multimodal facial nerve assessment and neurological examination method as described in claim 1, characterized in that: The electromyography (EMG) signals and video signals are spliced together to form a matrix with dimensions (C*2)×(T*f2), where C is the number of EMG signal channels, T is the time for the patient to perform a single examination task, and f2 is the sampling frequency of the video signal.
7. The multimodal facial nerve assessment and neurological examination method as described in claim 1, characterized in that: The steps for training a convolutional neural network—specifically, a multilayer perceptron-based neural network model—include: High-definition facial videos of normal individuals and patients performing examination tasks are collected, along with simultaneous electromyography (EMG) and video signals. Neurologists review these videos and score and label them based on dimensions such as task completion, task completion rating, and abnormal event labeling. The EMG signals, video signals, and the neurologists' scores and labels are compiled into a training set to train a convolutional neural network model based on a multilayer perceptron neural network.
8. A multimodal facial nerve assessment and neurological examination system, characterized in that, include: The task guidance module includes a head positioning submodule and an audiovisual task prompt submodule. The head positioning bracket sub-module is used to fix the patient's head using the head positioning bracket; The audiovisual task prompting submodule is used to play a demonstration video of the examination task to the patient and then guide the patient to perform the examination task. After one examination task is completed, the next one is performed. The examination tasks include closing the eyes, opening the eyes, showing teeth, smiling, and frowning. The multimodal signal acquisition module is used to attach electrode pads around the patient's eyes and mouth to acquire electromyographic signals when the patient performs examination tasks. Capture high-definition videos of the patient's face and collect video signals when the patient performs examination tasks; The signal processing module preprocesses the electromyographic signals and video signals before splicing them together. The concatenated data is input into a trained convolutional neural network model based on a multilayer perceptron, and the output is an evaluation result. The evaluation result includes a task completion label, a task completion score, and an anomaly event label.
9. A multimodal facial nerve assessment and neurological examination system as described in claim 8, characterized in that: The head positioning submodule also includes a method for guiding the patient to place their chin and forehead against the head positioning bracket when the patient's head is fixed by the head positioning bracket, and setting the contact position between the chin and forehead and the head positioning bracket as the positioning point; the multimodal signal acquisition module also includes a method for identifying key points of eyebrows, eyes, nose, lips, and facial contours, as well as positioning points, in the high-definition video of the patient's face based on YOLO-V8 key point recognition technology, and obtaining the two-dimensional coordinates of the key points and positioning points.
10. A multimodal facial nerve assessment and neurological examination system as described in claim 8, characterized in that: The signal processing module includes a training submodule and a prediction submodule.