Facial paralysis automatic evaluation system based on infrared thermogram and expression recognition

Abstract

The invention discloses a facial paralysis automatic evaluation system based on an infrared thermogram and expression recognition, and the system comprises a facial expression recognition module which is used for extracting the facial movement features of a patient; the infrared thermogram module is used for capturing and analyzing temperature distribution of the face of the patient; the data processing module is used for comparing the temperature difference between the affected side and the unaffected side of the face and generating an illness state assessment result; the output module is used for displaying and recording the illness state evaluation result; wherein the facial expression recognition module and the infrared thermogram module respectively acquire data and input the data to the data processing module, the data processing module outputs an analysis result to the output module, and an automatic and real-time facial paralysis condition assessment system is constructed. The whole process comprises data acquisition, feature extraction, data processing and result output, each link ensures the accuracy and objectivity of an evaluation result, the evaluation result is quickly generated, remote diagnosis is supported through remote transmission, and the method is an effective tool for facial paralysis condition evaluation.

Description

Technical Field

[0001] The invention relates to an automatic facial paralysis assessment system based on infrared thermal images and expression recognition, and belongs to the field of medical image processing and computer vision. Background Art

[0002] Facial palsy is a common facial muscle dysfunction, usually caused by viral infection, craniocerebral injury, tumor or inflammation. Existing methods for diagnosing and evaluating facial palsy mainly rely on doctors' experience and patients' self-reports, which are highly subjective, difficult to quantify and unsuitable for early diagnosis.

[0003] Existing technical methods and shortcomings:

[0004] 1. Physician experience and patient self-report:

[0005] At present, the diagnosis and evaluation of facial paralysis mainly rely on the clinical experience of doctors and self-reports of patients. Doctors will ask patients to perform a series of facial expression movements, and then evaluate the condition by observing the difference in muscle movements on the left and right sides of the patient's face.

[0006] This method is highly subjective, and different doctors may have different diagnoses, leading to inconsistent results. In addition, this method is difficult to quantify, as facial expressions are complex and difficult to quantify accurately, making comparisons at different time points and between different doctors difficult.

[0007] 2. Electromyography (EMG):

[0008] EMG measures the electrical activity of facial muscles using electrodes to assess muscle function.

[0009] Although this method can provide objective muscle activity data, it requires expensive equipment and professional operation, and the patient needs to cooperate during the test, which is not suitable for some special groups (such as children or the elderly). In addition, EMG is usually used to identify the specific location of muscle dysfunction rather than for early diagnosis and routine evaluation.

[0010] 3. Imaging examination (such as MRI, CT):

[0011] Magnetic resonance imaging (MRI) or computed tomography (CT) is done to check for structural abnormalities of the facial nerves and muscles.

[0012] Although these imaging studies can provide detailed anatomical information, they are mainly used to exclude structural lesions, are expensive and not suitable for frequent use, and are not suitable for functional evaluation of early facial palsy. Summary of the invention

[0013] The purpose of the present invention is to provide an automatic facial paralysis assessment system based on infrared thermal imaging and expression recognition, so as to overcome the deficiencies of the prior art and achieve objective, accurate and automated assessment of facial paralysis.

[0014] The present invention includes the following technical solutions:

[0015] An automatic facial paralysis assessment system based on infrared thermal imaging and expression recognition, comprising:

[0016] 1) Facial expression recognition module, used to extract the patient's facial movement features;

[0017] 2) Infrared thermography module, used to capture and analyze the temperature distribution on the patient's face;

[0018] 3) a data processing module, used to compare the temperature difference between the affected and unaffected sides of the face and generate a condition assessment result;

[0019] 4) Output module, used to display and record the results of disease assessment;

[0020] Among them, the facial expression recognition module and the infrared thermal imaging module respectively obtain data and input them into the data processing module, and the data processing module outputs the analysis results to the output module.

[0021] The present invention has the following advantages:

[0022] The present invention adopts infrared thermal imaging and facial expression recognition technology to extract objective temperature distribution and facial movement characteristics, thereby reducing the interference of human factors and providing more consistent and reliable evaluation results.

[0023] The present invention can quickly process facial expression videos and infrared thermal imaging data through an automated data processing module and generate evaluation results in real time, greatly improving the efficiency of facial paralysis condition evaluation and reducing the workload of doctors.

[0024] The present invention combines facial expression recognition and temperature difference analysis to comprehensively evaluate the patient's facial paralysis condition. By analyzing facial movement characteristics and temperature asymmetry, it can more comprehensively and accurately reflect the condition, especially capture early symptoms.

[0025] The present invention uses non-invasive facial expression recognition and infrared thermal imaging technology, does not require the cooperation of patients or complicated operations, and is suitable for patients of different ages, including children and the elderly, solving the assessment problem of special groups.

[0026] The present invention can detect slight differences in facial temperature changes and asymmetric expressions, significantly improving the accuracy of early diagnosis of the disease, so that facial paralysis can be discovered earlier and treated in a timely manner.

[0027] The present invention uses infrared thermal imaging and expression recognition technology, and does not require invasive operations. Patients only need to collect facial data through cameras and infrared devices to complete the assessment, which greatly improves the patient's comfort and willingness to accept.

[0028] The system of the present invention can remotely transmit the evaluation results to doctors through the network transmission function, and the doctors can make remote diagnosis and treatment recommendations based on real-time data, thus solving the problem of lack of medical resources in remote areas.

[0029] The present invention can record each assessment result through the data recording function, form a record of disease changes, provide doctors with long-term disease tracking data, and help formulate personalized treatment plans.

[0030] Through machine learning and deep learning technologies, the system of the present invention can continuously optimize the algorithm as the amount of data increases. The accuracy and reliability of the evaluation will be improved accordingly, ensuring long-term accurate diagnosis.

[0031] The present invention significantly improves the accuracy, efficiency and patient experience of facial paralysis diagnosis and management, and can provide strong technical support for early diagnosis, treatment and long-term management. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a diagram of the evaluation system of the present invention. DETAILED DESCRIPTION

[0033] Reference Figure 1 As shown in the figure, the complete working process of the facial paralysis automatic assessment system based on infrared thermal imaging and expression recognition technology includes facial expression recognition, infrared thermal imaging processing, data processing and evaluation, and result output. This process achieves automatic, accurate, and objective assessment of facial paralysis through the coordinated work of various modules, ensures the reliability and effectiveness of diagnosis, and can provide patients with timely and effective facial paralysis monitoring and diagnosis services.

[0034] 1. Facial Expression Recognition Module

[0035] Class Activation Mapping (CAM) technology: Generates activation maps of facial expressions through specific layers in a convolutional neural network (CNN). The activation map can highlight the key areas of the patient's facial movement and enhance the accuracy of expression recognition. This means that the system can effectively extract motion features by identifying areas related to expression and suppressing background interference, especially when facial expressions are not obvious or there is a lot of noise.

[0036] Axis segmentation and nose area retention: During the image processing process, the system will first segment the patient's face symmetrically along the axis to ensure that the comparative analysis of the left and right sides of the face can be performed accurately. The nose area is retained as a reference point to ensure the integrity and consistency of the symmetry between the left and right sides. Subsequently, the neural network will generate a facial feature matrix based on the images of these two sides, which contains the motion characteristics of each facial area.

[0037] Combined with optical flow information fusion: The facial expression recognition module also uses optical flow analysis to fuse the color information of the RGB image with the optical flow data in the xy direction. Optical flow captures the changing trend of the face in motion, and combined with RGB data can further improve the accuracy of expression recognition.

[0038] 2. Infrared thermal imaging module

[0039] Temperature difference comparison model: This module uses infrared thermography equipment to capture temperature distribution images of the patient's face. Based on the captured infrared thermographic data, the system analyzes the temperature difference between the affected facial area and the healthy area. Infrared thermography is particularly sensitive to temperature changes and can provide objective data on the functional status of the face, especially in terms of small temperature changes in the early stages.

[0040] Automatically divide the region of interest (ROI): The system will automatically identify key areas of the face, such as eyebrows, eyes, nose, corners of the mouth, etc., and use these areas as ROI. The temperature difference in these areas is an important basis for evaluating the condition of facial paralysis. By analyzing the temperature difference, area ratio, asymmetry and other characteristics of these areas, the system can derive the complete characteristics of the facial temperature distribution.

[0041] Temperature distribution feature extraction: The system not only extracts the temperature distribution features of each ROI, but also calculates the temperature difference between different ROIs and the area ratio of these regions, and finally quantifies the degree of temperature asymmetry between the left and right sides of the face. This provides the system with accurate temperature difference data for further evaluation.

[0042] 3. Data processing module

[0043] Comprehensive analysis of facial expression recognition and temperature difference data: The data processing module will comprehensively process the results of the facial expression recognition module and the infrared thermal imaging module. By analyzing the patient's facial movement characteristics and temperature distribution characteristics, the system can combine the two to generate a more comprehensive condition assessment result. This process relies on weighted algorithms or machine learning models, such as support vector machines, random forests, gradient boosting trees, or neural networks, to weight the expression and temperature features according to the importance of different features to ensure the accuracy and comprehensiveness of the analysis.

[0044] Automated evaluation system: The module's automated processing function can significantly reduce human intervention and subjective judgment. By automatically processing facial movement and temperature data, the system can quickly and objectively produce evaluation results. This not only improves the efficiency of diagnosis, but also increases the objectivity and consistency of the results.

[0045] 4. Output module

[0046] Display and recording function: The output module includes a display screen for real-time display of evaluation results, allowing doctors and patients to quickly understand the current condition. At the same time, the system uses a data logger to store the data of each evaluation to form a historical record of the condition. Long-term condition records provide doctors with tools to track changes in the condition and help adjust treatment plans.

[0047] Remote transmission function: The system supports remote diagnosis through the Internet, and the condition assessment results can be transmitted to remote doctors in real time. For patients in remote areas or with scarce medical resources, doctors can provide real-time diagnosis and treatment suggestions based on these data, greatly improving the accessibility of medical care.

[0048] Example:

[0049] 1. Data Collection

[0050] Use high-definition video equipment to record the patient's facial expression video and capture the facial movement characteristics. The equipment is mainly used to record the dynamic changes of facial muscles in real time when the patient makes a series of facial expressions (such as frowning, smiling, etc.).

[0051] At the same time, infrared thermal imaging equipment is used to obtain temperature distribution images of the patient's face. Infrared thermal images can capture temperature data of various areas of the face, especially the temperature difference between the diseased side and the healthy side of the face. The camera equipment is used to obtain facial movement data within the visible light range, and the infrared thermal imaging equipment is used to obtain temperature data within the invisible light range. The combination of the two can provide more comprehensive facial paralysis condition assessment data.

[0052] 2. Facial Expression Recognition

[0053] The collected patient facial expression videos are input into the convolutional neural network (CNN) model, and facial expression features are extracted through multi-layer convolution and pooling operations. The model is pre-trained and can efficiently identify differences in muscle movement on the patient's face. Class Activation Mapping (CAM) technology is used to generate activation maps of facial expressions. These activation maps can highlight key facial movement areas, such as the eyes and corners of the mouth, and suppress background interference and noise to ensure accurate extraction of facial movement features.

[0054] Image preprocessing:

[0055] During image preprocessing, the symmetry of the left and right faces is ensured by segmentation along the central axis, and the nose area is retained to maintain the consistency of key facial features. This step ensures that the left and right comparative analysis of facial expression recognition is based on complete and consistent data. The patient's facial video is input into the pre-trained CNN model, and the model extracts facial features through convolution operations. The facial activation map is generated using CAM technology to enhance the localization of facial motion features. The feature matrices of the left and right faces are compared and analyzed to identify differences in facial motion, especially between the diseased side and the healthy side.

[0056] 3. Infrared thermal image processing

[0057] Use infrared thermography equipment to obtain the temperature distribution of the patient's face. The image can accurately reflect the temperature difference in different areas of the face. The system will automatically divide the facial region of interest (ROI), such as the left and right eyebrows, eyes, nose wings, corners of the mouth and other key areas. The algorithm will analyze the temperature distribution characteristics of these areas, such as temperature difference, degree of asymmetry, etc. Infrared thermal imaging equipment can capture subtle temperature changes, especially the temperature of the affected side of the face of patients with facial paralysis is usually lower than the healthy side. The system uses a temperature difference comparison model to accurately compare the temperature of these areas to generate disease assessment data. The infrared thermal imaging device takes a picture of the patient's face and generates a temperature distribution image. The system uses an image processing algorithm to automatically divide the ROI area of ​​the face. The temperature difference between the left and right sides is calculated, the temperature asymmetry characteristics are extracted, and a preliminary disease assessment result is generated.

[0058] 4. Data processing and disease assessment

[0059] The expression recognition results and temperature difference analysis results are input into the data processing module. The data processing module combines the two and performs comprehensive analysis through weighted algorithms or machine learning models to generate automated assessment results of facial paralysis.

[0060] The core of the data processing module is to combine facial expression features and temperature features to comprehensively evaluate the patient's condition. By weighting the feature data or using a machine learning model, the system can generate more accurate facial paralysis assessment results. The analysis results of expression recognition and infrared thermal images are integrated into the data processing module.

[0061] The data is analyzed using a weighted algorithm or machine learning model to generate an assessment result, which includes information such as the severity of facial paralysis and the affected area.

[0062] 5. Result Output

[0063] The real-time display of the evaluation results on the display screen allows doctors and patients to quickly understand the condition. The system uses a data logger to record the results of each evaluation and generate a record of changes in the condition for long-term tracking and management. If necessary, the evaluation results can also be transmitted to remote doctors in real time through a network connection for remote diagnosis and treatment advice. The display screen displays data in real time, and doctors can adjust treatment plans based on the data. The data logger saves historical data and forms a long-term condition tracking file, which is crucial for long-term management and personalized treatment of the condition. The evaluation results are displayed in real time on the display screen for the doctor's reference. The data of each evaluation is saved using a data logger for subsequent condition comparison.

[0064] The data is transmitted to the remote doctor via the network for remote diagnosis.

[0065] The present invention combines facial expression recognition with infrared thermal imaging technology to build an automated, real-time facial paralysis assessment system. The entire process, from data collection, feature extraction, data processing to result output, ensures the accuracy and objectivity of the assessment results at every stage. The system can quickly generate assessment results and support remote diagnosis through remote transmission, making it an effective tool for facial paralysis assessment.

Claims

1. An automatic facial paralysis assessment system based on infrared thermal imaging and expression recognition, comprising: Facial expression recognition module, used to extract facial movement features of patients; Infrared thermography module to capture and analyze the temperature distribution on the patient’s face; A data processing module, used to compare the temperature difference between the affected and unaffected sides of the face and generate a condition assessment result; Output module, used to display and record the results of disease assessment; Among them, the facial expression recognition module and the infrared thermal imaging module respectively obtain data and input them into the data processing module, and the data processing module outputs the analysis results to the output module.

2. The system according to claim 1, characterized in that The facial expression recognition module extracts the activation map of facial expressions based on Class Activation Mapping (CAM) technology, thereby enhancing the positioning of facial motion features and suppressing background and noise interference.

3. The system according to claim 2, characterized in that The facial expression recognition module consists of the following parts: Class Activation Mapping (CAM): Through the Convolutional Neural Network, a specific layer in the CNN generates an activation map of facial expressions. The activation map can highlight the key areas of the patient's facial movements and enhance the accuracy of expression recognition. Axis segmentation and nose area retention: During the image processing process, the patient's face is first segmented symmetrically along the axis to ensure that the comparison and analysis of the left and right sides of the face can be accurately performed. The nose area is retained as a reference point to ensure the integrity and consistency of the symmetry of the left and right sides. Subsequently, the neural network will generate a facial feature matrix based on the images of these two sides. The matrix contains the motion characteristics of each facial area; Combined with optical flow information fusion: The facial expression recognition module also fuses the color information of the RGB image with the optical flow data in the xy direction through optical flow analysis. The optical flow captures the changing trend of the face in motion and combined with the RGB data can further improve the accuracy of expression recognition.

4. The system according to claim 1, characterized in that The infrared thermal imaging module is based on the temperature difference comparison model to evaluate the facial temperature change, automatically divide the region of interest, and extract the temperature distribution characteristics. The temperature difference comparison model uses infrared thermal imaging equipment to capture temperature distribution images of the patient's face. Based on the captured infrared thermal imaging data, the system analyzes the temperature difference between the affected facial area and the healthy area. Infrared thermal images are particularly sensitive to temperature changes and provide objective data for the functional status of the face, especially in terms of small temperature changes in the early stages. Automatically divide the region of interest (ROI): The system will automatically identify the key areas of the face, such as eyebrows, eyes, nose, and corners of the mouth, and use these areas as ROIs. The temperature difference in these areas is an important basis for evaluating the condition of facial paralysis. By analyzing the temperature difference, area ratio, asymmetry and other characteristics of these areas, the system can derive the complete characteristics of the facial temperature distribution. Temperature distribution feature extraction: The system will not only extract the temperature distribution features of each ROI, but also calculate the temperature differences between different ROIs and the area ratios of these regions, and finally quantify the degree of temperature asymmetry between the left and right sides of the face for further evaluation.

5. The system according to claim 1, characterized in that The data processing module can automatically divide the facial ROI and extract facial asymmetric temperature distribution features from the ROI, including temperature difference, area ratio and asymmetry degree.

6. The system according to claim 1, 2 or 3, characterized in that: The facial expression recognition module generates a feature matrix of facial expressions on the left and right sides by segmenting the central axis and retaining the nose area in combination with processing of a neural network.

7. The system according to claim 1, characterized in that The facial expression recognition and infrared thermal imaging modules fuse the red, green, blue, RGB data and the optical flow values ​​in the xy direction through optical flow information fusion to improve the accuracy of expression recognition.

8. The system according to claim 1, characterized in that The data processing module can comprehensively analyze the expression recognition results and the temperature difference analysis results, fuse and standardize the expression recognition features and temperature features, use machine learning models such as support vector machines, random forests, gradient boosting trees or neural networks for classification analysis, and after training and verification, achieve a comprehensive assessment of facial paralysis and generate automated assessment results of facial paralysis.

9. The system according to claim 1, characterized in that The output module includes a display screen and a data recorder, which are used to display and record the facial paralysis condition assessment results.

Images