A method and system for analyzing facial asymmetry

Abstract

This application provides a method and system for facial asymmetry analysis, relating to the interdisciplinary fields of computer vision and machine learning, and particularly to a method for facial asymmetry analysis, comprising: acquiring multiple frames of facial images to be evaluated; performing inter-frame difference operations on the multiple frames of facial images to obtain difference images, and inputting the difference images into a pre-trained first classification model to obtain probability values ​​predicted as pathological facial asymmetry, as dynamic features; detecting facial key points based on the multiple frames of facial images and a preset facial key point detection algorithm, and using the facial key points to calculate eyebrow tilt, the ratio of eyebrow height to inner canthus distance, mouth corner tilt, and mouth corner angle, as static features; fusing the dynamic and static features to obtain fused features; and inputting the fused features into a pre-trained second classification model to obtain classification results. This application can improve the accuracy of facial asymmetry analysis.

Description

Technical Field

[0001] This application relates to the interdisciplinary field of computer vision and machine learning, and in particular to a method and system for facial asymmetry analysis. Background Technology

[0002] Stroke is a common acute injury to the central nervous system. Patients often exhibit typical unilateral facial muscle movement abnormalities due to impaired facial nerve control, manifesting as asymmetry such as a crooked mouth, ptosis, and a shallow nasolabial fold. This type of pathological facial asymmetry is usually sudden, dynamic, and closely related to neurological function. Physiological facial asymmetry, on the other hand, is mostly caused by uneven skeletal development, differences in soft tissue distribution, or individual habits (such as unilateral chewing). It is usually stable, non-progressive, and not accompanied by neurological abnormalities.

[0003] Currently, clinical practice relies heavily on doctors' visual observation and subjective judgment to assess and differentiate between pathological and physiological facial asymmetry, which suffers from significant issues of strong subjectivity, poor repeatability, and difficulty in quantification.

[0004] Although several image processing algorithms have been proposed for facial symmetry analysis, most methods focus only on geometric differences in static images and fail to fully incorporate dynamic facial features. For example:

[0005] 1) Some studies have used landmarks in the left and right regions of static facial images to calculate Euclidean or geodesic distance matrices, and then averaged the left and right difference matrices as an asymmetry score. However, this type of method is only applicable to the assessment of static structures and cannot reflect the dynamic diagnostic information carried by changes in facial expressions.

[0006] 2) The MICCAI (CCF-B class) conference paper "Toward Rapid Stroke Diagnosis with Multimodal Deep Learning" extracts facial motion features by subtracting adjacent frames after performing 3×3 convolutions. While this can reflect abnormal muscle activity, it does not consider static structural changes such as those caused by facial paralysis. Furthermore, the feature distortion and information loss that easily occur when a single network processes heterogeneous data leads to a decrease in the accuracy of facial asymmetry assessment.

[0007] Therefore, how to improve the accuracy of facial asymmetry analysis is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0008] To address the aforementioned technical problems, this application provides a facial asymmetry analysis method that improves the accuracy of facial asymmetry analysis. This application also provides a facial asymmetry analysis system with the same technical effects.

[0009] The first objective of this application is to provide a method for analyzing facial asymmetry.

[0010] The aforementioned objective of this application is achieved through the following technical solution:

[0011] A method for analyzing facial asymmetry includes:

[0012] Acquire multiple frames of face images to be evaluated;

[0013] Based on multiple frames of the face images, inter-frame difference operations are performed to obtain difference images. These difference images are then input into a pre-trained first classification model to obtain the probability value predicted as pathological facial asymmetry, which serves as a dynamic feature.

[0014] Based on multiple frames of the face image and a preset face key point detection algorithm, face key points are detected, and the eyebrow tilt, eyebrow height to inner corner distance ratio, mouth corner tilt and mouth corner angle are calculated using the face key points as static features.

[0015] The dynamic features and the static features are fused to obtain fused features;

[0016] The fused features are input into a pre-trained second classification model to obtain the classification result.

[0017] Preferably, in the facial asymmetry analysis method, the step of performing inter-frame difference operations based on multiple frames of the facial images to obtain a difference image includes:

[0018] Extract two consecutive frames of the face image from the multiple frames of the face image to obtain the previous frame face image and the next frame face image;

[0019] Based on the RGB three-channel data of the previous frame face image and the RGB three-channel data of the subsequent frame face image, a difference operation is performed and the absolute value is taken to obtain the difference result;

[0020] Based on the difference results, a difference image is obtained.

[0021] Preferably, the facial asymmetry analysis method further includes:

[0022] For the previous frame face image and the next frame face image, a Gaussian mixture model is used to model the background, separate the foreground and background in the image, and segment the face from the background to obtain the processed previous frame face image and the processed next frame face image.

[0023] Preferably, in the facial asymmetry analysis method, the preset facial landmark detection algorithm adopts the 68-point facial landmark detection algorithm from the Dlib vision library. The step of detecting facial landmarks based on multiple frames of the facial images and the preset facial landmark detection algorithm includes:

[0024] Based on the multiple frames of the face images and the 68-point face key point detection algorithm, the ordered first to 68th face key points are obtained.

[0025] Preferably, in the facial asymmetry analysis method, the step of calculating the eyebrow tilt using the facial key points includes:

[0026] Based on the 20th and 25th facial key points, the first direction vector is obtained. Based on the 22nd and 23rd facial key points, the second direction vector is obtained. Based on the 21st and 24th facial key points, a third-party vector is obtained. ;

[0027] Calculate the first direction vector With the second direction vector The included angle between them and the third-direction vector With the second direction vector The average of the included angles between the two points is used to obtain the eyebrow tilt, and the calculation formula is:

[0028] ;

[0029] In the formula, Indicates the slant of the eyebrow.

[0030] Preferably, in the facial asymmetry analysis method, the step of calculating the ratio of eyebrow height to inner canthus distance using the facial key points includes:

[0031] Based on the 40th and 43rd facial key points, the fourth direction vector is obtained. Calculate the vector from the 20th, 21st, 24th, and 25th facial key points to the fourth direction vector, respectively. The distance is used to obtain the first distance. Second distance Third distance and the fourth distance ;

[0032] According to the first distance The second distance The third distance The fourth distance and the fourth direction vector The ratio of eyebrow height to inner corner distance is calculated using the following formula:

[0033] ;

[0034] In the formula, This indicates the ratio of eyebrow height to the distance between the inner corners of the eyes.

[0035] Preferably, in the facial asymmetry analysis method, the step of calculating the corner tilt of the mouth using the facial key points includes:

[0036] Based on the 49th and 55th facial key points, the fifth direction vector is calculated. ;

[0037] According to the fifth direction vector and the second direction vector The angle of inclination of the corner of the mouth is calculated using the following formula:

[0038] ;

[0039] In the formula, Indicates the degree of tilt of the corners of the mouth.

[0040] Preferably, in the facial asymmetry analysis method, the step of calculating the corner of the mouth angle using the facial key points includes:

[0041] Based on the 49th and 50th facial key points, the sixth direction vector is calculated. Based on the 49th and 60th facial key points, the seventh direction vector is calculated. ;

[0042] According to the sixth direction vector and the seventh direction vector The angle of the corner of the mouth is calculated using the following formula:

[0043] ;

[0044] In the formula, Indicates the angle of the corners of the mouth.

[0045] Preferably, in the facial asymmetry analysis method, the first classification model adopts the ResNet50 binary classification network model; the second classification model adopts the gradient boosting decision tree model.

[0046] The second objective of this application is to provide a facial asymmetry analysis system.

[0047] The second objective of this application is achieved through the following technical solution:

[0048] A facial asymmetry analysis system, comprising:

[0049] The acquisition unit is used to acquire multiple frames of face images;

[0050] The dynamic extraction unit is used to perform inter-frame difference operation based on multiple frames of the face image to obtain a difference image, and input the difference image into a pre-trained first classification model to obtain a probability value predicted as pathological facial asymmetry, which is used as a dynamic feature.

[0051] The static extraction unit is used to detect facial key points based on multiple frames of the face image and a preset facial key point detection algorithm, and to use the facial key points to calculate the eyebrow tilt, the ratio of eyebrow height to inner corner of the eye, the tilt of the mouth corner, and the angle of the mouth corner as static features.

[0052] A fusion unit is used to fuse the dynamic features and the static features to obtain fused features;

[0053] The classification unit is used to input the fused features into a pre-trained second classification model to obtain the classification result.

[0054] The above technical solution first acquires multiple frames of face images to be evaluated; then, based on the multiple frames of face images, performs inter-frame difference operations to obtain difference images, and inputs the difference images into a pre-trained first classification model to obtain the probability value of predicting pathological facial asymmetry; and then, based on the first classification model, independently extracts dynamic features to obtain dynamic features.

[0055] Based on multiple frames of facial images and a preset facial landmark detection algorithm, facial landmarks are detected. Using these landmarks, the eyebrow tilt, the ratio of eyebrow height to inner canthus distance, the corner of the mouth tilt, and the corner of the mouth angle are calculated. Based on these, static features are extracted independently using geometric calculations, resulting in four feature parameters that have the strongest correlation and the most stable discriminative power with facial nerve palsy caused by stroke, which are then used as static features.

[0056] This dynamic and static feature decoupling extraction mechanism avoids feature distortion and information loss caused by a single network processing heterogeneous data, enabling subsequent feature fusion to be learned based on purer and more representative features.

[0057] Next, the dynamic and static features are fused to obtain fused features. The fused features are then input into a pre-trained second classification model to obtain the classification result. This achieves the fusion of dynamic and static features, which can effectively improve the classification accuracy of the model compared to single-feature model classification methods.

[0058] In summary, the above technical solutions can improve the accuracy of facial asymmetry analysis. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a flowchart illustrating a facial asymmetry analysis method according to an embodiment of this application.

[0061] Figure 2 This is a schematic diagram of 68 facial key points in an embodiment of this application;

[0062] Figure 3 This is a schematic diagram of the structure of a facial asymmetry analysis system according to an embodiment of this application. Detailed Implementation

[0063] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0064] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described below are merely illustrative. For example, the division of units and modules is only a logical functional division. In actual implementation, there may be other division methods, such as multiple units or modules can be combined, or integrated into another system, or some features can be ignored or not executed.

[0065] Furthermore, the coupling, direct coupling, or communication connection between the components shown or discussed may be indirect coupling or communication connection through some interface, device, or module, and may be electrical, mechanical, or other forms.

[0066] In addition, each functional unit in the various embodiments of this application can be integrated into a single processor, or each unit can be a separate device, or two or more units can be integrated into a single device; each functional unit in the various embodiments of this application can be implemented in hardware or in the form of hardware plus software functional units.

[0067] Those skilled in the art will understand that all or part of the steps of the following method embodiments can be implemented by program instructions and related hardware. The aforementioned program instructions can be stored in a computer-readable storage medium. When the program instructions are executed, they perform the steps of the following method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0068] It should be understood that the use of terms such as "system," "device," "unit," and / or "module" in this application is merely one method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0069] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "a plurality of" or "several" means two or more, unless otherwise explicitly specified.

[0070] If a flowchart is used in this application, it is used to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0071] It should also be noted that, in this document, terms such as “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the article or device that includes the aforementioned element.

[0072] The embodiments in this application are written in a progressive manner.

[0073] like Figure 1 As shown in the figure, this application provides a method for analyzing facial asymmetry, including:

[0074] S101. Acquire multiple frames of face images to be evaluated;

[0075] Specifically, in S101, multiple frames of facial images of the person to be evaluated can be continuously acquired under natural conditions to obtain multiple frames of facial images to be evaluated. The person to be evaluated may have facial symptoms and require evaluation for physiological or pathological facial asymmetry.

[0076] In some embodiments, one implementation of this step specifically includes: acquiring a face video to be evaluated, processing the face video, and obtaining multiple frames of face images to be evaluated.

[0077] Specifically, under natural conditions, a video recording device can be used to have the person being evaluated read a pre-set passage to capture their facial video. The facial video is then preprocessed, using a face detector from the Dlib vision library to locate the face, and an ECO (Efficient Convolution Operators for Tracking) target tracker to trace the facial bounding box. Through a linear transformation from 2D coordinates to a 3D average face, pitch, roll, and yaw angles are calculated. Frames with excessively large angles (>5) or unstable motion are excluded to extract a stable frame sequence containing only frontal or near-frontal facial movements, which serves as the multi-frame facial image to be evaluated. The number of frames extracted can be set based on actual needs; for example, five frames can be extracted at equal intervals from a single facial video as the facial image to be evaluated. This application is not limited to this.

[0078] S102. Based on multiple frames of face images, perform inter-frame difference operation to obtain difference images, and input the difference images into the pre-trained first classification model to obtain the probability value of predicting pathological facial asymmetry, which is used as dynamic features;

[0079] In S102, specifically, differential operations can be performed on two or more consecutive frames of face images in a multi-frame face image to obtain a differential image for dynamic image analysis.

[0080] A first classification model can be pre-constructed, taking the difference image as input and predicting the probability of pathological facial asymmetry as output. Then, facial image samples of people with pathological facial asymmetry and people with physiological facial asymmetry are collected, and difference image samples are calculated. The first classification model is trained using the difference image samples to obtain the trained first classification model.

[0081] The first classification model can be an existing image classification model. The calculated difference image is input into the pre-trained first classification model, which is then used to classify and predict the difference image. The probability value of the prediction of pathological facial asymmetry output by the first classification model is used as a dynamic feature.

[0082] In some embodiments, one implementation of the step of performing inter-frame difference operations on multiple frames of face images to obtain a difference image specifically includes:

[0083] Extract face images from two consecutive frames of multi-frame face images to obtain the face image of the previous frame and the face image of the next frame;

[0084] Based on the RGB three-channel data of the previous frame face image and the RGB three-channel data of the next frame face image, a difference operation is performed and the absolute value is taken to obtain the difference result;

[0085] Based on the difference results, the difference image is obtained.

[0086] Specifically, two consecutive face images are extracted from multiple face images to obtain the previous frame face image and the next frame face image. Then, the RGB three-channel separation is performed on the previous and next frame face images respectively to obtain their respective RGB three-channel data. Then, based on their respective RGB three-channel data, the difference operation of the next frame minus the previous frame is performed, and the absolute value is taken to obtain the difference result. Finally, based on the difference result, the difference image is generated.

[0087] In this embodiment, differential operations are performed on the RGB three channels respectively, which can fully preserve the inter-frame changes of different color channels, avoid the loss of information in a single channel, and help the subsequent first classification model focus on abnormal facial movements.

[0088] In other embodiments, the first classification model employs a ResNet50 binary classification network model.

[0089] Specifically, the ResNet50 binary classification network model takes the difference image as input and outputs the probability value of pathological facial asymmetry through the sigmoid activation layer at the end of the network. The ResNet50 binary classification network model has powerful feature extraction capabilities; it solves the gradient vanishing problem in deep network training through residual connections, enabling it to better capture subtle dynamic features related to pathological facial asymmetry in the difference image.

[0090] S103. Based on multiple frames of facial images and a preset facial key point detection algorithm, facial key points are detected, and using the facial key points, eyebrow tilt, ratio of eyebrow height to inner corner of eye distance, mouth corner tilt and mouth corner angle are calculated as static features.

[0091] In S103, specifically, the preset facial landmark detection algorithm can be an existing facial landmark detection algorithm. One face image can be extracted from multiple frames of face images, and the preset facial landmark detection algorithm can be used to detect facial landmarks, obtaining them. Then, geometric calculations are performed based on the facial landmarks to obtain four feature parameters: eyebrow tilt, ratio of eyebrow height to inner corner of eye, corner of mouth tilt, and corner of mouth angle, which are used as static features. Alternatively, multiple face images can be extracted from multiple frames of face images, and facial landmark detection can be performed on each to obtain the facial landmarks corresponding to each face image. Then, geometric calculations are performed to obtain the aforementioned four feature parameters corresponding to each face image, and the average of these values ​​is taken as static features. This application is not limited to this method.

[0092] Although numerous quantifiable parameters exist for the face, these four specific parameters (eyebrow tilt, eyebrow height ratio, corner of mouth tilt, and corner of mouth angle) exhibit the strongest correlation and most stable discriminative power with facial nerve palsy caused by stroke. The meanings of these four characteristic parameters are shown in Table 1 below:

[0093] Table 1 Meaning of Feature Parameters

[0094]

[0095] In addition, this embodiment adopts a dynamic and static feature decoupling extraction mechanism, which avoids feature distortion and information loss caused by a single network processing heterogeneous data, so that subsequent feature fusion can be learned based on purer and more representative features.

[0096] In some embodiments, the preset facial landmark detection algorithm adopts the 68-point facial landmark detection algorithm in the Dlib vision library. One implementation of the step of detecting facial landmarks based on multiple frames of facial images and the preset facial landmark detection algorithm includes: obtaining the ordered first to 68th facial landmarks based on multiple frames of facial images and the 68-point facial landmark detection algorithm.

[0097] Specifically, the 68-point facial landmark detection algorithm can be the "shape_predictor_68_face_landmarks" algorithm in the Dlib vision library. Using this algorithm, face images can be detected, and the coordinates of 68 facial landmarks can be predicted within the detected face regions to obtain the ordered first to 68th facial landmarks.

[0098] The data for 68 facial landmarks are shown in Table 2 below:

[0099] Table 2. Explanation of 68 Key Points

[0100]

[0101] Combination Figure 2 As shown, in the right eyebrow area, the three points from the inner corner of the right eyebrow to the outer corner are the 22nd, 21st, and 20th facial key points; in the left eyebrow area, the three points from the inner corner of the left eyebrow to the outer corner are the 23rd, 24th, and 25th facial key points; the inner corner of the right eye is the 40th facial key point, the inner corner of the left eye is the 43rd facial key point, the right corner of the mouth in the outer lip contour area is the 49th facial key point, the left corner of the mouth in the outer lip contour area is the 55th facial key point, the two points adjacent to the right corner of the mouth in the outer lip contour area are the 50th and 60th facial key points, and the two points adjacent to the left corner of the mouth in the outer lip contour area are the 54th and 56th facial key points.

[0102] In other embodiments, one implementation of the step of calculating the eyebrow tilt, the ratio of eyebrow height to inner corner distance, the corner tilt of the mouth, and the corner angle of the mouth using facial key points includes:

[0103] S1031. Based on the 20th and 25th facial key points, obtain the first direction vector. Based on the 22nd and 23rd facial key points, the second direction vector is obtained. Based on the 21st and 24th facial key points, a third-party vector is obtained. ;

[0104] Specifically, the coordinates of the 20th and 25th facial key points are determined, and the first direction vector is calculated. The coordinates of the 22nd and 23rd facial key points were determined, and the second direction vector was calculated. The coordinates of the 21st and 24th facial key points were determined, and the third-party vector was calculated. .

[0105] S1032. Calculate the first direction vector. With the second direction vector The included angle and the third-direction vector With the second direction vector The average of the included angles between the two points is used to obtain the eyebrow tilt, and the calculation formula is:

[0106] ;

[0107] In the formula, This indicates the inclination of the eyebrow; an increase in the inclination value suggests increased asymmetry in the motor function of the eyebrow area. arccos represents the inverse cosine function.

[0108] S1033. Based on the 40th and 43rd facial key points, obtain the fourth direction vector. Calculate the vectors from the 20th, 21st, 24th, and 25th facial keypoints to the fourth direction. The distance is used to obtain the first distance. Second distance Third distance and the fourth distance ;

[0109] Specifically, the coordinates of the 40th and 43rd facial key points are determined, and the fourth direction vector is calculated. The coordinates of the four facial key points are determined, and then the vector from each facial key point to the fourth direction is calculated based on the coordinates of each facial key point. The distance is used to obtain the vector from the 20th facial keypoint to the fourth direction. First distance The 21st facial keypoint to the fourth direction vector The second distance The 24th facial keypoint to the fourth direction vector The third distance and the fourth direction vector of the 25th facial keypoint The fourth distance .

[0110] S1034. Based on the first distance Second distance Third distance Fourth distance and the fourth direction vector The ratio of eyebrow height to inner corner distance is calculated using the following formula:

[0111] ;

[0112] In the formula, This indicates the ratio of eyebrow height to the distance between the inner corners of the eyes.

[0113] ;

[0114] b= ;

[0115] ;

[0116] In the formula, The x-coordinate of the 40th facial landmark is represented. The vertical coordinate of the 40th facial landmark is represented. The x-coordinate of the 43rd facial landmark is represented. The vertical coordinate represents the key facial landmark of the 43rd person. b represents the slope, and b represents the intercept. This represents the x-coordinate of the i-th facial landmark. This represents the y-coordinate of the i-th facial landmark. Represents the facial key points of the i-th person ( , The line connecting the inner corners of the eyes (i.e., the fourth direction vector) The vertical distance (i.e., eyebrow height) of the i-th facial key point ( , In the given information, i can take the values ​​20, 21, 24, or 25.

[0117] S1035. Based on the 49th and 55th facial key points, calculate the fifth direction vector. ;

[0118] Specifically, the coordinates of the 49th and 55th facial key points are determined, and the fifth direction vector is calculated. .

[0119] S1036. Based on the fifth direction vector Second direction vector The angle of inclination of the corner of the mouth is calculated using the following formula:

[0120] ;

[0121] In the formula, Indicates the degree of tilt of the corners of the mouth.

[0122] S1037. Based on the 49th and 50th facial key points, calculate the sixth direction vector. Based on the 49th and 60th facial key points, the seventh direction vector is calculated. ;

[0123] Specifically, the coordinates of the 49th and 50th facial key points are determined, and the sixth direction vector is calculated. The coordinates of the 49th and 60th facial key points were determined, and the seventh direction vector was calculated. In other embodiments, the sixth direction vector can also be calculated based on the coordinates of the 55th and 54th facial keypoints. Alternatively, the seventh direction vector can be calculated based on the coordinates of the 55th and 56th facial key points. This application is not limited to this.

[0124] S1038. Based on the sixth direction vector and the seventh direction vector The angle of the corner of the mouth is calculated using the following formula:

[0125] ;

[0126] In the formula, Indicates the angle of the corners of the mouth.

[0127] In other embodiments, after calculating the eyebrow tilt, the ratio of eyebrow height to inner corner of the eye, the corner of the mouth tilt, and the corner of the mouth angle using facial key points, the method further includes: standardizing the eyebrow tilt, the ratio of eyebrow height to inner corner of the eye, the corner of the mouth tilt, and the corner of the mouth angle to obtain the processed eyebrow tilt, the ratio of eyebrow height to inner corner of the eye, the corner of the mouth tilt, and the corner of the mouth angle as static features. The standardization process can employ Z-score standardization to unify the dimensions of the four feature parameters.

[0128] S104. Perform feature fusion on dynamic and static features to obtain fused features;

[0129] Specifically, in S104, dynamic features and static features can be concatenated to obtain fused features, thereby realizing the fusion of dynamic and static features. This is different from the existing single feature extraction and analysis methods that rely solely on static or dynamic features, and is beneficial to improving the accuracy of subsequent model classification.

[0130] S105. Input the fused features into the pre-trained second classification model to obtain the classification result.

[0131] In step S105, specifically, a second classification model can be pre-constructed, taking fusion features as input and predicting probability values ​​as output. Then, facial image samples of individuals with pathological facial asymmetry and physiological facial asymmetry are collected. Referring to steps S102 to S104 above, fusion feature samples are calculated. These fusion feature samples are then used to train the second classification model, resulting in a trained model. The second classification model can employ an existing machine learning model. The calculated fusion features are input into the pre-trained second classification model, which then performs classification prediction on the fusion features, outputting predicted probability values. Based on these predicted probability values ​​and a classification threshold, the classification result is obtained.

[0132] In some embodiments, the second classification model employs a Gradient Boosting Decision Tree (GBDT) model. Traditional machine learning models, due to their equal weighting, can easily lead to key pathological features being overwhelmed by noise. In this embodiment, the Gradient Boosting Decision Tree model is used as an adaptive fusion engine for heterogeneous features. Unlike fusion methods with fixed weights or simple addition, the Gradient Boosting Decision Tree model can automatically learn and assign dynamically changing, sample-dependent weights to dynamic features (reflecting immediate lesions) and static features (reflecting individual baselines). For example, for a patient with extremely high static symmetry, any slight dynamic asymmetry will be assigned extremely high weight; conversely, for a patient with inherently high static asymmetry, the model will rely more on the intensity of their dynamic changes. This ability to adaptively weight each individual is key to solving the problem of confusion between physiological and pathological asymmetries, and is difficult to achieve with existing fixed-weight or end-to-end networks.

[0133] Specifically, a gradient boosting decision tree model is used for supervised learning. The dynamic-static feature weights are adaptively allocated through feature importance. The input of the gradient boosting decision tree model is the fusion feature, and the output is the predicted probability value. (1) Training phase: Obtain fusion feature samples and set labels for the fusion feature samples (e.g., samples related to pathological facial asymmetry are set to 1, and samples related to physiological facial asymmetry are set to 0). Set the maximum tree depth to 5, the learning rate to 0.1, the number of trees to 200, and use early stopping (if the verification loss does not decrease after 5 rounds, the process is terminated). Feature importance calculation: Based on the information gain when the tree node splits, the cumulative split gain ratio of each feature in the entire tree is calculated and normalized to obtain the weight vector. (2) Prediction phase: Multiply the feature vector F with the weight W element by element to generate the weighted feature F. weighted =F⊙W, input to the leaf node of GBDT for probability prediction. Classification decision: The model outputs the predicted probability value P. [0,1], set the classification threshold to 0.5, P<0.5 is judged as physiological facial asymmetry, P≥0.5 is judged as pathological facial asymmetry.

[0134] In a specific embodiment, comparative experiments were conducted: classification using only static features achieved an accuracy of 85%; classification using only dynamic features output by ResNet achieved an accuracy of 89%; and the static-dynamic fusion method of this application achieved an accuracy of 91%. The experimental results show that the static-dynamic fusion framework of this application exhibits a significant synergistic effect, rather than a simple information superposition.

[0135] Most existing methods for facial symmetry analysis based on image processing algorithms focus only on geometric differences in static images, failing to fully incorporate dynamic facial features. Furthermore, feature distortion and information loss that easily occur when a single network processes heterogeneous data lead to a decrease in the accuracy of facial asymmetry assessment.

[0136] The above embodiment first acquires multiple frames of facial images to be evaluated. Based on these frames, inter-frame differencing is performed to obtain differencing images, which are then input into a pre-trained first classification model to obtain the probability value predicted as pathological facial asymmetry. Dynamic features are then extracted independently based on this first classification model. Next, facial landmarks are detected using the multiple frames of facial images and a pre-defined facial landmark detection algorithm. Using these landmarks, eyebrow tilt, the ratio of eyebrow height to inner canthus distance, mouth corner tilt, and mouth corner angle are calculated. Static features are then extracted independently based on geometric calculations, yielding four feature parameters with the strongest correlation and most stable discriminative power to facial nerve palsy caused by stroke, which are used as static features. This decoupling mechanism for dynamic and static feature extraction avoids feature distortion and information loss that occurs when a single network processes heterogeneous data, allowing subsequent feature fusion to be learned based on purer and more representative features. Next, dynamic and static features are fused to obtain fused features. These fused features are then input into a pre-trained second classification model to obtain the classification result. This achieves the fusion of dynamic and static features, effectively improving classification accuracy compared to single-feature model classification methods. In summary, the above embodiments can improve the accuracy of facial asymmetry analysis.

[0137] Existing facial dynamic analysis methods are susceptible to the effects of phone / camera shake in natural shooting scenarios. When there is background motion, the differential operation mixes facial motion signals with environmental noise, leading to feature distortion. This problem is particularly pronounced in stroke diagnosis scenarios where patients are unable to hold the device.

[0138] To address the aforementioned problems, in other embodiments of this application, after the step of extracting two consecutive face images from multiple frames of face images to obtain the previous frame face image and the next frame face image in the above-described facial asymmetry analysis method, the method further includes:

[0139] S301. For the previous frame face image and the next frame face image, use a Gaussian mixture model to model the background, separate the foreground and background in the image, and segment the face from the background to obtain the processed previous frame face image and the processed next frame face image.

[0140] Specifically, a Gaussian Mixture Model (GMM) is used for background modeling to distinguish between the foreground and background in the image, achieving background segmentation and extracting the face region, resulting in a processed front-frame face image and a processed rear-frame face image. Subsequent differencing operations are performed based on the processed face images, effectively reducing errors caused by camera shake and improving the accuracy of the differencing image calculation, thereby achieving more precise extraction of dynamic facial features.

[0141] The expression for background modeling using the Gaussian mixture model is as follows:

[0142] ;

[0143] In the formula, Represents background dynamic modeling features. It is the input image of frame t. It is a Gaussian kernel. It is the previous frame image after affine transformation. It is a face mask that is to be obscured, leaving only the background face visible.

[0144] In other embodiments of this application, the above-described facial asymmetry analysis method further includes:

[0145] S401. Perform interpretability analysis using the SHAP (Shapley Additive Explanations) algorithm.

[0146] Specifically, the SHAP algorithm can be used to perform interpretability analysis on the aforementioned GBDT model. First, based on the trained GBDT model, a model-compatible SHAP interpreter is used to calculate the SHAP value of each feature in each sample—these values ​​quantify the contribution of each feature to the model's prediction results relative to the average baseline. Subsequently, various visualization techniques (such as summary diagrams, waterfall plots, dependency graphs, etc.) can be used to interpret the model behavior from both global and local perspectives: summary diagrams reveal the overall importance of features and their direction of influence, while waterfall plots detail the constituent logic of individual predictions. This method not only has a solid game theory foundation, ensuring consistency and accuracy in interpretation, but also efficiently handles large-scale tree models, significantly improving computational efficiency. This results in a facial asymmetry analysis method that can integrate dynamic and static features, possessing high recognition accuracy and interpretability, to assist in the screening and risk prediction of neurological diseases such as stroke.

[0147] like Figure 3 As shown, in another embodiment of this application, a facial asymmetry analysis system is provided, comprising:

[0148] Acquisition unit 10 is used to acquire multiple frames of face images;

[0149] The dynamic extraction unit 11 is used to perform inter-frame difference operation based on multiple frames of face images to obtain a difference image, and input the difference image into the pre-trained first classification model to obtain the probability value of predicting pathological facial asymmetry, which is used as dynamic feature.

[0150] The static extraction unit 12 is used to detect facial key points based on multiple frames of facial images and a preset facial key point detection algorithm, and to use the facial key points to calculate the eyebrow tilt, the ratio of eyebrow height to inner corner of the eye, the tilt of the mouth corner and the angle of the mouth corner as static features.

[0151] Fusion unit 13 is used to fuse dynamic features and static features to obtain fused features;

[0152] Classification unit 14 is used to input the fused features into a pre-trained second classification model to obtain the classification result.

[0153] In other embodiments of this application, in the above-described facial asymmetry analysis system, the dynamic extraction unit 11, when performing inter-frame difference operations based on multiple frames of facial images to obtain a difference image, is specifically used for:

[0154] Extract face images from two consecutive frames of multi-frame face images to obtain the face image of the previous frame and the face image of the next frame;

[0155] Based on the RGB three-channel data of the previous frame face image and the RGB three-channel data of the next frame face image, a difference operation is performed and the absolute value is taken to obtain the difference result;

[0156] Based on the difference results, the difference image is obtained.

[0157] In other embodiments of this application, in the above-described facial asymmetry analysis system, the dynamic extraction unit 11 is further configured to: perform background modeling using a Gaussian mixture model on the previous frame face image and the next frame face image, separate the foreground and background in the image, segment the face from the background, and obtain the processed previous frame face image and the processed next frame face image.

[0158] In other embodiments of this application, in the above-described facial asymmetry analysis system, the preset facial landmark detection algorithm adopts the 68-point facial landmark detection algorithm from the Dlib vision library. The static extraction unit 12, when executing the detection of facial landmarks based on multiple frames of facial images and the preset facial landmark detection algorithm, is specifically used for:

[0159] Based on multiple frames of face images and a 68-point facial landmark detection algorithm, the ordered first to 68th facial landmarks are obtained.

[0160] In other embodiments of this application, in the above-described facial asymmetry analysis system, the static extraction unit 12, when performing calculations using facial key points to obtain eyebrow tilt, eyebrow height to inner corner distance ratio, mouth corner tilt, and mouth corner angle, is specifically used for:

[0161] Based on the 20th and 25th facial key points, the first direction vector is obtained. Based on the 22nd and 23rd facial key points, the second direction vector is obtained. Based on the 21st and 24th facial key points, a third-party vector is obtained. ;

[0162] Calculate the first direction vector With the second direction vector The included angle and the third-direction vector With the second direction vector The average of the included angles between the two points is used to obtain the eyebrow tilt, and the calculation formula is:

[0163] ;

[0164] In the formula, Indicates the degree of eyebrow slant;

[0165] Based on the 40th and 43rd facial key points, the fourth direction vector is obtained. Calculate the vectors from the 20th, 21st, 24th, and 25th facial keypoints to the fourth direction. The distance is used to obtain the first distance. Second distance Third distance and the fourth distance ;

[0166] Based on the first distance Second distance Third distance Fourth distance and the fourth direction vector The ratio of eyebrow height to inner corner distance is calculated using the following formula:

[0167] ;

[0168] In the formula, This indicates the ratio of eyebrow height to the distance between the inner corners of the eyes;

[0169] Based on the 49th and 55th facial key points, the fifth direction vector is calculated. ;

[0170] According to the fifth direction vector Second direction vector The angle of inclination of the corner of the mouth is calculated using the following formula:

[0171] ;

[0172] In the formula, Indicates the degree of tilt of the corners of the mouth;

[0173] Based on the 49th and 50th facial key points, the sixth direction vector is calculated. Based on the 49th and 60th facial key points, the seventh direction vector is calculated. ;

[0174] According to the sixth direction vector and the seventh direction vector The angle of the corner of the mouth is calculated using the following formula:

[0175] ;

[0176] In the formula, Indicates the angle of the corners of the mouth.

[0177] In other embodiments of this application, in the above-described facial asymmetry analysis system, the first classification model adopts a ResNet50 binary classification network model; the second classification model adopts a gradient boosting decision tree model.

[0178] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of facial asymmetry analysis, characterized by, The method comprises the following steps: obtaining a plurality of frames of face images to be evaluated; performing inter-frame difference operation on the plurality of frames of face images to obtain a difference image, and inputting the difference image into a pre-trained first classification model to obtain a probability value of predicting pathological facial asymmetry as a dynamic feature; detecting face key points according to the plurality of frames of face images and a preset face key point detection algorithm, and calculating eyebrow tilt, eyebrow height to inner canthus distance ratio, mouth corner tilt and mouth corner angle using the face key points as static features; performing feature fusion on the dynamic features and the static features to obtain fused features; inputting the fused features into a pre-trained second classification model to obtain a classification result, wherein the classification result is one of physiological facial asymmetry and pathological facial asymmetry.

2. The method as claimed in claim 1, characterized in that, The inter-frame difference operation on the plurality of frames of face images comprises the following steps: extracting two consecutive frames of face images from the plurality of frames of face images to obtain a front frame of face image and a rear frame of face image; performing difference operation on RGB three-channel data of the front frame of face image and RGB three-channel data of the rear frame of face image and taking absolute value to obtain a difference result; obtaining a difference image according to the difference result.

3. The method as claimed in claim 2, characterized in that, The method further comprises the following steps: using a Gaussian mixture model to model the background of the front frame of face image and the rear frame of face image, separating the foreground and the background in the image, and segmenting the face from the background to obtain a processed front frame of face image and a processed rear frame of face image.

4. The method as claimed in claim 1, wherein, The preset face key point detection algorithm adopts a 68-point face key point detection algorithm in a Dlib vision library, and the detection of face key points according to the plurality of frames of face images and the preset face key point detection algorithm comprises the following steps: obtaining an ordered first face key point to 68th face key point according to the plurality of frames of face images and the 68-point face key point detection algorithm.

5. The method as claimed in claim 4, characterized in that, The calculation of eyebrow tilt using the face key points comprises the following steps: According to the 20th face key point and the 25th face key point, a first direction vector is obtained According to the 22nd face key point and the 23rd face key point, a second direction vector is obtained According to the 21st face key point and the 24th face key point, a third direction vector is obtained ; computing the average of the angles between the first direction vector and the second direction vector and the third direction vector and the second direction vector eyebrow tilt, computed as ； In the formula, represents the eyebrow tilt.

6. The method as claimed in claim 5, characterized in that, The calculation of eyebrow height to inner canthus distance ratio using the face key points comprises the following steps: According to the 40th face key point and the 43rd face key point, a fourth direction vector is obtained , respectively calculate the distance from the 20th face key point, the 21st face key point, the 24th face key point and the 25th face key point to the fourth direction vector , obtain the first distance , the second distance , the third distance and the fourth distance ; According to the first distance , the second distance , the third distance , the fourth distance , and the fourth direction vector , the ratio of the eyebrow height to the inner corner distance is calculated, and the calculation formula is: ； In the formula, represents the ratio of eyebrow height to inner corner distance.

7. The method as claimed in claim 6, characterized in that, The calculation of mouth corner tilt using the face key points comprises the following steps: A fifth direction vector is calculated based on the 49th facial landmark and the 55th facial landmark ; According to the fifth direction vector and the second direction vector , the mouth corner tilt is calculated, and the calculation formula is: ； In the formula, represents the mouth corner tilt.

8. The method as claimed in claim 7, characterized in that, The calculation of mouth corner angle using the face key points comprises the following steps: A sixth directional vector is calculated according to the 49th facial key point and the 50th facial key point A seventh directional vector is calculated according to the 49th facial key point and the 60th facial key point ; According to the sixth direction vector and the seventh direction vector , the mouth corner angle is calculated, and the calculation formula is: ； In the formula, represents the mouth corner angle.

9. The method according to any one of claims 1 to 8, characterized in that, The first classification model adopts a ResNet50 binary classification network model, and the second classification model adopts a gradient boosting decision tree model.

10. A facial asymmetry analysis system characterized by, The method comprises the following steps: an acquisition unit configured to acquire a plurality of frames of face images; a dynamic extraction unit configured to perform inter-frame difference operation on the plurality of frames of face images to obtain a difference image, and input the difference image into a pre-trained first classification model to obtain a probability value of predicting pathological facial asymmetry as a dynamic feature; a static extraction unit configured to detect face key points according to the plurality of frames of face images and a preset face key point detection algorithm, and calculate eyebrow tilt, eyebrow height to inner canthus distance ratio, mouth corner tilt and mouth corner angle using the face key points as static features; a fusion unit configured to perform feature fusion on the dynamic features and the static features to obtain fused features; and an input unit configured to input the fused features into a pre-trained second classification model to obtain a classification result, wherein the classification result is one of physiological facial asymmetry and pathological facial asymmetry. A classification unit is configured to input the fused features into a pre-trained second classification model to obtain a classification result, wherein the classification result is one of a physiological facial asymmetry and a pathological facial asymmetry.

Images