Facial paralysis recognition and grade evaluation system and method based on expression analysis technology

Abstract

The invention discloses a facial paralysis recognition and grade evaluation system and method based on an expression analysis technology, and relates to the technical field of medical image intelligent analysis, and the method comprises the steps: carrying out the virtual correction of a face image in an action segment set based on a three-dimensional muscle group activation field, generating a healthy twin face, and carrying out the recognition and grade evaluation. Calculating a feature residual error between the face image and the healthy twin face to form a residual error severity index; performing feature fusion on the residual severity index, the three-dimensional muscle group activation field and the two-dimensional expression feature vector to form a muscle group associated residual feature graph, performing graph-level feature aggregation and multi-task reasoning through a graph neural network, outputting a facial paralysis recognition result and a graph-level feature vector, and performing multi-classification prediction on the graph-level feature vector to obtain a facial paralysis recognition result. And outputting the facial paralysis severity level. The facial paralysis identification and severity level evaluation method realizes more accurate and explainable facial paralysis identification and severity level evaluation, and improves the reliability and practicability of clinical auxiliary diagnosis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image intelligent analysis, in particular to a facial paralysis recognition and grade evaluation system and method based on expression analysis technology. BACKGROUND

[0002] With the development of computer vision, deep learning and medical image processing, the automatic recognition and grade evaluation technology of facial nerve dysfunction has gradually formed a system. Traditional facial paralysis evaluation mainly depends on the subjective judgment of doctors based on facial static images or short-term action observation. In recent years, video sequence analysis technology, expression dynamics modeling method and three-dimensional face reconstruction algorithm make it possible to quantitatively analyze facial movement function. With the maturity of differentiable three-dimensional face model and expression parameterization model, reconstructing three-dimensional facial structure from two-dimensional images to reflect muscle deformation trend has become an important technical direction, which can to some extent assist the medical scene to realize automatic pathological feature recognition. In general, the existing technology has a relatively complete processing framework in expression understanding and facial function analysis, which lays a foundation for facial paralysis intelligent evaluation method.

[0003] Although the existing technology can recognize facial static and dynamic expressions, in the facial paralysis scene, the pathological facial movement is often characterized by uneven left and right movement amplitude and complex activation mode change between muscle groups. Therefore, it is difficult to accurately capture the potential muscle group differences by relying only on two-dimensional key points or expression features. In addition, although some methods can generate three-dimensional facial morphology, the modeling of muscle groups is usually rough, and it is difficult to present the fine-grained influence of local activation of different muscle groups on expression changes in three-dimensional space. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the facial paralysis recognition and grade evaluation method based on expression analysis technology solves the problems of difficult to finely depict the local activation difference of three-dimensional muscle groups and accurately quantify the degree of pathological asymmetry.

[0006] To solve the above technical problems, the present application provides the following technical solutions: In a first aspect, the present application provides a facial paralysis recognition and grade evaluation method based on expression analysis technology, which comprises: acquiring a facial expression video sequence, performing face detection and face region cropping on the facial expression video sequence, and cutting into a plurality of action segments to obtain an action segment set; performing geometric alignment and normalization processing on the face images in the action segment set, and extracting corresponding two-dimensional expression feature vectors; According to the two-dimensional expression feature vector, three-dimensional face reconstruction is performed on the face images in the action segment set, and a three-dimensional muscle activation field is constructed according to a facial muscle group division rule; Based on the three-dimensional muscle activation field, virtual correction is performed on the face images in the action segment set to generate a healthy twin face, and a feature residual error between the face images and the healthy twin face is calculated to form a residual severity index; The residual severity index is fused with the three-dimensional muscle activation field and the two-dimensional expression feature vector to form a muscle group associated residual feature map, a graph neural network is used for graph-level feature aggregation and multi-task reasoning, and a facial paralysis recognition result and a graph-level feature vector are output, and the graph-level feature vector is subjected to multi-classification prediction to output a facial paralysis severity level.

[0007] As a preferred scheme of the facial paralysis recognition and level evaluation method based on the expression analysis technology, the facial expression video sequence is obtained by capturing the facial expression actions of the patient and synchronously recording the start and end time stamps of each facial expression action and the corresponding video frame index.

[0008] As a preferred scheme of the facial paralysis recognition and level evaluation method based on the expression analysis technology, the facial expression video sequence is obtained by capturing the facial expression actions of the patient and synchronously recording the start and end time stamps of each facial expression action and the corresponding video frame index. The facial expression video sequence is read frame by frame according to the start and end time stamps, and the face position area in each frame is determined by performing face detection on each frame to form a face area frame sequence. The face area frame sequence is used to crop the face area of the facial expression video sequence to obtain face area images, and the size and format of the face area images are unified according to a preset cropping standardization rule to form a standardized face image sequence. Starting from the start time stamp of the facial expression action, the action segments are read from the standardized face image sequence one by one until the end time stamp, and a plurality of action segments are extracted to form an action segment set.

[0009] As a preferred scheme of the facial paralysis recognition and level evaluation method based on the expression analysis technology, the face images in the action segment set are subjected to geometric alignment and normalization processing, and the corresponding two-dimensional expression feature vectors are extracted, and the specific steps are as follows, The face feature points in the action segment set are detected one by one to determine the spatial position layout of each frame of face and form a face feature point set; Based on a preset reference face template, geometric alignment processing is performed on each frame of face image according to the face feature point set to obtain an aligned face image set; The normalized face image set is normalized, and the normalized face image is quantitatively coded to generate a two-dimensional expression feature vector.

[0010] As a preferred scheme of the facial paralysis recognition and grade evaluation method based on the expression analysis technology, wherein: the three-dimensional face reconstruction is performed according to the two-dimensional expression feature vector and the face image in the action segment set, and the specific steps are as follows, The two-dimensional expression feature vector and the preset three-dimensional reference face template are established to determine the face deformation control parameter. According to the face deformation control parameter and the face image in the action segment set, the preset three-dimensional reference face template is processed frame by frame to generate a three-dimensional face reconstruction sequence.

[0011] As a preferred scheme of the facial paralysis recognition and grade evaluation method based on the expression analysis technology, wherein: the three-dimensional muscle group activation field is constructed according to the face muscle group division rule, and the specific steps are as follows, The three-dimensional face surface in the three-dimensional face reconstruction sequence is divided into a plurality of three-dimensional regions corresponding to different face muscle groups to form a three-dimensional muscle group division result. The three-dimensional face surface in the three-dimensional face reconstruction sequence is divided into a plurality of three-dimensional regions corresponding to different face muscle groups to form a three-dimensional muscle group division result.

[0012] As a preferred scheme of the facial paralysis recognition and grade evaluation method based on the expression analysis technology, wherein: the residual severity index is formed, and the specific steps are as follows, Based on the muscle group activation intensity in the three-dimensional muscle group activation field, the three-dimensional face surface is combined to inversely estimate the correction deformation of the face image in the action segment set to form a three-dimensional correction deformation parameter. According to the three-dimensional correction deformation parameter, the abnormal muscle group deformation of the three-dimensional face surface in the three-dimensional face reconstruction sequence is inversely offset, and the projection is performed under the face imaging condition to generate a healthy twin face. The healthy twin face is normalized to generate a two-dimensional twin face feature vector. The two-dimensional twin face feature vector and the two-dimensional expression feature vector are calculated for feature difference, and the residual feature vector is quantitatively processed according to the preset residual aggregation rule to obtain the residual severity index.

[0013] As a preferred scheme of the facial paralysis recognition and grade evaluation method based on the expression analysis technology, wherein: the output facial paralysis recognition result and the graph grade feature vector are output, and the specific steps are as follows, Each facial muscle group in the three-dimensional muscle activation field is taken as a muscle node, and local expression features corresponding to the muscle node are extracted from the two-dimensional expression feature vector; The three-dimensional muscle activation information, the local expression features, and the residual severity index corresponding to the muscle node are spliced and fused to form a muscle node initial feature vector, a connection edge is established between each muscle node, and a muscle correlation residual feature map is constructed based on the muscle node initial feature vector and the connection edge; The muscle correlation residual feature map is input into a graph neural network for multi-layer feature propagation and aggregation operation to form high-order feature representations, and the graph-level feature vector is generated by global pooling of all high-order feature representations; The graph-level feature vector is input into a multi-task reasoning rule to output a facial paralysis recognition result by performing a facial paralysis discrimination task in a unified feature space.

[0014] As a preferred scheme of the facial paralysis recognition and grade evaluation method based on the expression analysis technology, the output facial paralysis severity grade is obtained by matching and comparing the graph-level feature vector with the grade feature representation template, obtaining the grade matching degree, and selecting the highest grade matching degree corresponding to the severity grade after descending order sorting of the grade matching degree.

[0015] In a second aspect, the present application provides a facial paralysis recognition and grade evaluation system based on expression analysis technology, which comprises the facial paralysis recognition and grade evaluation method based on expression analysis technology, and is characterized in that it comprises, An expression sequence acquisition module is configured to acquire a facial expression video sequence, perform face detection and face region cropping on the facial expression video sequence, and cut the facial expression video sequence into a plurality of action segments to obtain an action segment set. A face feature extraction module is configured to perform geometric alignment and normalization processing on face images in the action segment set, and extract corresponding two-dimensional expression feature vectors. A three-dimensional reconstruction activation module is configured to perform three-dimensional face reconstruction based on the two-dimensional expression feature vectors and the face images in the action segment set, and construct a three-dimensional muscle activation field according to a facial muscle group division rule. A healthy twin correction module is configured to perform virtual correction on the face images in the action segment set based on the three-dimensional muscle activation field to generate healthy twin faces, and calculate feature residuals between the face images and the healthy twin faces to form residual severity indexes. A graph feature evaluation module is configured to perform feature fusion on the residual severity indexes, the three-dimensional muscle activation field, and the two-dimensional expression feature vectors to form a muscle correlation residual feature map, perform graph-level feature aggregation and multi-task reasoning on the muscle correlation residual feature map by a graph neural network to output a facial paralysis recognition result and a graph-level feature vector, and perform multi-classification prediction on the graph-level feature vector to output a facial paralysis severity grade.

[0016] The application has the beneficial effects that: by constructing a three-dimensional muscle activation field and generating an individualized healthy twin face on this basis, the abnormal deformation is accurately quantified with the patient's own as a control, and an objective and reliable residual severity index is obtained; meanwhile, a muscle group correlation residual feature map is constructed with the muscle group as a node, and a graph neural network is introduced for high-order feature aggregation, so that the recognition not only focuses on local muscle group abnormalities, but also can capture overall coordinated imbalance; more accurate and more interpretable facial paralysis recognition and severity grade evaluation are realized, and the reliability and practicality of clinical auxiliary diagnosis are improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Fig. 1 The flowchart of the facial paralysis recognition and grade evaluation method based on expression analysis technology.

[0019] Fig. 2 The schematic diagram of the facial paralysis recognition and grade evaluation system based on expression analysis technology.

[0020] Fig. 3 The flowchart of the two-dimensional expression feature vector extraction process.

[0021] Fig. 4 The flowchart of the facial paralysis recognition result generation. DETAILED DESCRIPTION

[0022] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification.

[0023] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0024] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0025] REFERENCE Figs. 1-4For an embodiment of the present application, the embodiment provides a facial paralysis recognition and grading evaluation method based on expression analysis technology, including the following steps: S1, acquire a facial expression video sequence, perform face detection and face region cropping on the facial expression video sequence, and cut it into a plurality of action segments to obtain an action segment set.

[0026] S1.1, acquire a facial expression video sequence by capturing facial expression actions of a patient and synchronously recording start and end time stamps of each facial expression action and corresponding video frame indexes.

[0027] Further, acquire an original facial expression video stream through a facial expression acquisition device, and establish a "video frame time index table" containing a video frame sequence number and a video frame time stamp for each frame; mark the start and end time stamps of each facial expression action during the acquisition process to form a "facial expression action marking table"; retrieve the corresponding video frame sequence number in the "video frame time index table" according to the time range in the "facial expression action marking table", and extract the corresponding frame segment from the original facial expression video stream to generate a facial expression video sequence.

[0028] S1.2, read the facial expression video sequence frame by frame according to the start and end time stamps, and perform face detection on each frame to determine the face position region in each frame to form a face region frame sequence.

[0029] Further, according to the start and end time stamps corresponding to each facial expression action in the facial expression video sequence, read the video frame images in the facial expression video sequence frame by frame in time sequence; perform face detection on each video frame image to obtain the rectangular bounding box coordinates of the face position in the current video frame image, and store the rectangular bounding box coordinates in association with the corresponding video frame sequence number and video frame time stamp; sequentially concatenate the face rectangular bounding box coordinates of all video frame images in time sequence to form a face region frame sequence.

[0030] S1.3, crop the face region of the facial expression video sequence using the face region frame sequence to obtain face region images, and perform size unification and format unification according to a preset cropping standardization rule to form a standardized face image sequence.

[0031] Further, each frame of video image in the facial expression video sequence is traversed in time sequence, and the video frame image is cropped according to the face rectangle boundary box coordinates corresponding to the video frame image in the face region frame sequence, and a face region image containing only the face region is intercepted, each face region image is input into a preset cropping standardization rule, and the face region image is scaled, edge padded and format converted according to unified image size, image aspect ratio, image resolution and image encoding format, all the face region images after cropping and standardization are arranged in the original video frame order to form a standardized face image sequence with consistent face timing, uniform size and uniform format.

[0032] It should be noted that the cropping standardization rule is obtained by extracting unified face size parameters and proportion parameters and solidifying them into fixed specifications after statistical analysis of a large number of face samples.

[0033] S1.4, starting from the start timestamp of the facial expression action, the action segments are sequentially read from the standardized face image sequence until the end timestamp, a plurality of action segments are extracted, and an action segment set is formed by collection.

[0034] Further, according to the start timestamp and the end timestamp of each facial expression action in the facial expression action marking table, and in combination with the correspondence between the timestamp and the video frame number in the video frame time index table, the start frame number and the end frame number of the standardized face image corresponding to each facial expression action are determined in the standardized face image sequence. According to the determined start frame number and end frame number of the standardized face image, the standardized face images between them are continuously read from the standardized face image sequence to form the standardized face image action segment corresponding to the facial expression action. The standardized face image action segments corresponding to all facial expression actions are collected and organized according to the facial expression action identification, and finally the action segment set required for face expression analysis is formed.

[0035] S2, the face images in the action segment set are geometrically aligned and normalized, and the corresponding two-dimensional expression feature vectors are extracted.

[0036] S2.1, the face images in the action segment set are sequentially subjected to face feature point detection, the spatial position layout of each frame face part is determined, and a face feature point set is formed.

[0037] Furthermore, each frame of standardized face image in the action segment set is traversed in chronological order. Facial feature point detection is performed on each frame of standardized face image to locate the position coordinates of key parts such as eyes, eyebrows, nostrils, nose tip, lips, and jawline. The two-dimensional coordinates of each set of key facial parts are associated and stored with the corresponding action segment identifier and video frame number to obtain the facial feature point data of a single frame. The facial feature point data corresponding to all video frames are collected in chronological order to form a facial feature point set.

[0038] S2.2. Based on the preset reference face template, perform geometric alignment processing on each frame of face image according to the set of face feature points to obtain the aligned face image set.

[0039] Furthermore, based on the standard positions of key features in the preset reference face template, the coordinates of facial feature points corresponding to each frame of the face image in the action segment set are registered one-to-one with the key point positions (such as eyes, eyebrows, nostrils, nose tip, lips, and jawline) of the reference face template, and nonlinear deformation parameters that make the shapes of the two consistent are calculated. The nonlinear deformation parameters are then used to perform a spatial geometric transformation on the current frame of the face image, aligning key features such as eyes, eyebrows, nose, and mouth to the standard positions of the reference face template. After performing the geometric transformation on all frames sequentially, the transformed images are collected in chronological order to form an aligned face image set.

[0040] It should be noted that the reference face template is constructed by statistically analyzing the feature point positions of a large number of normal face samples, aggregating the average or median coordinates of each key part, and generating an average face shape with standardized face proportions and structure.

[0041] S2.3 Normalize the aligned face image set and quantitatively encode the normalized face images to generate a two-dimensional expression feature vector.

[0042] Furthermore, illumination correction is performed on the aligned face image set to eliminate brightness deviations. Then, pixel values ​​are linearly normalized according to a unified range. The aligned face image set is scaled and edge-painted according to a preset unified size to form a normalized face image. In each key facial region of the normalized face image, local texture change features (such as local gray-level contrast, local texture gradient, and subtle wrinkle changes) are extracted by quantifying the local differences in regional gray-level distribution. Contour shape features (including edge shape, curvature changes, or quantitative descriptions of contour geometry extracted along the outer contour of the face and key curves) are extracted at the facial contour positions. All local texture change features and contour shape features are quantized and organized according to the spatial order of facial regions to output a two-dimensional expression feature vector.

[0043] It should be noted that the uniform size is obtained by collecting a large number of original face images, first calculating the average width, average height and average aspect ratio of the face region based on the face region bounding box of each original face image, and then taking the average or median of these statistical results to obtain a set of stable face scale parameters and solidifying them as the target size.

[0044] The spatial order of facial regions is obtained by identifying key facial regions in each sample and numbering them according to a fixed spatial sorting rule from top to bottom and from left to right, based on the spatial layout of these key facial regions in a natural face.

[0045] S3. Perform three-dimensional face reconstruction based on the two-dimensional expression feature vector combined with the face images in the action segment set, and construct a three-dimensional muscle activation field according to the facial muscle group division rules.

[0046] S3.1 Establish a deformation control relationship between the two-dimensional facial feature vector and the preset three-dimensional reference face template, and determine the facial deformation control parameters.

[0047] Furthermore, based on the spatial order of facial regions, a one-to-one feature mapping table is established between the two-dimensional expression feature vector and the facial muscle groups, mesh vertex sets, and deformation control points on the preset three-dimensional reference face template. According to the feature mapping table, the feature values ​​of each dimension in the two-dimensional expression feature vector are converted into deformation control components such as displacement, rotation, and scaling of each deformation control point of the three-dimensional reference face template according to preset linear mapping rules and numerical normalization rules, forming a set of original deformation control parameters. The set of original deformation control parameters is then subjected to range constraints and smoothing. The processed deformation control components are then organized according to the control point numbering order to output the facial deformation control parameters.

[0048] It should be noted that the 3D reference face template is obtained by collecting multiple 3D human face scans, spatially registering all 3D human face scans according to unified 3D key points, and then calculating the average or median coordinates of each point on the aligned 3D human face scans. It serves as the standard basic structure for all deformation control points, facial muscle areas, and surface meshes, enabling 2D expression feature vectors to generate corresponding 3D displacement, rotation, or scaling changes on this fixed shape. This ensures that different action segments and different face samples can perform comparable, controllable, and reproducible deformation operations on a unified 3D structure.

[0049] The linear mapping rule is obtained by collecting multiple sets of corresponding samples of two-dimensional facial expression feature vectors and the actual three-dimensional deformation amplitude, performing linear regression or interval comparison analysis on each two-dimensional feature and the corresponding three-dimensional deformation variable, and then solidifying the linear proportional coefficient and offset coefficient that can stably describe the relationship of "two-dimensional feature change → three-dimensional deformation amplitude".

[0050] The numerical normalization rule is formed by statistically analyzing the minimum, maximum, mean, and standard deviation of a large number of two-dimensional facial feature vectors one dimension at a time, and matching these statistics to the corresponding control parameter range based on the available range of the target deformation control parameters.

[0051] S3.2. Based on the facial deformation control parameters and the facial images in the action segment set, perform frame-by-frame deformation processing on the preset three-dimensional reference facial template to generate a three-dimensional facial reconstruction sequence.

[0052] Furthermore, based on facial deformation control parameters, corresponding displacement, rotation, and scaling are applied to each deformation control point in the preset three-dimensional reference face template frame by frame, so that the three-dimensional reference face template forms a three-dimensional local shape change that matches the two-dimensional expression feature vector in each frame; according to the time sequence of face images in the action segment set, the three-dimensional face surface that has completed deformation frame by frame is arranged in the frame order to form a stereoscopic face reconstruction sequence.

[0053] S3.3 Divide the three-dimensional face surface of each frame in the stereoscopic face reconstruction sequence into multiple three-dimensional regions corresponding to different facial muscle groups to form a stereoscopic muscle group division result.

[0054] Furthermore, the mesh vertices of the 3D face surface in each frame of the 3D face reconstruction sequence are matched one-to-one with the muscle group region indexes in the facial muscle group segmentation template to determine the facial muscle group region to which each mesh vertex belongs. According to the region number of the facial muscle group segmentation template, the set of mesh vertices of the 3D face surface in each frame is classified according to the muscle group region to form 3D muscle group regions corresponding to different facial muscle groups. The segmentation results of all 3D muscle group regions in each frame are sorted in chronological order to form the 3D muscle group segmentation results.

[0055] It should be noted that, based on the medical facial muscle anatomy diagram, the three-dimensional coverage area of ​​the facial expression muscle groups is marked on the standard three-dimensional human face surface; according to the anatomical boundaries of each facial expression muscle group, the grid vertices of the three-dimensional human face surface are assigned to the muscle group regions one by one; the medical facial muscle anatomy diagram is obtained by organizing and drawing after anatomical research and structural measurement of the human face, and is a professional reference diagram used to show the structural position and boundary relationship of each facial expression muscle group, and is used to guide the division of facial regions and the localization of muscle groups.

[0056] S3.4. Using the three-dimensional face surface in a static expression state as a reference, calculate the deformation intensity of each frame in the three-dimensional face reconstruction sequence relative to the reference state in different facial muscle group regions based on the three-dimensional muscle group segmentation results, and collect and construct a three-dimensional muscle group activation field in three-dimensional space.

[0057] Furthermore, by performing 3D reconstruction processing on a relaxed facial image and fixing it as an expressionless reference shape, a 3D facial surface in a static expression state is obtained and used as a reference 3D facial surface. Each grid vertex of the 3D facial surface in each frame of the 3D facial reconstruction sequence is registered one-to-one according to the spatial position corresponding to the 3D muscle group segmentation result. The point-by-point position difference between the grid vertex coordinates of the reference 3D facial surface and the grid vertex coordinates of the current frame's 3D facial surface is measured, and the deformation displacement magnitude is obtained by calculating the spatial distance between the two sets of coordinate points. Based on the set of grid vertices contained in each facial muscle group region in the 3D muscle group segmentation result, the deformation displacement magnitude of all grid vertices belonging to the same facial muscle group region in each frame's 3D facial surface is aggregated to obtain the deformation intensity value. In the 3D coordinate space of the 3D facial surface, the deformation intensity values ​​of each facial muscle group region corresponding to each frame are mapped back to their respective 3D facial muscle group region positions according to the 3D muscle group segmentation result, and then aggregated according to the time sequence of the 3D facial reconstruction sequence to form a 3D muscle group activation field.

[0058] It should be noted that the numerical value of the deformation intensity is obtained by performing aggregation operations, and the expression is: ; In the formula, Indicates the first In the time frame and the first The numerical value of the deformation intensity of each facial muscle group region is used to describe the overall degree of deformation of the facial muscle group region in the current frame relative to the static expression state. Indicating the first facial muscle group segmentation template The number of grid vertices contained in each facial muscle group region; It is in the Belonging to the first time frame The first facial muscle group area The magnitude of the deformation displacement of each grid vertex; It is the index of the grid vertex; These are the facial muscle group area numbers; It is the index of the time frame.

[0059] S4. Based on the three-dimensional muscle activation field, the face images in the action segment set are virtually corrected to generate healthy twin faces, and the feature residuals between the face images and the healthy twin faces are calculated to form a residual severity index.

[0060] S4.1 Based on the activation intensity of each muscle group in the three-dimensional muscle activation field, combined with the three-dimensional face surface, the correction deformation of the face image in the action segment set is back-estimated to form three-dimensional correction deformation parameters.

[0061] Furthermore, the set of mesh vertices corresponding to each facial muscle group region is located on the 3D face surface, and the muscle group activation intensity is used as the inverse correction target to indicate the degree of deformation change that needs to be rectified. The position of the mesh vertex on the 3D face surface of the current frame in the stereo face reconstruction sequence is used to estimate the correction deformation in the direction of the position of the reference 3D face surface. In each facial muscle group region, a stereo correction displacement is generated for each mesh vertex that is opposite to the original deformation direction and whose size is related to the corresponding muscle group activation intensity value. The stereo correction displacements of the mesh vertices in each facial muscle group region in each frame are organized and parameterized according to the facial muscle group region number and the time frame number, and the stereo correction deformation parameters are output.

[0062] S4.2 Based on the stereoscopic correction deformation parameters, the abnormal muscle group deformation of the three-dimensional face surface in each frame of the stereoscopic face reconstruction sequence is reversed and projected under the face imaging conditions to generate a healthy twin face.

[0063] Furthermore, based on the stereoscopic correction deformation parameters, a correction displacement operation is performed on the mesh vertices of each facial muscle group region on the surface of the three-dimensional face in each frame of the stereoscopic face reconstruction sequence. The mesh vertices are adjusted in position along the spatial direction opposite to the abnormal muscle group deformation, and the adjusted mesh vertex positions are gradually brought closer to the corresponding mesh vertex positions on the surface of the three-dimensional face in a static expression state. This reverses the cancellation of abnormal muscle group deformation on the surface of the three-dimensional face. The three-dimensional face surface of each frame after being corrected by the stereoscopic correction deformation parameters is aligned with the face images in the action segment set under the face imaging conditions. Using the same viewpoint parameters, projection parameters, and imaging plane coordinate relationships as the face images in the action segment set, the corrected three-dimensional face surface of each frame is projected onto the two-dimensional image plane under the face imaging conditions. Healthy twin faces corresponding to each face image are generated according to the time sequence of the face images in the action segment set.

[0064] It should be noted that the face imaging conditions are obtained by analyzing the facial key point positions in the face images in the action segment set, and inferring the viewing direction, imaging scale and projection relationship based on the spatial distribution of the key points.

[0065] Abnormal muscle group deformation is calculated by comparing the spatial displacement difference of the grid vertices in each muscle group region in each frame of the stereo face reconstruction sequence with respect to the reference state. This calculation represents the abnormal muscle group deformation of the muscle group in the current frame relative to the normal state.

[0066] S4.3 Normalize the healthy twin faces to generate two-dimensional twin facial feature vectors.

[0067] Furthermore, the healthy twin faces are normalized according to the rules of illumination correction, pixel value normalization, and uniform size scaling to obtain normalized healthy twin face images. Local texture features and contour shape features are extracted from the normalized healthy twin face images according to the spatial order of facial regions, and output as two-dimensional twin face feature vectors in a fixed arrangement.

[0068] S4.4 Calculate the feature difference between the two-dimensional twin facial feature vector and the two-dimensional expression feature vector, and quantize the residual feature vector according to the preset residual aggregation rule to obtain the residual severity index.

[0069] Furthermore, the difference operation is performed on the two-dimensional twin facial feature vector and the two-dimensional expression feature vector in the corresponding feature dimensions to obtain the residual feature vector composed of the feature difference of each dimension. The feature difference of each dimension in the residual feature vector is classified and organized according to the facial region to generate residual feature groups. According to the preset residual aggregation rules, the residual feature group corresponding to each facial region is aggregated within each facial region to obtain the regional residual intensity value. The regional residual intensity values ​​corresponding to all facial regions are aggregated again, and the overall residual intensity value is mapped to the preset evaluation interval through normalization to output the residual severity index.

[0070] It should be noted that the residual aggregation rule is obtained by statistically comparing the residual feature distributions of a large number of normal face samples and abnormal face samples, and selecting the aggregation method that can stably reflect the degree of regional deviation from the statistical results. The evaluation interval is obtained by statistically analyzing the residual severity index of a large number of normal and abnormal face samples, and determining the upper and lower bounds based on the distribution range.

[0071] S5. The residual severity index is fused with the three-dimensional muscle group activation field and the two-dimensional facial expression feature vector to form a muscle group association residual feature map. The graph neural network is used to perform graph-level feature aggregation and multi-task reasoning to output the facial paralysis recognition result and graph-level feature vector. The graph-level feature vector is then used for multi-class prediction to output the facial paralysis severity level.

[0072] S5.1. Take each facial muscle group in the three-dimensional muscle activation field as a muscle group node, and extract the local expression features corresponding to the muscle group node from the two-dimensional expression feature vector.

[0073] Furthermore, based on the facial muscle group segmentation results in the three-dimensional muscle group activation field, each facial muscle group region is set as a muscle group node, and a unique muscle group node number is assigned to each muscle group node. According to the mapping relationship table between facial muscle group regions and the spatial order of facial regions, the feature dimension index range corresponding to each muscle group node is located in the two-dimensional expression feature vector. According to the feature dimension index range, the corresponding local expression features are extracted from the two-dimensional expression feature vector.

[0074] S5.2. The three-dimensional muscle activation information, local expression features and residual severity index corresponding to the muscle group nodes are spliced ​​and fused to form the initial feature vector of the muscle group nodes. Connection edges are established between each muscle group node, and a muscle group association residual feature map is constructed based on the initial feature vector of the muscle group nodes and the connection edges.

[0075] Furthermore, three-dimensional muscle group activation information, local expression features, and residual severity index are concatenated along the feature dimension to form initial feature vectors for muscle group nodes. Based on the spatial adjacency and functional association relationships of facial muscle groups recorded in the facial muscle group segmentation template, muscle group node connection edges are established between two muscle group nodes that are spatially adjacent or functionally synergistic. All muscle group node connection edges are organized according to the connection start and connection end points to form a set of muscle group node connection edges. All muscle group nodes and their corresponding initial feature vectors are used as a node set, and the set of muscle group node connection edges is used as an edge set. The initial feature vectors of muscle group nodes and the connection edges of muscle group nodes are organized in a unified graph structure representation to form a muscle group association residual feature graph.

[0076] S5.3 Input the muscle group association residual feature map into the graph neural network and perform multi-layer feature propagation and aggregation operations to form a high-order feature representation. Generate graph-level feature vectors by global pooling of all high-order feature representations.

[0077] Furthermore, the residual feature map of muscle group association is used as the input to the graph neural network (Graph Neural Network), the initial feature vectors of muscle group nodes are used as the node input features of the Graph Neural Network, and the set of edges connecting muscle group nodes is used as the topological structure description of the Graph Neural Network, thus completing the correspondence between the residual feature map of muscle group association and the input structure of the Graph Neural Network. In multiple feature propagation layers, the Graph Neural Network receives the feature information of neighboring muscle group nodes from adjacent muscle group nodes according to the set of edges connecting muscle group nodes. It then weights and combines the feature information of neighboring muscle group nodes with the feature information of the current muscle group node, updating and obtaining a progressively higher-order feature representation of the muscle group nodes. After completing the feature propagation and feature aggregation operations at a preset number of layers, a global pooling operator, such as global average pooling or global max pooling, is applied to all higher-order feature representations of muscle group nodes, compressing all higher-order feature representations of muscle group nodes into a fixed-length vector as the output graph-level feature vector.

[0078] It should be noted that the training process of the graph neural network collects muscle group association residual feature maps with facial paralysis recognition result labels and facial paralysis severity level labels as training samples. Each muscle group association residual feature map is input into the graph neural network to obtain the corresponding prediction result. The prediction result and the labels in the training samples are used to construct a loss function. The trainable parameters in the graph neural network are iteratively updated through the backpropagation algorithm, so that the loss function is continuously reduced until the training process reaches the convergence condition.

[0079] S5.4 Input the graph-level feature vector into the multi-task inference rule, and output the facial paralysis recognition result by performing the facial paralysis discrimination task in a unified feature space.

[0080] Furthermore, a facial paralysis discrimination sub-rule is constructed based on the correspondence between graph-level feature vectors in the training samples and the labels of facial paralysis recognition results, and a facial paralysis severity level discrimination sub-rule is constructed based on the correspondence between graph-level feature vectors and facial paralysis severity level labels; the facial paralysis discrimination sub-rule and the facial paralysis severity level discrimination sub-rule are combined to form a multi-task inference rule; The graph-level feature vector is input into the facial paralysis discrimination sub-rule. The features in each dimension of the graph-level feature vector are weighted and combined according to the facial paralysis discrimination weight to obtain the facial paralysis discrimination score. The facial paralysis discrimination score is compared with the preset facial paralysis discrimination threshold. When the facial paralysis discrimination score is greater than the facial paralysis discrimination threshold, the facial paralysis identification result is output as positive. When the facial paralysis discrimination score is less than or equal to the facial paralysis discrimination threshold, the facial paralysis identification result is output as negative.

[0081] It should be noted that the facial paralysis discrimination weight coefficients were obtained by performing iterative optimization on training samples labeled with facial paralysis recognition results; The facial paralysis discrimination threshold is obtained by calculating the distribution of facial paralysis discrimination score values ​​on training samples labeled with facial paralysis recognition results; the value range of the facial paralysis discrimination threshold is determined to be between 0 and 1 based on the minimum and maximum values ​​of facial paralysis discrimination score values ​​in the training samples.

[0082] S5.5. Match and compare each of the graph-level feature vectors with the grade feature representation template to obtain the grade matching degree. Sort the grade matching degrees in descending order and select the severity grade corresponding to the highest grade matching degree to output the severity grade of facial paralysis.

[0083] Furthermore, the graph-level feature vectors are sequentially compared with each level feature representation template using a dimensional difference or similarity metric along a unified feature dimension. The level matching degree is obtained through methods such as cosine similarity or inverse distance mapping, expressed as: ; In the formula, It is a level identifier The corresponding level matching degree is used to measure the degree of matching between the graph-level feature vector and the level feature representation template; The graph-level eigenvector is at the th... The square of the feature values ​​in each feature dimension; The graph-level eigenvector is at the th... Feature values ​​in each feature dimension; It is a level identifier The corresponding level feature representation template is in the first Feature values ​​in each feature dimension; It is a level identifier The corresponding level feature representation template is in the first The square of the feature values ​​in each feature dimension; It is from 1 to Feature dimension index within the range; It refers to the number of feature dimensions; It is a level indicator; All level matching scores are grouped according to level identifiers to form a level matching score list consisting of level identifiers and corresponding level matching scores. The level matching scores in the level matching score list are sorted in descending order of numerical value, and the level identifier with the highest level matching score in the sorted level matching score sequence is selected as the severity level of facial paralysis.

[0084] It should be noted that the graded feature representation template divides the training samples into multiple graded sample sets according to the severity level label of facial paralysis; within each graded sample set, the values ​​of all graph-level feature vectors in each feature dimension are statistically averaged or statistically medianized to obtain feature values, and each statistical feature value is obtained by arranging them in a unified feature dimension order.

[0085] The severity level is determined by organizing the facial paralysis severity level labels of the training samples and assigning a unique number to each severity level.

[0086] This embodiment also provides a facial paralysis recognition and level assessment system based on expression analysis technology, including: an expression sequence acquisition module, used to acquire facial expression video sequences, perform face detection and face region cropping on the facial expression video sequences, and divide them into several action segments to acquire a set of action segments; The facial feature extraction module is used to perform geometric alignment and normalization on facial images in the action segment set, and extract the corresponding two-dimensional expression feature vectors. The 3D reconstruction activation module is used to perform 3D face reconstruction based on the 2D facial feature vector combined with the face images in the action segment set, and to construct a 3D muscle activation field according to the facial muscle group division rules. The health twin correction module is used to virtually correct facial images in a set of action segments based on a three-dimensional muscle activation field, generate a health twin face, and calculate the feature residual between the facial image and the health twin face to form a residual severity index. The graph feature assessment module is used to fuse the residual severity index with the three-dimensional muscle group activation field and the two-dimensional facial expression feature vector to form a muscle group association residual feature map. The graph neural network is used to perform graph-level feature aggregation and multi-task reasoning to output the facial paralysis recognition result and graph-level feature vector. The graph-level feature vector is then used for multi-classification prediction to output the facial paralysis severity level.

[0087] In summary, this invention achieves more accurate and interpretable facial paralysis identification and severity assessment by: constructing a three-dimensional muscle activation field and generating an individualized healthy twin face based on it, thereby accurately quantifying abnormal deformation using the patient as a control and obtaining an objective and reliable residual severity index; simultaneously, by constructing a muscle group association residual feature map with muscle groups as nodes and introducing a graph neural network for high-order feature aggregation, the identification not only focuses on local muscle group abnormalities but also captures overall synergistic imbalances; thus improving the reliability and practicality of clinical auxiliary diagnosis.

[0088] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for identifying and grading facial paralysis based on facial expression analysis technology, characterized in that: The application relates to a facial expression video sequence acquisition method and device, a facial expression video sequence processing method and device, a facial expression video sequence facial paralysis recognition method and device, and a facial expression video sequence facial paralysis recognition system. The application relates to a facial expression video sequence acquisition method and device, a facial expression video sequence processing method and device, a facial expression video sequence facial paralysis recognition method and device, and a facial expression video sequence facial paralysis recognition system. The facial expression video sequence is acquired by capturing facial expression actions of a patient and synchronously recording start and end time stamps and corresponding video frame indexes of each facial expression action. The facial expression video sequence is read frame by frame according to the start and end time stamps, and face detection is performed on each frame to determine the face position area in each frame, thereby forming a face area frame sequence. The face area frame sequence is used to crop the face area of the facial expression video sequence, face area images are acquired, and size unification and format unification are performed according to preset cropping standardization rules, thereby forming a standardized face image sequence. The start time stamp of the facial expression action is used to read action segments from the standardized face image sequence one by one until the end time stamp, a plurality of action segments are extracted, and the action segment set is formed by collection.

2. The method of claim 1, wherein the method comprises: The face image in the action segment set is geometrically aligned and normalized, and the corresponding two-dimensional expression feature vector is extracted.

3. The method of claim 2, wherein the method further comprises: The two-dimensional expression feature vector is combined with the face image in the action segment set to perform three-dimensional face reconstruction, and a three-dimensional muscle group activation field is constructed according to a face muscle group division rule. Virtual correction is performed on the face image in the action segment set based on the three-dimensional muscle group activation field, a healthy twin face is generated, a feature residual error between the face image and the healthy twin face is calculated, and a residual error severity index is formed. The residual error severity index is fused with the three-dimensional muscle group activation field and the two-dimensional expression feature vector to form a muscle group associated residual feature map, graph neural network is used for graph-level feature aggregation and multi-task reasoning, a facial paralysis recognition result and a graph-level feature vector are output, and the graph-level feature vector is subjected to multi-classification prediction to output a facial paralysis severity level. The facial expression video sequence is acquired by capturing facial expression actions of a patient and synchronously recording start and end time stamps and corresponding video frame indexes of each facial expression action.

4. The method of claim 3, wherein the method further comprises: The facial expression video sequence is read frame by frame according to the start and end time stamps, and face detection is performed on each frame to determine the face position area in each frame, thereby forming a face area frame sequence. The face area frame sequence is used to crop the face area of the facial expression video sequence, face area images are acquired, and size unification and format unification are performed according to preset cropping standardization rules, thereby forming a standardized face image sequence. The start time stamp of the facial expression action is used to read action segments from the standardized face image sequence one by one until the end time stamp, a plurality of action segments are extracted, and the action segment set is formed by collection. The face image in the action segment set is geometrically aligned and normalized, and the corresponding two-dimensional expression feature vector is extracted.

5. The method of facial paralysis recognition and grading based on facial analysis technology as claimed in claim 4, wherein: The two-dimensional expression feature vector is combined with the face image in the action segment set to perform three-dimensional face reconstruction, and a three-dimensional muscle group activation field is constructed according to a face muscle group division rule. Virtual correction is performed on the face image in the action segment set based on the three-dimensional muscle group activation field, a healthy twin face is generated, a feature residual error between the face image and the healthy twin face is calculated, and a residual error severity index is formed. The residual error severity index is fused with the three-dimensional muscle group activation field and the two-dimensional expression feature vector to form a muscle group associated residual feature map, graph neural network is used for graph-level feature aggregation and multi-task reasoning, a facial paralysis recognition result and a graph-level feature vector are output, and the graph-level feature vector is subjected to multi-classification prediction to output a facial paralysis severity level.

6. The method of facial paralysis recognition and grading using facial analysis technology as claimed in claim 1, wherein: The facial expression video sequence is acquired by capturing facial expression actions of a patient and synchronously recording start and end time stamps and corresponding video frame indexes of each facial expression action. The facial expression video sequence is read frame by frame according to the start and end time stamps, and face detection is performed on each frame to determine the face position area in each frame, thereby forming a face area frame sequence. The face area frame sequence is used to crop the face area of the facial expression video sequence, face area images are acquired, and size unification and format unification are performed according to preset cropping standardization rules, thereby forming a standardized face image sequence. The start time stamp of the facial expression action is used to read action segments from the standardized face image sequence one by one until the end time stamp, a plurality of action segments are extracted, and the action segment set is formed by collection. The face image in the action segment set is geometrically aligned and normalized, and the corresponding two-dimensional expression feature vector is extracted. The two-dimensional expression feature vector is combined with the face image in the action segment set to perform three-dimensional face reconstruction, and a three-dimensional muscle group activation field is constructed according to a face muscle group division rule. Virtual correction is performed on the face image in the action segment set based on the three-dimensional muscle group activation field, a healthy twin face is generated, a feature residual error between the face image and the healthy twin face is calculated, and a residual error severity index is formed. The residual error severity index is fused with the three-dimensional muscle group activation field and the two-dimensional expression feature vector to form a muscle group associated residual feature map, graph neural network is used for graph-level feature aggregation and multi-task reasoning, a facial paralysis recognition result and a graph-level feature vector are output, and the graph-level feature vector is subjected to multi-classification prediction to output a facial paralysis severity level. The three-dimensional face surface of each frame in the stereoscopic face reconstruction sequence is divided into a plurality of three-dimensional regions corresponding to different facial muscle groups, forming a stereoscopic muscle group division result; With the three-dimensional face surface in a static expression state as a reference, the deformation change intensity of each frame in the stereoscopic face reconstruction sequence in different facial muscle group regions relative to the reference state is calculated according to the stereoscopic muscle group division result, and the three-dimensional muscle group activation field is constructed by collecting in the three-dimensional space.

7. The method of facial paralysis recognition and grading based on facial analysis technology as claimed in claim 1, wherein: The residual severity index is formed, and the specific steps are as follows, Based on the activation intensity of each muscle group in the three-dimensional muscle group activation field, the virtual correction deformation of the face image in the action segment set is inversely estimated in combination with the three-dimensional face surface, and a stereoscopic correction deformation parameter is formed; According to the stereoscopic correction deformation parameter, the abnormal muscle group deformation of the three-dimensional face surface of each frame in the stereoscopic face reconstruction sequence is inversely offset, and the healthy twin face is generated by projection under the face imaging condition; The healthy twin face is normalized to generate a two-dimensional twin facial feature vector; The two-dimensional twin facial feature vector and the two-dimensional expression feature vector are calculated for feature difference, and the residual feature vector is quantitatively processed according to the preset residual aggregation rule to obtain the residual severity index.

8. The method of facial paralysis recognition and grading based on facial analysis technology as claimed in claim 1, wherein: The output facial paralysis recognition result and the graph-level feature vector, and the specific steps are as follows, Each facial muscle group in the three-dimensional muscle group activation field is taken as a muscle group node, and the local expression feature corresponding to the muscle group node is extracted from the two-dimensional expression feature vector; The three-dimensional muscle group activation information, the local expression feature and the residual severity index corresponding to the muscle group node are spliced and fused to form a muscle group node initial feature vector, a connection edge is established between each muscle group node, and a muscle group associated residual feature graph is constructed based on the muscle group node initial feature vector and the connection edge; The muscle group associated residual feature graph is input into the graph neural network for multi-layer feature propagation and aggregation operation to form a high-order feature representation, and the graph-level feature vector is generated by global pooling of all high-order feature representations; The graph-level feature vector is input into the multi-task reasoning rule to output the facial paralysis recognition result by executing the facial paralysis discrimination task in the unified feature space.

9. The method of claim 8, wherein the method further comprises: determining a degree of the facial paralysis based on the facial expression analysis. The output facial paralysis severity level means that the graph-level feature vector is matched and compared with the grade feature representation template item by item to obtain the grade matching degree, and the highest grade matching degree is selected after the grade matching degree is sorted in descending order. The severity level corresponding to the highest grade matching degree.

10. The facial paralysis recognition and grading system based on facial expression analysis technology according to any one of claims 1 to 9, characterized in that: It includes, The expression sequence acquisition module is used to acquire a facial expression video sequence, perform face detection and face region cropping on the facial expression video sequence, and cut into a plurality of action segments to obtain an action segment set; The face feature extraction module is used to perform geometric alignment and normalization processing on the face images in the action segment set, and extract corresponding two-dimensional expression feature vectors; The three-dimensional reconstruction activation module is used to perform three-dimensional face reconstruction according to the two-dimensional expression feature vectors in combination with the face images in the action segment set, and construct a three-dimensional muscle group activation field according to a facial muscle group division rule; The healthy twin correction module is used to perform virtual correction on the face images in the action segment set based on the three-dimensional muscle group activation field, generate a healthy twin face, and calculate the feature residual between the face images and the healthy twin face to form a residual severity index; The graph feature evaluation module is configured to perform feature fusion on the residual severity index, the three-dimensional muscle group activation field and the two-dimensional expression feature vector to form a muscle group associated residual feature graph, perform graph-level feature aggregation and multi-task reasoning on the muscle group associated residual feature graph through a graph neural network, output a facial paralysis recognition result and a graph-level feature vector, and perform multi-classification prediction on the graph-level feature vector to output a facial paralysis severity level.

Images