Single facial paralysis multi-branch evaluation method, system, medium and equipment based on multi-feature fusion and multi-task learning

Abstract

The present application relates to the field of computer vision, and discloses a unilateral facial paralysis multi-branch evaluation method and system based on multi-feature fusion and multi-task learning, medium and equipment, which comprises: after pre-processing the collected front face designated expression action video, retaining the video frame in which the face region can be detected; extracting the local dynamic symmetry feature of the target facial nerve branch region and the global static face feature in the limit action state, and performing feature fusion; the fused features are added and merged with the global static face feature to obtain the merged shared features; the shared features are input into the classification subnetwork and the regression subnetwork at the same time to obtain the classification probability value of whether facial paralysis and the preliminary regression estimation value of the severity of facial paralysis; combining the classification probability value with the learnable parameters, the preliminary regression estimation value is enhanced regression to obtain the final regression estimation value; the regression estimation value is data post-processed to obtain the final evaluation level of the severity of facial paralysis.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method, system, medium, and device for multi-branch assessment of unilateral facial paralysis based on multi-feature fusion and multi-task learning. Background Technology

[0002] Unilateral peripheral facial paralysis (UPFP) is a facial dysfunction caused by damage to the facial nerve nucleus and downstream nerves on one side. It typically manifests as weakness in closing the eye, frowning, puffing out the cheeks, showing teeth, and closing the lips on the affected side, as well as deviation of the corner of the mouth to the opposite side. It severely impacts the patient's function, appearance, and mental health. Currently, clinical diagnosis of facial paralysis relies primarily on the physician's subjective judgment. The patient is instructed to perform specific facial movements such as raising the eyebrows, closing the eyes, puffing out the cheeks, and showing teeth. The physician then grades the severity of the facial paralysis based on the asymmetry observed in these movements.

[0003] Compared to clinical methods, computer-aided methods often offer promising alternatives by providing more objective and effective assessments. With advancements in deep learning, related technologies have been widely applied in medical image analysis, such as medical image segmentation, computer-aided diagnosis, and detection. In the field of facial paralysis detection, many researchers utilize deep learning methods to automate facial paralysis detection and severity assessment. Compared to traditional methods, deep learning-based detection methods demonstrate higher accuracy and efficiency, providing valuable support for clinicians.

[0004] Existing deep learning-based methods for facial paralysis detection and grading can generally be divided into three categories: based on feature point coordinate differences, based on static image symmetry differences, or based on dynamic facial expression changes. Existing dynamic video-based facial paralysis detection algorithms have made significant progress in facial paralysis grading detection; however, these methods only use single features, do not consider the fusion of multiple features, and only provide a comprehensive rating and evaluation of the overall performance of unilateral facial paralysis. In the field of oral and maxillofacial head and neck surgery, unilateral peripheral facial paralysis usually involves damage to one or more unilateral peripheral branches of the facial nerve. Assessing the overall function of the unilateral facial nerve cannot accurately reflect the damage and recovery of individual branches. Currently, there is no automated detection method capable of performing multi-level assessments of different facial nerve branches.

[0005] In summary, the research demonstrates that a multi-level assessment method for unilateral facial paralysis that effectively integrates and utilizes various features and targets different facial nerve branches has significant medical clinical value and social benefits. Therefore, a new algorithm for assessing the severity of unilateral facial paralysis is urgently needed to address current pressing clinical needs and further improve the accuracy of grading. Summary of the Invention

[0006] To address the aforementioned problems, the purpose of this invention is to provide a method, system, medium, and device for assessing unilateral facial paralysis by multiple branches based on multi-feature fusion and multi-task learning. This method can process multi-feature extraction and fusion in facial videos and can also perform multi-level assessments of the degree of facial paralysis for different facial nerve branches.

[0007] To achieve the above objectives, in a first aspect, the technical solution adopted by the present invention is as follows: a multi-branch assessment method for unilateral facial paralysis based on multi-feature fusion and multi-task learning, comprising: preprocessing the acquired frontal facial expression and action video, and retaining video frames in which the facial region can be detected; uniformly sampling the preprocessed video frame sequence, and extracting the local dynamic symmetry features and the global static facial features under extreme action states of the target facial nerve branch region; fusing the local dynamic symmetry features and the global static facial features, and adding and merging them with the global static facial features to obtain the merged shared features; simultaneously inputting the shared features into the classification subnetwork and the regression subnetwork to obtain the classification probability value of whether facial paralysis occurs and the preliminary regression estimate of the severity of facial paralysis; combining the classification probability value and learnable parameters to perform enhanced regression on the preliminary regression estimate to obtain the final regression estimate; and performing data post-processing on the regression estimate to obtain the final facial paralysis severity assessment level.

[0008] Furthermore, after preprocessing the collected frontal facial expression and action video, the video frames in which the face region can be detected are retained. This includes: using OpenCV to extract video frames, and using Dlib to perform face detection, alignment and cropping, retaining the coordinate information of 68 sets of detected facial feature points, and reading the facial nerve branch paralysis degree label corresponding to the expression and action video.

[0009] Furthermore, local dynamic symmetry features and global static facial features under extreme action states are extracted from the target facial nerve branch regions, including:

[0010] After uniform sampling, the keyframe sequence is extracted. Based on expert priors, the facial bounding box corresponding to each facial nerve branch is extracted from the keyframe sequence as the region corresponding to the facial nerve branch.

[0011] The area corresponding to the facial nerve branch is symmetrically divided into left and right parts. The left half is subtracted from the mirrored right half to obtain the symmetry difference information of the target facial area.

[0012] The adjusted local symmetry difference image is used to extract spatial features through a convolutional neural network, and then temporal features are extracted through the transformer's encoder structure to obtain a local symmetry difference image sequence.

[0013] The intermediate frame is selected from the keyframe sequence as the global facial expression reference under extreme action conditions, and the image scale is readjusted to obtain the global face image.

[0014] Furthermore, feature fusion is performed between local dynamic symmetric features and global static face features, including: using the transformer's decoder structure as a multi-feature fusion module, using the obtained local dynamic symmetric features as the Query, and using the obtained global static face features as the Key and Value, to obtain the fused features.

[0015] Furthermore, the shared features are simultaneously input into the classification subnetwork and the regression subnetwork to obtain the classification probability value of whether facial paralysis is present and the preliminary regression estimate of the severity of facial paralysis. This includes: using a multi-task learning approach, simultaneously performing a binary classification task b for facial paralysis detection and a regression task r for facial paralysis intensity estimation. The merged features are then fed as shared features into both the classification subnetwork and the regression subnetwork to obtain the probability value Y of being diagnosed with facial paralysis. b Non-negative facial paralysis assessment score Y′ r .

[0016] Furthermore, the classification subnetwork consists of a fully connected layer and a softmax layer forming a binary classification head, which outputs the probability value Y of being diagnosed with facial paralysis. b ;

[0017] The regression subnetwork consists of a fully connected layer and a ReLU layer as the regression head, outputting a non-negative facial paralysis assessment score Y′. r ;

[0018] Introducing a learnable threshold parameter θ t and proportional parameter θ s Combined with binary classification prediction probability Y b For the initial regression value Y′ r Perform enhanced regression.

[0019] Furthermore, the regression estimates are post-processed to obtain the final severity assessment level of facial paralysis, including:

[0020] For binary classification tasks, if the output probability value exceeds the learned threshold parameter, it is judged as a positive result for facial paralysis; otherwise, it is judged as a negative result for facial paralysis.

[0021] For regression tasks, the regression estimates obtained after augmentation will be used to classify the degree of facial paralysis according to the interval they fall into.

[0022] Secondly, the technical solution adopted by this invention is as follows: a multi-branch assessment system for unilateral facial paralysis based on multi-feature fusion and multi-task learning, comprising: a preprocessing module, which preprocesses the acquired frontal facial expression and action video and retains video frames in which the facial region can be detected; a feature extraction module, which uniformly samples the preprocessed video frame sequence and extracts the local dynamic symmetry features of the target facial nerve branch region and the global static facial features under extreme action states; a feature fusion and merging module, which fuses the local dynamic symmetry features and the global static facial features and adds them to the global static facial features to obtain the merged shared features; a sub-network processing module, which simultaneously inputs the shared features into the classification sub-network and the regression sub-network to obtain the classification probability value of whether facial paralysis is present and the preliminary regression estimate of the severity of facial paralysis, and combines the classification probability value and learnable parameters to perform enhanced regression on the preliminary regression estimate to obtain the final regression estimate; and an assessment module, which performs data post-processing on the regression estimate to obtain the final assessment level of the severity of facial paralysis.

[0023] Thirdly, the technical solution adopted by the present invention is: a computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any of the methods described above.

[0024] Fourthly, the technical solution adopted by the present invention is: a computing device comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described above.

[0025] The present invention has the following advantages due to the adoption of the above technical solutions:

[0026] 1. This invention can assess the severity of unilateral facial paralysis for the temporal branch, zygomatic branch, buccal branch, and mandibular branch of the face. By combining prior medical knowledge, this invention selects specific facial key points to divide different regions of interest, obtaining the local dynamic symmetry features corresponding to each facial nerve branch. These features are then fused with static global facial features under selected extreme movement states, thereby achieving facial paralysis detection and assessment for different facial nerve branches.

[0027] 2. This invention improves the assessment performance of facial paralysis severity. It combines a multi-task learning strategy, simultaneously performing a binary classification task (whether it is facial paralysis) and a regression task (the severity of facial paralysis). An enhanced regression module is designed, introducing learnable threshold and proportional parameters. These parameters are combined with the classification prediction results to enhance the regression results, thereby improving the regression accuracy of the facial paralysis level score. Attached Figure Description

[0028] Figure 1 This is a diagram of the overall network structure in an embodiment of the present invention;

[0029] Figure 2 These are different methods of dividing the facial nerve region bounding box used in the embodiments of the present invention;

[0030] Figure 3 This is a method for extracting local symmetry information in the embodiments of the present invention. Detailed Implementation

[0031] To address the current technological gap in automated detection methods for multi-level assessment of different facial nerve branches, this invention provides a method, system, medium, and device for assessing unilateral facial paralysis based on multi-feature fusion and multi-task learning. The method includes: acquiring video of a frontal face with specified facial expressions using a standard RGB camera, and preprocessing the video to retain video frames showing detectable facial regions; extracting dynamic local symmetry features and static global facial features under extreme movement states from the uniformly sampled video frame data; fusing the local dynamic symmetry features and global static facial features, and adding them to the global static facial features; simultaneously inputting the merged features into a classification subnetwork and a regression subnetwork to obtain a classification probability value for whether facial paralysis is present and a preliminary regression estimate of the severity of facial paralysis; combining the classification probability value and learnable parameters to perform enhanced regression on the preliminary estimate to obtain a final regression estimate; and post-processing the regression results to obtain the final assessment level of facial paralysis severity.

[0032] This invention uses convolutional neural networks and transformer models for multi-feature extraction and fusion. It combines expert prior knowledge to design a method for local feature extraction and fusion with global facial features for different facial nerve regions. Furthermore, it adopts a multi-task learning approach to simultaneously perform facial paralysis detection and facial paralysis level regression, and designs an enhanced regression module to improve regression accuracy.

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0034] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0035] In one embodiment of the present invention, a multi-branch assessment method for unilateral facial paralysis based on multi-feature fusion and multi-task learning is provided. The aim is that, after inputting a video of a person's face with specified facial movements, the algorithm can detect whether the person in the video suffers from unilateral facial paralysis and provide the unilateral facial paralysis level for each facial nerve branch. During the training phase, the algorithm's input is a frontal video of a facial paralysis patient with specified facial expressions and their corresponding facial paralysis severity level labels; during the testing phase, the algorithm's input is the video of the face to be detected, and the output is a binary determination of whether the person suffers from facial paralysis and an estimated facial paralysis severity value.

[0036] Specifically, in this embodiment, as follows: Figure 1 As shown, this multi-branch assessment method for unilateral facial paralysis includes the following steps:

[0037] 1) After preprocessing the collected frontal facial expression and action video, retain the video frames in which the facial area can be detected;

[0038] 2) Uniformly sample the preprocessed video frame sequence and extract the local dynamic symmetry features of the target facial nerve branch region and the global static face features under extreme action conditions, respectively;

[0039] 3) The local dynamic symmetry features and the global static face features are fused together and then added to the global static face features to obtain the merged shared features;

[0040] 4) Input the shared features into the classification subnetwork and the regression subnetwork at the same time to obtain the classification probability value of whether or not facial paralysis occurs and the preliminary regression estimate of the severity of facial paralysis. Combine the classification probability value and the learnable parameters to perform enhanced regression on the preliminary regression estimate to obtain the final regression estimate.

[0041] 5) Perform post-processing on the regression estimates to obtain the final assessment level of facial paralysis severity.

[0042] In step 1) above, after preprocessing the collected frontal facial expression and action video, the video frames in which the face region can be detected are retained. Specifically, OpenCV is used to extract video frames, and Dlib is used to perform face detection, alignment and cropping. The coordinate information of the 68 sets of detected facial feature points is retained, and the facial nerve branch paralysis degree label corresponding to the expression and action video is read.

[0043] In this embodiment, the UPFP-SG dataset (the Unilateral Peripheral Facial Paralysis Severity Grading Dataset) was used for training. This dataset contains video data of four types of facial movements from 59 patients with facial paralysis. The subjects included different age groups, genders, and severity of the disease. The specified facial expressions included raising eyebrows, closing eyes, puffing out cheeks, and showing teeth, respectively targeting the state detection of the temporal branch, zygomatic branch, buccal branch, and mandibular marginal branch of the face. During testing, frontal facial videos of the specified movements could be captured using any ordinary RGB camera.

[0044] In step 2) above, the local dynamic symmetry features of the target facial nerve branch region and the global static facial features under extreme action states are extracted, including the following steps:

[0045] 2.1) After uniform sampling, the keyframe sequence is extracted. Based on expert priors, the facial bounding box corresponding to each facial nerve branch is extracted from the keyframe sequence as the region corresponding to the facial nerve branch.

[0046] The specific extraction method is as follows: the feature point numbers are in the Dlib-68 point labeling format: temporal branch region (bounded box formed by feature points 0, 16 to the top of the image), zygomatic branch region (bounded box formed by feature points 1, 15, 21, 22), buccal branch region (bounded box formed by feature points 3, 13, 29, 57), and mandibular marginal branch region (bounded box formed by feature points 3, 8, 13).

[0047] 2.2) Divide the area corresponding to the facial nerve branch symmetrically into left and right parts, and subtract the left half from the mirrored right half to obtain the symmetry difference information of the target facial area;

[0048] 2.3) The adjusted local symmetry difference image is used to extract spatial features through a convolutional neural network, and then temporal features are extracted through the transformer's Encoder structure to obtain a local symmetry difference image sequence;

[0049] 2.4) Select intermediate frames from the keyframe sequence as global facial expression references under extreme action states, and readjust the image scale to obtain a global face image.

[0050] In this embodiment, the preprocessed video frame sequence is uniformly sampled to extract 16 keyframes. For example... Figure 2 The diagram shows the extraction of the corresponding facial bounding box based on the target facial branch. (Example:) Figure 3As shown, the symmetry difference between the left and right regions is calculated, and after resizing the image, a local symmetry difference image sequence of 16×3×224×224 is obtained as input. The middle frame is selected from the 16-frame keyframe sequence as the global facial expression reference under extreme action conditions, and the image scale is readjusted to obtain a global face image input of 3×224×224. The obtained local symmetry difference image sequence and global face image are respectively fed into the local dynamic feature extraction module and the face static feature extraction module.

[0051] In this embodiment, the local dynamic feature extraction module uses a pre-trained ResNet-18 convolutional neural network to extract the spatial features of each image frame, and adds positional codes sequentially to this 16×512 feature sequence. The positional codes are calculated as follows:

[0052]

[0053] Where, d model Let i represent the feature dimension (512 in this case), i = 0, 1, ..., d. model / 2, where p is the actual position of the corresponding feature in the sequence, PE p This represents the location vector of this feature. This represents the i-th element in the position vector, where k indicates whether it is odd or even. If i = 2k, then i is even; if i = 2k+1, then i is odd. The spatial feature sequence with added position encoding is then input into the encoder structure of the transformer model for temporal feature extraction, resulting in a 16×512 dynamic local feature flow F. d .

[0054] The global face image is then processed using a pre-trained Inception-ResNetV1 to extract static facial features, resulting in a 1×512 static face feature F. s .

[0055] In step 3) above, the local dynamic symmetry features and the global static face features are fused, specifically as follows:

[0056] In obtaining static facial features F s and dynamic feature flow F d Subsequently, to effectively combine dynamic and static, global and local features, a transformer decoder structure is used as a multi-feature fusion module; specifically, during multi-head attention computation, the obtained local dynamic symmetric features F are... d As a query, the obtained global static face features F s As the key and value, the fused feature F is obtained. f The calculation process for feature fusion is as follows:

[0057]

[0058] Where, d s Representing F s Dimensions.

[0059] In step 4) above, the shared features are simultaneously input into the classification subnetwork and the regression subnetwork to obtain the classification probability value of whether or not facial paralysis occurs and the preliminary regression estimate of the severity of facial paralysis. Specifically, a multi-task learning approach is adopted, simultaneously performing the binary classification task b for facial paralysis detection and the regression task r for facial paralysis intensity estimation. The merged features are then fed into both the classification subnetwork and the regression subnetwork as shared features to obtain the probability value Y of being diagnosed with facial paralysis. b Non-negative facial paralysis assessment score Y′ r .

[0060] In this embodiment, the classification subnetwork consists of a fully connected layer and a Softmax layer forming a binary classification head, which outputs a probability value Y for being diagnosed as facial paralysis. b ;

[0061] The regression subnetwork consists of a fully connected layer and a ReLU layer as the regression head, outputting a non-negative facial paralysis assessment score Y′. r ;

[0062] Introducing a learnable threshold parameter θ t and proportional parameter θ s Combined with binary classification prediction probability Y b For the initial regression value Y′ r Perform enhanced regression to obtain the final regression estimate Y. r .

[0063] Y r =tanh(θ) s ·(Y b -Sigmoid(θ t )))-Y′ r

[0064] Specifically, for binary classification task b, data with label 1 are considered negative examples, and data with labels 2-6 are considered positive examples. The predicted probability Y is calculated using the binary cross-entropy function. b With truth labels The loss between L b The batch size is N, and the calculation process is as follows:

[0065]

[0066] For regression tasks, the SmoothL1 function is used to calculate the regression estimate Y. r With strength rating label The loss between L r The calculation process is as follows:

[0067]

[0068] The combined loss of task group T = {b, r} is calculated using the multi-task loss function, and the automatic weighted loss function is calculated as follows:

[0069]

[0070] Where X represents the input data, Y t This is a single-task prediction value. For each task, a truth value label is provided. This represents the loss function for a single task, with learnable parameters w. t =(θ t c t ) by single-task network parameters θ t sum coefficient c t constitute.

[0071] In step 5) above, the regression estimates are post-processed to obtain the final facial paralysis severity assessment level, including the following steps:

[0072] 5.1) For a binary classification task, if the output probability value exceeds the learned threshold parameter, i.e., Y... b >θ t If the result is positive, it is considered a positive result for facial paralysis; otherwise, it is considered a negative result for facial paralysis.

[0073] 5.2) For regression tasks, the regression estimate Y obtained after regression will be enhanced. r The severity of facial paralysis is determined by the range in which the patient falls.

[0074] Specifically, the post-processing method for the final regression results is as follows: If Y r If the values ​​fall within the intervals (-∞,0), (0,1), (1,2), (2,3), (3,4), and (4,+∞), the facial paralysis level is determined to be I, II, III, IV, V, or VI.

[0075] In this embodiment, the computer executing the algorithm uses an AMD EPYC 7282 CPU, runs Ubuntu 18.04 with CUDA version 11.7, and employs PyTorch version 2.0 as the neural network architecture. During training, the initial learning rate is 2e-4, the batch size is 32, and the training runs for 40 epochs. Parameters can be adjusted appropriately based on different training set sizes. After training, the network weights are obtained. In the testing phase, a video of the target action is input, and the final output is obtained through the facial paralysis grading evaluation network.

[0076] In summary, this invention can achieve graded assessment of the degree of facial paralysis of facial nerve branches in facial videos of specified facial expressions and movements.

[0077] To verify the effectiveness and practicality of the method proposed in this invention, an example on the UPFP-SG dataset is given below. Table 1 shows the average detection results of the example using three-fold cross-validation. The evaluation metrics are Accuracy and F1 score. The names of each facial nerve branch and its corresponding action in the dataset are: Temporal Branch / Brows Lift, Zygomatic Branch / Eyes Close, Buccal Branch / Cheeks Blow, and Marginal Mandibular Branch / Teeth Show.

[0078] Table 1 shows the average metrics of the three-fold cross-validation on the UPFP-SG dataset for the examples.

[0079]

[0080] Table 2 shows the comparison of examples on the validation set with Parra et al. (GSParra-Dominguez, RESanchez-Yanez, and CHGarcia-Capulin, "Facial paralysis detection on images using keypoint analysis," Applied Sciences, vol.11, no.5, p.2435, 2021.), Ban et al. (S.Ban, HSNam, and E.Park, "Detecting paralysis of stroke symptom in video: Transferlearning with gated recurrent unit using public big data of facial images," in 2022IEEE International Conference on Big Data (Big Data), 2022, pp.6587–6589.), Sari et al. (CASari et al., "Facial palsy detection through changes in facialmuscle functionality using CNN algorithm," in 2023 6th International Conference of Computer and Informatics The results of the grading assessment of facial paralysis severity were compared (Engineering (IC2IE), 2023, pp.297–302.).

[0081] Table 2 shows the evaluation results of the method on the UPFP-SG dataset using the Acc and F1 metrics.

[0082] Method Acc F1 Parra et al. 54.5 31.75 Ben et al. 70. 36.35 Sari et al. 66.04 38.96 Ours 72.6 52.54

[0083] As shown in Table 2, compared with other facial paralysis level assessment methods based on single features, the improved algorithm proposed in this invention has better performance for multi-branch, multi-level assessment of unilateral facial paralysis. This embodiment demonstrates significant improvement for the vast majority of test categories, proving the effectiveness of the invention.

[0084] In one embodiment of the present invention, a multi-branch assessment system for unilateral facial paralysis based on multi-feature fusion and multi-task learning is provided, comprising:

[0085] The preprocessing module preprocesses the acquired frontal facial expression and action video and retains the video frames in which the facial region can be detected.

[0086] The feature extraction module performs uniform sampling on the preprocessed video frame sequence to extract local dynamic symmetry features of the target facial nerve branch region and global static facial features under extreme action conditions.

[0087] The feature fusion and merging module fuses local dynamic symmetrical features and global static face features, and adds them to the global static face features to obtain the merged shared features.

[0088] The sub-network processing module simultaneously inputs the shared features into the classification sub-network and the regression sub-network to obtain the classification probability value of whether or not facial paralysis occurs and the preliminary regression estimate of the severity of facial paralysis. Combining the classification probability value with the learnable parameters, the preliminary regression estimate is enhanced to obtain the final regression estimate.

[0089] The assessment module performs data post-processing on the regression estimates to obtain the final assessment level of the severity of facial paralysis.

[0090] In the above embodiments, after preprocessing the acquired frontal facial expression and action video, video frames in which the facial region can be detected are retained, including:

[0091] The video was extracted using OpenCV, and face detection, alignment and cropping were performed using Dlib. The coordinate information of 68 detected facial feature points was preserved, and the facial nerve branch paralysis degree label corresponding to the facial expression video was read.

[0092] In the above embodiments, the local dynamic symmetry features of the target facial nerve branch region and the global static facial features under extreme action states are extracted, including:

[0093] After uniform sampling, the keyframe sequence is extracted. Based on expert priors, the facial bounding box corresponding to each facial nerve branch is extracted from the keyframe sequence as the region corresponding to the facial nerve branch.

[0094] The area corresponding to the facial nerve branch is symmetrically divided into left and right parts. The left half is subtracted from the mirrored right half to obtain the symmetry difference information of the target facial area.

[0095] The adjusted local symmetry difference image is used to extract spatial features through a convolutional neural network, and then temporal features are extracted through the transformer's encoder structure to obtain a local symmetry difference image sequence.

[0096] The intermediate frame is selected from the keyframe sequence as the global facial expression reference under extreme action conditions, and the image scale is readjusted to obtain the global face image.

[0097] In the above embodiments, feature fusion is performed between local dynamic symmetry features and global static face features, including:

[0098] The transformer decoder structure is used as the multi-feature fusion module. The obtained local dynamic symmetrical features are used as the query, and the obtained global static face features are used as the key and value to obtain the fused features.

[0099] In the above embodiments, shared features are simultaneously input into the classification subnetwork and the regression subnetwork to obtain the classification probability value of whether or not facial paralysis occurs and the preliminary regression estimate of the severity of facial paralysis, including:

[0100] A multi-task learning approach is adopted, simultaneously performing a binary classification task (b) for facial paralysis detection and a regression task (r) for facial paralysis intensity estimation. The merged features are fed as shared features into both the classification and regression sub-networks to obtain the probability value Y for being diagnosed with facial paralysis. b Non-negative facial paralysis assessment score Y′ r .

[0101] In this embodiment, the classification subnetwork consists of a fully connected layer and a Softmax layer forming a binary classification head, which outputs a probability value Y for being diagnosed as facial paralysis. b ;

[0102] The regression subnetwork consists of a fully connected layer and a ReLU layer as the regression head, outputting a non-negative facial paralysis assessment score Y′. r ;

[0103] Introducing a learnable threshold parameter θ t and proportional parameter θ s Combined with binary classification prediction probability Y b For the initial regression value Y′ r Perform enhanced regression.

[0104] In the above embodiments, the regression estimates are post-processed to obtain the final facial paralysis severity assessment level, including:

[0105] For binary classification tasks, if the output probability value exceeds the learned threshold parameter, it is judged as a positive result for facial paralysis; otherwise, it is judged as a negative result for facial paralysis.

[0106] For regression tasks, the regression estimates obtained after augmentation will be used to classify the degree of facial paralysis according to the interval they fall into.

[0107] The system provided in this embodiment is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0108] In one embodiment of the present invention, a computing device is provided, which can be a terminal and may include: a processor, a communication interface, memory, a display screen, and an input device. The processor, communication interface, and memory communicate with each other via a communication bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs, which, when executed by the processor, implement the methods described in the above embodiments. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals. Wireless communication can be achieved through Wi-Fi, a management network, NFC (Near Field Communication), or other technologies. The display screen can be a liquid crystal display or an e-ink display. The input device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad mounted on the casing of the computing device, or an external keyboard, touchpad, or mouse. The processor can call logical instructions stored in the memory.

[0109] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0110] In one embodiment of the present invention, a computer program product is provided, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to perform the methods provided in the above-described method embodiments.

[0111] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided, which stores server instructions that cause a computer to perform the methods provided in the above embodiments.

[0112] The computer-readable storage medium provided in the above embodiments has a similar implementation principle and technical effect to the above method embodiments, and will not be described again here.

[0113] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0114] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-branch assessment method for unilateral facial paralysis based on multi-feature fusion and multi-task learning, characterized in that, include: After preprocessing the collected frontal facial expression and action video, retain the video frames in which the facial area can be detected. The preprocessed video frame sequence is uniformly sampled to extract local dynamic symmetry features of the target facial nerve branch region and global static facial features under extreme action states, including: After uniform sampling, the keyframe sequence is extracted. Based on expert priors, the facial bounding box corresponding to each facial nerve branch is extracted from the keyframe sequence as the region corresponding to the facial nerve branch. The area corresponding to the facial nerve branch is symmetrically divided into left and right parts. The left half is subtracted from the mirrored right half to obtain the symmetry difference information of the target facial area. The adjusted local symmetry difference image is used to extract spatial features through a convolutional neural network, and then temporal features are extracted through the transformer's encoder structure to obtain a local symmetry difference image sequence. The intermediate frames are selected from the keyframe sequence as global facial expression references under extreme action conditions, and the image scale is readjusted to obtain the global face image; The local dynamic symmetric features and global static face features are fused together and then added to the global static face features to obtain the merged shared features. The shared features are simultaneously input into the classification subnetwork and the regression subnetwork to obtain the classification probability value of whether facial paralysis occurs and the preliminary regression estimate of the severity of facial paralysis. The classification probability value and the learnable parameters are combined to perform enhanced regression on the preliminary regression estimate to obtain the final regression estimate. The regression estimates were post-processed to obtain the final assessment level of the severity of facial paralysis.

2. The method for multi-branch assessment of unilateral facial paralysis based on multi-feature fusion and multi-task learning as described in claim 1, characterized in that, After preprocessing the acquired frontal facial expression and action video, video frames in which the face region can be detected are retained, including: The video was extracted using OpenCV, and face detection, alignment and cropping were performed using Dlib. The coordinate information of 68 detected facial feature points was preserved, and the facial nerve branch paralysis degree label corresponding to the facial expression video was read.

3. The unilateral facial paralysis multi-branch assessment method based on multi-feature fusion and multi-task learning as described in claim 1, characterized in that, Feature fusion is performed between local dynamic symmetry features and global static facial features, including: The transformer decoder structure is used as the multi-feature fusion module. The obtained local dynamic symmetrical features are used as the query, and the obtained global static face features are used as the key and value to obtain the fused features.

4. The unilateral facial paralysis multi-branch assessment method based on multi-feature fusion and multi-task learning as described in claim 1, characterized in that, By simultaneously inputting shared features into both the classification and regression subnetworks, preliminary regression estimates of the classification probability of facial paralysis and the severity of facial paralysis are obtained, including: A multi-task learning approach is adopted, simultaneously performing a binary classification task (b) for facial paralysis detection and a regression task (r) for facial paralysis intensity estimation. The merged features are fed as shared features into both the classification and regression sub-networks to obtain the probability value of being diagnosed with facial paralysis. Non-negative facial paralysis assessment scores .

5. The unilateral facial paralysis multi-branch assessment method based on multi-feature fusion and multi-task learning as described in claim 4, characterized in that, The classification subnetwork consists of a fully connected layer and a softmax layer forming a binary classification head, which outputs the probability value of being diagnosed with facial paralysis. ; The regression subnetwork consists of a fully connected layer and a ReLU layer as the regression head, outputting a non-negative facial paralysis assessment score. ; Introducing a learnable threshold parameter and proportional parameters Combined with binary classification prediction probability For the initial regression values Perform enhanced regression.

6. The multi-branch assessment method for unilateral facial paralysis based on multi-feature fusion and multi-task learning as described in claim 1, characterized in that, Post-processing of the regression estimates yields the final facial paralysis severity assessment level, including: For binary classification tasks, if the output probability value exceeds the learned threshold parameter, it is judged as a positive result for facial paralysis; otherwise, it is judged as a negative result for facial paralysis. For regression tasks, the regression estimates obtained after augmentation will be used to classify the degree of facial paralysis according to the interval they fall into.

7. A multi-branch assessment system for unilateral facial paralysis based on multi-feature fusion and multi-task learning, used to implement the multi-branch assessment method for unilateral facial paralysis based on multi-feature fusion and multi-task learning as described in any one of claims 1 to 6, characterized in that, include: The preprocessing module preprocesses the acquired frontal facial expression and action video and retains the video frames in which the facial region can be detected. The feature extraction module performs uniform sampling on the preprocessed video frame sequence to extract local dynamic symmetry features of the target facial nerve branch region and global static facial features under extreme action conditions. The feature fusion and merging module fuses local dynamic symmetrical features and global static face features, and adds them to the global static face features to obtain the merged shared features. The sub-network processing module simultaneously inputs the shared features into the classification sub-network and the regression sub-network to obtain the classification probability value of whether or not facial paralysis occurs and the preliminary regression estimate of the severity of facial paralysis. Combining the classification probability value with the learnable parameters, the preliminary regression estimate is enhanced to obtain the final regression estimate. The assessment module performs data post-processing on the regression estimates to obtain the final assessment level of the severity of facial paralysis.

8. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods described in claims 1 to 6.

9. A computing device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described in claims 1 to 6.

Images