Single-sided paralysis multi-branch evaluation method and system based on multi-feature fusion and multi-task learning, medium and equipment

Abstract

The invention relates to the field of computer vision, and discloses a single-sided paralysis multi-branch evaluation method and system based on multi-feature fusion and multi-task learning, a medium and equipment, and the method comprises the steps: carrying out the preprocessing of a collected front face specified expression action video, and reserving a video frame capable of detecting a face region; extracting a local dynamic symmetric feature of the target facial nerve branch region and a global static face feature in a limit action state, performing feature fusion, and adding and merging the fused feature and the global static face feature to obtain a merged shared feature; inputting the shared features into the classification sub-network and the regression sub-network at the same time to obtain a classification probability value of whether the facial paralysis exists or not and a preliminary regression estimation value of the severity of the facial paralysis, and performing enhanced regression on the preliminary regression estimation value in combination with the classification probability value and learnable parameters to obtain a final regression estimation value; and performing data post-processing on the regression estimation value to obtain a final facial paralysis severity assessment level.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method, system, medium and device for evaluating unilateral facial paralysis based on multi-feature fusion and multi-task learning. Background Art

[0002] Unilateral peripheral facial palsy (UPFP) is a facial dysfunction caused by damage to the facial nerve nucleus and downstream nerves on one side. It is usually manifested as weakness in closing eyes, frowning, puffing cheeks, showing teeth, and closing lips on the affected side, and the corner of the mouth is tilted to the opposite side, which seriously affects the patient's function, appearance, and mental health. At present, the clinical diagnosis of facial palsy mainly relies on the subjective judgment of doctors. Patients are instructed to perform specific facial movements such as raising eyebrows, closing eyes, puffing cheeks, and showing teeth, and then doctors grade the severity of facial palsy based on the asymmetry observed from these movements.

[0003] Compared with clinical methods, computer-assisted methods often provide a promising alternative by providing more objective and effective evaluations. With the advancement of deep learning, related technical means have also been widely used in the field of medical image analysis, such as medical image segmentation, computer-aided diagnosis and detection. In the field of facial paralysis detection, many researchers have used deep learning methods to automatically detect and assess facial paralysis grades. Compared with traditional methods, deep learning-based detection methods have shown higher accuracy and efficiency, providing valuable support for clinical practitioners.

[0004] Existing facial paralysis detection and grading methods based on deep learning can generally be divided into three categories: based on feature point coordinate differences, based on static image symmetry differences, or based on dynamic expression changes. Existing facial paralysis detection algorithms based on dynamic videos have made significant progress in facial paralysis grading detection. However, these methods only use a single feature, do not consider the fusion of multiple features, and only conduct a comprehensive rating assessment of the overall performance of unilateral facial paralysis. In the field of oral and maxillofacial head and neck surgery, unilateral peripheral facial paralysis usually involves damage to one or more unilateral peripheral branches of the facial nerve. The assessment of the overall function of the unilateral facial nerve cannot accurately reflect the damage and recovery of a single branch. At present, there is no automatic detection method that can perform multi-level assessments on different facial nerve branches.

[0005] Based on the above analysis, the study of a multi-level assessment method for unilateral facial paralysis that can effectively integrate and utilize multiple features and target different facial nerve branches is of great medical clinical value and social benefit. Therefore, a new unilateral facial paralysis degree assessment algorithm is urgently needed to meet the current urgent clinical needs and further improve the grading accuracy. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide a unilateral facial paralysis multi-branch evaluation method, system, medium and equipment based on multi-feature fusion and multi-task learning, which can handle multi-feature extraction and fusion in face videos, and can perform multi-level evaluation of facial paralysis degree for different facial nerve branches.

[0007] To achieve the above-mentioned purpose, in a first aspect, the technical solution adopted by the present invention is: a multi-branch evaluation method for unilateral facial paralysis based on multi-feature fusion and multi-task learning, which comprises: preprocessing the collected frontal face specified expression action video, and retaining the video frames in which the face area can be detected; uniformly sampling the preprocessed video frame sequence, and extracting the local dynamic symmetry features of the target facial nerve branch area and the global static face features under the extreme action state respectively; fusing the local dynamic symmetry features and the global static face features, and adding and merging them with the global static face features to obtain the merged shared features; inputting the shared features into the classification subnetwork and the regression subnetwork at the same time, obtaining the classification probability value of whether there is facial paralysis and the preliminary regression estimate of the severity of facial paralysis, combining the classification probability value with the learnable parameters, and performing enhanced regression on the preliminary regression estimate to obtain the final regression estimate; performing data post-processing on the regression estimate to obtain the final facial paralysis severity assessment level.

[0008] Furthermore, after preprocessing the collected frontal face specified expression and action video, the video frames in which the face area can be detected are retained, including: using OpenCV tools to extract video frames, and using Dlib for face detection, alignment and cropping, retaining the coordinate information of 68 groups of detected facial feature points, and reading the facial nerve branch paralysis degree label corresponding to the expression and action video.

[0009] Furthermore, the local dynamic symmetry features of the target facial nerve branch area and the global static face features under extreme motion states are extracted respectively, including:

[0010] After uniform sampling, a key frame sequence is extracted, and the facial bounding box corresponding to each facial nerve branch is extracted from the key frame sequence according to expert prior knowledge as the area corresponding to the facial nerve branch;

[0011] The area corresponding to the facial nerve branches is symmetrically divided into left and right parts, and the left half is subtracted from the mirror-flipped right half to obtain the symmetrical difference information of the target facial area;

[0012] The adjusted local symmetric difference image is used to extract spatial features through a convolutional neural network, and then the temporal features are extracted through the encoder structure of the transformer to obtain a local symmetric difference image sequence;

[0013] The middle frame is selected from the key frame sequence as a reference for the global facial expression under the extreme action state, and the image scale is re-adjusted to obtain the global facial image.

[0014] Furthermore, the local dynamic symmetry features and the global static face features are fused, including: using the transformer decoder structure as a multi-feature fusion module, using the obtained local dynamic symmetry features as a query, and using the obtained global static face features as a key and a value to obtain fused features.

[0015] Furthermore, the shared features are simultaneously input into the classification subnetwork and the regression subnetwork to obtain the classification probability value of whether facial paralysis occurs and the preliminary regression estimate of the severity of facial paralysis, including: using a multi-task learning method to simultaneously perform the binary classification task b of facial paralysis detection and the regression task r of facial paralysis intensity estimation, and the merged features are simultaneously input into the classification subnetwork and the regression subnetwork as shared features to obtain the probability value Y of being diagnosed with facial paralysis b and a non-negative facial paralysis assessment score Y′ r .

[0016] Furthermore, the classification subnetwork consists of a fully connected layer and a Softmax layer to form a binary classification head, and outputs the probability value Y of being diagnosed with facial paralysis b ;

[0017] The regression subnetwork consists of a fully connected layer and a ReLU layer to form a regression head, outputting a non-negative facial paralysis evaluation score Y′ r ;

[0018] Introducing a learnable threshold parameter θ t and the scale parameter θ s , combined with the binary prediction probability Y b For the initial regression value Y′ r Perform enhanced regression.

[0019] Furthermore, the regression estimate is post-processed to obtain the final facial paralysis severity assessment level, including:

[0020] For the binary classification task, if the output probability value exceeds the learned threshold parameter, it is judged as positive for facial paralysis, otherwise it is judged as negative for facial paralysis;

[0021] For the regression task, the regression estimate obtained after enhanced regression is divided into facial paralysis levels according to the interval it falls into.

[0022] In the second aspect, the technical solution adopted by the present invention is: a unilateral facial paralysis multi-branch evaluation system based on multi-feature fusion and multi-task learning, which includes: a preprocessing module, which preprocesses the collected frontal face specified expression and action video, and retains the video frames in which the face area can be detected; a feature extraction module, which uniformly samples the preprocessed video frame sequence, and respectively extracts the local dynamic symmetry features of the target facial nerve branch area and the global static face features under the extreme action state; a feature fusion and merging module, which fuses the local dynamic symmetry features and the global static face features, and adds and merges them with the global static face features to obtain the merged shared features; a subnetwork processing module, which simultaneously inputs the shared features into the classification subnetwork and the regression subnetwork to obtain the classification probability value of whether there is facial paralysis and the preliminary regression estimate of the severity of facial paralysis, and combines the classification probability value with the learnable parameters to perform enhanced regression on the preliminary regression estimate to obtain the final regression estimate; an evaluation module, which performs data post-processing on the regression estimate to obtain the final facial paralysis severity assessment level.

[0023] In a third aspect, the technical solution adopted by the present invention is: a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by a computing device, the computing device executes any one of the above methods.

[0024] In a fourth aspect, the technical solution adopted by the present invention is: a computing device, comprising: one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the above methods.

[0025] The present invention adopts the above technical solution, which has the following advantages:

[0026] 1. The present invention can evaluate the severity of unilateral facial paralysis for the temporal branch, zygomatic branch, buccal branch and mandibular marginal branch of the face. The present invention combines medical prior knowledge, selects specific facial key points to divide different regions of interest, obtains local dynamic symmetry features corresponding to each facial nerve branch, and performs multi-feature fusion with the static global facial features under the selected extreme action state, thereby realizing facial paralysis detection and evaluation for different facial nerve branches.

[0027] 2. The present invention improves the performance of facial paralysis severity assessment. Combining a multi-task learning strategy, the binary classification task of whether it is facial paralysis and the regression task of facial paralysis severity are performed simultaneously, and an enhanced regression module is designed, which introduces learnable threshold parameters and ratio parameters, and enhances the regression results in combination with the classification prediction results, thereby improving the regression accuracy of the facial paralysis grade score value. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a diagram of the overall network structure in an embodiment of the present invention;

[0029] Figure 2 Different facial nerve region bounding box division methods used in the embodiments of the present invention;

[0030] Figure 3 This is the local symmetry information extraction method in the embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to solve the technical gap of the current automatic detection method for multi-level evaluation of different facial nerve branches, the present invention provides a unilateral facial paralysis multi-branch evaluation method, system, medium and equipment based on multi-feature fusion and multi-task learning, which comprises: using an ordinary RGB camera to collect a frontal face specified expression action video, and pre-processing to retain the video frame in which the face area can be detected; from the uniformly sampled video frame data, respectively extracting the dynamic local symmetry features of the target facial nerve branch area and the static global face features under the extreme action state; fusing the local dynamic symmetry features and the global static face features, and adding and merging them with the global static face features; inputting the merged features into the classification subnetwork and the regression subnetwork at the same time to obtain the classification probability value of whether there is facial paralysis and the preliminary regression estimate value of the severity of facial paralysis; combining the classification probability value with the learnable parameter, performing enhanced regression on the preliminary estimate value to obtain the final regression estimate value; performing data post-processing on the regression result to obtain the final facial paralysis severity evaluation level.

[0032] The present invention performs multi-feature extraction and fusion based on convolutional neural networks and transformer models, and designs a method for extracting local features of different facial nerve areas and fusing them with global facial features in combination with expert priors. It also adopts a multi-task learning method to simultaneously perform facial paralysis detection and facial paralysis grade regression, and designs an enhanced regression module to improve regression accuracy.

[0033] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0034] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0035] In one embodiment of the present invention, a multi-branch evaluation method for unilateral facial paralysis based on multi-feature fusion and multi-task learning is provided, the purpose of which is to input a video of a specified action of a face to be detected, so that the algorithm can detect whether the person in the video suffers from unilateral facial paralysis, and give the unilateral facial paralysis level of each facial nerve branch. In the training stage, the input of the algorithm is a frontal video of a facial paralysis patient with a specified expression and action and its corresponding facial paralysis level label; in the testing stage, the input of the algorithm is a video of a facial action to be detected, and the output is a binary judgment of whether the person suffers from facial paralysis and an estimated facial paralysis level value.

[0036] Specifically, in this embodiment, Figure 1 As shown, the unilateral facial paralysis multi-branch assessment method includes the following steps:

[0037] 1) Preprocessing the collected frontal face specified expression action video, retaining the video frame in which the face area can be detected;

[0038] 2) Uniformly sample the preprocessed video frame sequence to extract the local dynamic symmetry features of the target facial nerve branch area and the global static face features under extreme motion states;

[0039] 3) Fusing local dynamic symmetry features and global static face features, and adding and merging them with global static face features to obtain merged shared features;

[0040] 4) The shared features are simultaneously input into the classification subnetwork and the regression subnetwork to obtain the classification probability value of whether facial paralysis occurs and the preliminary regression estimate of the severity of facial paralysis. The classification probability value is combined with the learnable parameters to perform enhanced regression on the preliminary regression estimate to obtain the final regression estimate.

[0041] 5) Perform data post-processing on the regression estimate to obtain the final facial paralysis severity assessment level.

[0042] In the above step 1), the collected frontal face specified expression and action video is preprocessed, and the video frames in which the face area can be detected are retained. Specifically, the OpenCV tool is used to extract the video frames, and Dlib is used for face detection, alignment and cropping. The coordinate information of the 68 groups of facial feature points detected is retained, and the facial nerve branch paralysis degree label corresponding to the expression and action video is read.

[0043] In this embodiment, the UPFP-SG dataset (the Unilateral Peripheral FacialParalysis Severity Grading Dataset) is used for training. This dataset contains four types of facial action video data of 59 patients with facial paralysis. The collected objects cover different age groups, different genders, and different degrees of illness. The specified facial expressions include raising eyebrows, closing eyes, puffing cheeks, and showing teeth, respectively for the state detection of the temporal branch, zygomatic branch, buccal branch, and mandibular marginal branch of the face. During the test, the frontal face video of the specified action can be shot by any ordinary RGB camera.

[0044] In the above step 2), respectively extracting the local dynamic symmetry features of the target facial nerve branch area and the global static face features under the extreme action state includes the following steps:

[0045] 2.1) extracting a key frame sequence after uniform sampling, and extracting the facial bounding box corresponding to each facial nerve branch from the key frame sequence according to expert prior knowledge as the area corresponding to the facial nerve branch;

[0046] The specific extraction method is that the feature point serial number follows the Dlib-68 point calibration format: temporal branch area (the bounding box formed by feature points 0, 16 to the top of the image), zygomatic branch area (the bounding box formed by feature points 1, 15, 21, and 22), buccal branch area (the bounding box formed by feature points 3, 13, 29, and 57), and mandibular marginal branch area (the bounding box formed by feature points 3, 8, and 13).

[0047] 2.2) The area corresponding to the facial nerve branches is symmetrically divided into left and right parts, and the left half is subtracted from the mirror-flipped right half to obtain the symmetric difference information of the target facial area;

[0048] 2.3) Adjust the obtained local symmetric difference image, extract spatial features through a convolutional neural network, and then extract temporal features through the encoder structure of the transformer to obtain a local symmetric difference image sequence;

[0049] 2.4) Select the middle frame from the key frame sequence as the global facial expression reference under the extreme action state, and rescale the image to obtain the global facial image.

[0050] In this embodiment, the pre-processed video frame sequence is uniformly sampled to extract 16 key frames. Figure 2 As shown in FIG. 1 , the corresponding facial bounding box is extracted according to the target facial branch. Figure 3As shown in the figure, the symmetric difference between the left and right regions is calculated, and the image scale is rescaled to obtain a 16×3×224×224 local symmetric difference image sequence input. The middle frame is selected from the 16-frame key frame sequence as the global facial expression reference under the extreme action state, and the image scale is rescaled to obtain a 3×224×224 global face image input. The obtained local symmetric difference image sequence and global face image are respectively sent to the local dynamic feature extraction module and the face static feature extraction module.

[0051] In this embodiment, the local dynamic feature extraction module uses a pre-trained ResNet-18 convolutional neural network to extract the spatial features of each image frame, and sequentially adds position codes to the feature sequence of size 16×512. The position code is calculated as follows:

[0052]

[0053] Among them, d model represents the feature dimension (here 512), i = 0, 1, ..., d model / 2, p is the actual position of the corresponding feature in the sequence, PE p represents the position vector of this feature, Then it represents the i-th element in this position vector, k represents odd and even numbers, if i=2k, it means i is an even number, and i=2k+1, it means i is an odd number. Then the spatial feature sequence with position encoding is input into the encoder structure of the transformer model for temporal feature extraction, and a 16×512 dynamic local feature flow F is obtained. d .

[0054] The global face image is extracted through the pre-trained Inception-ResNetV1 to obtain a 1×512 static face feature F s .

[0055] In the above step 3), the local dynamic symmetry features and the global static face features are fused, specifically:

[0056] When acquiring static facial features F s and dynamic feature flow F d Finally, in order to effectively combine dynamic and static, global and local features, the decoder structure of transformer is used as the multi-feature fusion module; in which, when performing multi-head attention calculation, the local dynamic symmetric feature F d As Query, the obtained global static face feature F s As Key and Value, we get the fused feature F f , the calculation process of feature fusion is as follows:

[0057]

[0058] Among them, d s Represents F s Dimension.

[0059] In the above step 4), the shared features are simultaneously input into the classification subnetwork and the regression subnetwork to obtain the classification probability value of whether facial paralysis occurs and the preliminary regression estimate of the severity of facial paralysis. Specifically, a multi-task learning method is used to simultaneously perform the binary classification task b of facial paralysis detection and the regression task r of facial paralysis intensity estimation. The merged features are simultaneously input into the classification subnetwork and the regression subnetwork as shared features to obtain the probability value Y of being diagnosed with facial paralysis. b and a non-negative facial paralysis assessment score Y′ r .

[0060] In this embodiment, the classification subnetwork consists of a fully connected layer and a Softmax layer to form a binary classification head, and outputs the probability value Y of being diagnosed with facial paralysis. b ;

[0061] The regression subnetwork consists of a fully connected layer and a ReLU layer to form a regression head, outputting a non-negative facial paralysis evaluation score Y′ r ;

[0062] Introducing a learnable threshold parameter θ t and the scale parameter θ s , combined with the binary prediction probability Y b For the initial regression value Y′ r Perform enhanced regression to obtain the final regression estimate Y r .

[0063] Y r =tanh(θ s ·(Y b -Sigmoid(θ t )))-Y′ r

[0064] Specifically, for the binary classification task b, the data with label 1 is regarded as negative examples, and the data with labels 2-6 are regarded as positive examples. The binary cross entropy function is used to calculate the predicted probability Y b and the true value label The loss between b , the batch size is N, and the calculation process is as follows:

[0065]

[0066] For regression tasks, use the SmoothL1 function to calculate the regression estimate Y r With intensity grade label The loss between r , the calculation process is as follows:

[0067]

[0068] The multi-task loss function is used to calculate the combined loss of the task group T = {b, r}. The automatic weight loss function is calculated as follows:

[0069]

[0070] Among them, X represents the input data, Y t is the single-task prediction value, is the true value label corresponding to a single task, Represents the loss function of a single task, and can learn parameters w t =(θ t , c t ) is composed of single-task network parameters θ t and coefficient c t constitute.

[0071] In the above step 5), the regression estimate is post-processed to obtain the final facial paralysis severity assessment level, including the following steps:

[0072] 5.1) For a binary classification task, if the output probability value exceeds the learned threshold parameter, that is, Y b >θ t , it is judged as positive for facial paralysis, otherwise it is judged as negative for facial paralysis;

[0073] 5.2) For regression tasks, the regression estimate Y obtained after the enhanced regression r The level of facial paralysis is divided according to the range it falls into.

[0074] Specifically, the post-processing method for the final regression result is: if Y r They fall into the intervals (-∞,0), (0,1), (1,2), (2,3), (3,4), (4,+∞) respectively, and the level of facial paralysis is determined as Ⅰ, Ⅱ, Ⅲ, Ⅳ, Ⅴ, Ⅵ.

[0075] In this embodiment, the CPU of the computer executing the algorithm is AMD EPYC 7282, the computer operating system is Ubuntu 18.04, the CUDA version is 11.7, and the neural network architecture used is Pytorch, version 2.0. During the training process, the initial learning rate is 2e-4, the batch size is 32, and a total of 40 rounds of training are performed. According to the size of the training set, the parameters can be adjusted appropriately, and the weight of the network can be obtained after the training is completed. In the test phase, the video to be tested of the target action is input, and the final output is obtained through the facial paralysis grading evaluation network.

[0076] In summary, the present invention can implement graded evaluation of the degree of facial paralysis of facial nerve branches for a face video with a specified expression and action.

[0077] To verify the effectiveness and practicality of the method proposed in the present invention, an example on the UPFP-SG dataset is given below. Table 1 shows the average detection results of the three-fold cross validation of the example, with the evaluation indicators Acc (Accuracy) and F1Score. The names of each facial nerve branch and its corresponding action in the dataset are: Temporal Branch / Brows Lift, Zygomatic Branch / Eyes Close, Buccal Branch / Cheeks Blow, Marginal Mandibular Branch / Teeth Show.

[0078] Table 1. Average index results of three-fold cross validation on the UPFP-SG dataset

[0079]

[0080] Table 2 shows the comparison of examples on the validation set with Parra et al. (GSParra-Dominguez, RESanchez-Yanez, and CHGarcia-Capulin, "Facial paralysis detection on images using keypoint analysis," Applied Sciences, vol.11, no.5, p.2435, 2021.), Ban et al. (S.Ban, HSNam, and E.Park, "Detecting paralysis of stroke symptom in video: Transferlearning with gated recurrent unit using public big data of facial images," in 2022IEEE International Conference on Big Data (Big Data), 2022, pp.6587–6589.), Sari et al. (CASari et al., "Facial palsy detection through changes in facialmuscle functionality using CNN algorithm," in 2023 6th International Conference of Computer and Informatics Engineering (IC2IE), 2023, pp. 297–302.) Comparison of facial paralysis degree grading assessment results.

[0081] Table 2 Evaluation results of the methods on the UPFP-SG dataset using the Acc and F1 indicators

[0082] Method Acc F1 Parra et al. 54.5 31.75 Ben et al. 70. 36.35 Sari et al. 66.04 38.96 Ours 72.6 52.54

[0083] As can be seen from Table 2, compared with other facial paralysis grade assessment methods based on a single feature, the improved algorithm proposed in the present invention has better performance for multi-branch multi-level assessment of unilateral facial paralysis. The use of this embodiment has a significant improvement for most test categories, proving the effectiveness of the invention.

[0084] In one embodiment of the present invention, a multi-branch assessment system for unilateral facial paralysis based on multi-feature fusion and multi-task learning is provided, which includes:

[0085] The preprocessing module preprocesses the collected frontal face specified expression and action video and retains the video frames where the face area can be detected;

[0086] The feature extraction module uniformly samples the preprocessed video frame sequence to extract the local dynamic symmetry features of the target facial nerve branch area and the global static face features under extreme action states;

[0087] The feature fusion and merging module fuses the local dynamic symmetry features and the global static face features, and adds and merges them with the global static face features to obtain the merged shared features;

[0088] The sub-network processing module simultaneously inputs the shared features into the classification sub-network and the regression sub-network to obtain the classification probability value of whether there is facial paralysis and the preliminary regression estimate of the severity of facial paralysis, and combines the classification probability value with the learnable parameters to perform enhanced regression on the preliminary regression estimate to obtain the final regression estimate;

[0089] The evaluation module performs data post-processing on the regression estimation value to obtain the final facial paralysis severity assessment level.

[0090] In the above embodiment, after the collected frontal face specified expression action video is preprocessed, the video frame in which the face area can be detected is retained, including:

[0091] OpenCV tools were used to extract video frames, and Dlib was used for face detection, alignment and cropping. The coordinate information of 68 groups of facial feature points detected were retained, and the facial nerve branch paralysis degree labels corresponding to the expression and action video were read.

[0092] In the above embodiment, the local dynamic symmetry features of the target facial nerve branch area and the global static face features under the extreme action state are extracted respectively, including:

[0093] After uniform sampling, a key frame sequence is extracted, and the facial bounding box corresponding to each facial nerve branch is extracted from the key frame sequence according to expert prior knowledge as the area corresponding to the facial nerve branch;

[0094] The area corresponding to the facial nerve branches is symmetrically divided into left and right parts, and the left half is subtracted from the mirror-flipped right half to obtain the symmetrical difference information of the target facial area;

[0095] The adjusted local symmetric difference image is used to extract spatial features through a convolutional neural network, and then the temporal features are extracted through the encoder structure of the transformer to obtain a local symmetric difference image sequence;

[0096] The middle frame is selected from the key frame sequence as a reference for the global facial expression under the extreme action state, and the image scale is re-adjusted to obtain the global facial image.

[0097] In the above embodiment, the local dynamic symmetry feature and the global static face feature are fused, including:

[0098] The transformer decoder structure is used as the multi-feature fusion module, the local dynamic symmetry features are used as the query, and the global static face features are used as the key and value to obtain the fused features.

[0099] In the above embodiment, the shared features are simultaneously input into the classification subnetwork and the regression subnetwork to obtain the classification probability value of whether facial paralysis occurs and the preliminary regression estimation value of the severity of facial paralysis, including:

[0100] Using multi-task learning, the binary classification task b of facial paralysis detection and the regression task r of facial paralysis intensity estimation are performed simultaneously. The merged features are sent to the classification subnetwork and the regression subnetwork as shared features to obtain the probability value Y of being diagnosed with facial paralysis. b and a non-negative facial paralysis assessment score Y′ r .

[0101] In this embodiment, the classification subnetwork consists of a fully connected layer and a Softmax layer to form a binary classification head, and outputs the probability value Y of being diagnosed with facial paralysis. b ;

[0102] The regression subnetwork consists of a fully connected layer and a ReLU layer to form a regression head, outputting a non-negative facial paralysis evaluation score Y′ r ;

[0103] Introducing a learnable threshold parameter θ t and the scale parameter θ s , combined with the binary prediction probability Y b For the initial regression value Y′ r Perform enhanced regression.

[0104] In the above embodiment, the regression estimate is post-processed to obtain the final facial paralysis severity assessment level, including:

[0105] For the binary classification task, if the output probability value exceeds the learned threshold parameter, it is judged as positive for facial paralysis, otherwise it is judged as negative for facial paralysis;

[0106] For the regression task, the regression estimate obtained after enhanced regression is divided into facial paralysis levels according to the interval it falls into.

[0107] The system provided in this embodiment is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for specific processes and detailed contents, which will not be repeated here.

[0108] A computing device provided in one embodiment of the present invention may be a terminal, which may include: a processor, a communications interface, a memory, a display screen, and an input device. The processor, the communications interface, and the memory communicate with each other through a communication bus. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory, wherein the non-volatile storage medium stores an operating system and a computer program, and the computer program is executed by the processor to implement the methods in the above embodiments; the internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a management network, NFC (near field communication) or other technologies. The display screen may be a liquid crystal display screen or an electronic ink display screen, and the input device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computing device, or an external keyboard, touchpad or mouse, etc. The processor may call the logic instructions in the memory.

[0109] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0110] In one embodiment of the present invention, a computer program product is provided, wherein the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments.

[0111] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores server instructions, and the computer instructions enable a computer to execute the methods provided in the above embodiments.

[0112] The above embodiment provides a computer-readable storage medium, whose implementation principle and technical effect are similar to those of the above method embodiment, and will not be repeated here.

[0113] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0114] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0115] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-branch evaluation method for unilateral facial paralysis based on multi-feature fusion and multi-task learning, characterized in that: include: After pre-processing the collected frontal face specified expression and action video, retain the video frame where the face area can be detected; The preprocessed video frame sequence is uniformly sampled to extract the local dynamic symmetry features of the target facial nerve branch area and the global static face features under extreme motion states. The local dynamic symmetry features and the global static face features are fused, and then added and merged with the global static face features to obtain the merged shared features; The shared features are simultaneously input into the classification subnetwork and the regression subnetwork to obtain the classification probability value of whether facial paralysis occurs and the preliminary regression estimate of the severity of facial paralysis. The classification probability value is combined with the learnable parameters to perform enhanced regression on the preliminary regression estimate to obtain the final regression estimate. The regression estimates were post-processed to obtain the final facial paralysis severity assessment level.

2. The multi-branch evaluation method for unilateral facial paralysis based on multi-feature fusion and multi-task learning as claimed in claim 1, characterized in that: After preprocessing the collected frontal face specified expression and action video, retain the video frames where the face area can be detected, including: OpenCV tools were used to extract video frames, and Dlib was used for face detection, alignment and cropping. The coordinate information of 68 groups of facial feature points detected were retained, and the facial nerve branch paralysis degree labels corresponding to the expression and action video were read.

3. The multi-branch evaluation method for unilateral facial paralysis based on multi-feature fusion and multi-task learning as claimed in claim 1, characterized in that: The local dynamic symmetry features of the target facial nerve branch area and the global static face features under extreme motion states are extracted respectively, including: After uniform sampling, a key frame sequence is extracted, and the facial bounding box corresponding to each facial nerve branch is extracted from the key frame sequence according to expert prior knowledge as the area corresponding to the facial nerve branch; The area corresponding to the facial nerve branches is symmetrically divided into left and right parts, and the left half is subtracted from the mirror-flipped right half to obtain the symmetrical difference information of the target facial area; The adjusted local symmetric difference image is used to extract spatial features through a convolutional neural network, and then the temporal features are extracted through the encoder structure of the transformer to obtain a local symmetric difference image sequence; The middle frame is selected from the key frame sequence as a reference for the global facial expression under the extreme action state, and the image scale is re-adjusted to obtain the global facial image.

4. The multi-branch evaluation method for unilateral facial paralysis based on multi-feature fusion and multi-task learning as claimed in claim 1, characterized in that: The local dynamic symmetry features and the global static face features are fused, including: The transformer decoder structure is used as the multi-feature fusion module, the local dynamic symmetry features are used as the query, and the global static face features are used as the key and value to obtain the fused features.

5. The multi-branch evaluation method for unilateral facial paralysis based on multi-feature fusion and multi-task learning as claimed in claim 1, characterized in that: The shared features are simultaneously input into the classification subnetwork and the regression subnetwork to obtain the classification probability value of facial paralysis and the preliminary regression estimate of the severity of facial paralysis, including: Using multi-task learning, the binary classification task b of facial paralysis detection and the regression task r of facial paralysis intensity estimation are performed simultaneously. The merged features are sent to the classification subnetwork and the regression subnetwork as shared features to obtain the probability value Y of being diagnosed with facial paralysis. b and a non-negative facial paralysis assessment score Y ′ r .

6. The multi-branch evaluation method for unilateral facial paralysis based on multi-feature fusion and multi-task learning as claimed in claim 5, characterized in that: The classification subnetwork consists of a fully connected layer and a Softmax layer to form a binary classification head, and outputs the probability value Y of being diagnosed with facial paralysis. b ; The regression subnetwork consists of a fully connected layer and a ReLU layer to form a regression head, outputting a non-negative facial paralysis evaluation score Y ′ r ; Introducing a learnable threshold parameter θ t and the scale parameter θ s , combined with the binary prediction probability Y b For the initial regression value Y ′ r Perform enhanced regression.

7. The multi-branch evaluation method for unilateral facial paralysis based on multi-feature fusion and multi-task learning as claimed in claim 1, characterized in that: The regression estimates were post-processed to obtain the final facial paralysis severity assessment level, including: For the binary classification task, if the output probability value exceeds the learned threshold parameter, it is judged as positive for facial paralysis, otherwise it is judged as negative for facial paralysis; For the regression task, the regression estimate obtained after enhanced regression is divided into facial paralysis levels according to the interval it falls into.

8. A multi-branch evaluation system for unilateral facial paralysis based on multi-feature fusion and multi-task learning, characterized in that: include: The preprocessing module preprocesses the collected frontal face specified expression and action video and retains the video frames where the face area can be detected; The feature extraction module uniformly samples the preprocessed video frame sequence to extract the local dynamic symmetry features of the target facial nerve branch area and the global static face features under extreme action states; The feature fusion and merging module fuses the local dynamic symmetry features and the global static face features, and adds and merges them with the global static face features to obtain the merged shared features; The subnetwork processing module simultaneously inputs the shared features into the classification subnetwork and the regression subnetwork to obtain the classification probability value of whether there is facial paralysis and the preliminary regression estimate of the severity of facial paralysis, and combines the classification probability value with the learnable parameters to perform enhanced regression on the preliminary regression estimate to obtain the final regression estimate; The evaluation module performs data post-processing on the regression estimation value to obtain the final facial paralysis severity assessment level.

9. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any one of the methods of claims 1 to 7.

10. A computing device, characterized in that include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any one of the methods described in claims 1 to 7.

Images