Facial paralysis auxiliary diagnosis method based on generative model

Abstract

The invention discloses a facial paralysis auxiliary diagnosis method based on a generative model. The facial paralysis auxiliary diagnosis method comprises the following steps: S1, constructing a model framework for normal face missing image restoration; s2, loading a portrait data set, and training the model until convergence by adopting a two-stage training method of step-by-step mask optimization; s3, a digital image of a facial paralysis patient is collected, a lesion part is shielded, and the processed image is input into the model; s4, comparing the generated normal face image with the original image, calculating the difference by jointly using the structural similarity index and the absolute difference, obtaining a lesion mask, and pasting the lesion mask back to the original image to obtain a facial paralysis mark image; and S5, taking an SWD evaluation result as a quantitative index to evaluate the disease degree of the facial paralysis patient. According to the method, the facial image of the healthy side of the facial paralysis patient can be utilized to generate the facial image after rehabilitation, and corresponding quantitative indexes are provided, so that a doctor can more intuitively evaluate the severity of facial paralysis and formulate a more accurate treatment scheme according to the severity of facial paralysis.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and deep learning, and in particular to a facial paralysis auxiliary diagnosis method based on a generative model. Background Art

[0002] Facial palsy, also known as facial nerve paralysis, is a common neurological disease characterized by unilateral or bilateral paralysis of facial muscles. Currently, the diagnosis of facial palsy mainly relies on clinical symptoms, physical examination, and auxiliary examinations, such as electrophysiological examination and imaging examination. However, these diagnostic methods have many limitations, such as strong subjectivity, long diagnosis time, and difficulty in early diagnosis.

[0003] In recent years, with the development of deep learning and generative modeling technology, automatic diagnosis methods based on images and videos have gradually attracted attention. Generative models based on image restoration, such as generative adversarial networks (GANs), variational autoencoders (VAEs), and Transformer models, have performed well in image generation and feature extraction, providing new ideas and methods for auxiliary diagnosis of facial paralysis.

[0004] The feature extraction of facial paralysis needs to take into account the subtle changes of facial muscles, such as the asymmetry of eyebrows, the degree of eyelid closure, the drooping or rising of the corners of the mouth, etc. These subtle changes are crucial for the accurate diagnosis of facial paralysis because they can reflect the functional status of the facial nerve. However, existing methods have some shortcomings.

[0005] On the one hand, the image and video data of patients with facial paralysis are relatively scarce, which makes it difficult to meet the large amount of data required by deep learning models. This may cause the model to overfit when learning facial paralysis features and fail to effectively generalize to new, unseen data, thus affecting the diagnostic accuracy of the model and limiting its application in clinical practice.

[0006] On the other hand, in practical applications, facial images may be affected by many factors, such as lighting conditions, shooting angles, expression changes, etc. When dealing with these interfering factors, the robustness of existing methods is often insufficient, resulting in unstable feature extraction results, which in turn affects the reliability of diagnosis. At the same time, facial images are usually high-dimensional, which makes feature extraction and processing complicated. When dealing with high-dimensional data, existing methods often face the problem of dimensionality disaster, resulting in high computational cost and low feature extraction efficiency.

[0007] The generative model based on image restoration can automatically learn the potential distribution of data and generate high-quality normal face images by masking half of the face of patients with facial paralysis through multi-scale feature extraction methods. This method can effectively use limited data to train high-performance models and improve the accuracy and robustness of feature extraction. Although generative models have been widely used in many fields, they have not yet been used for auxiliary diagnosis of facial paralysis. Therefore, it is urgent to develop an auxiliary diagnosis method for facial paralysis based on generative models to solve the above problems. Summary of the invention

[0008] The present invention proposes a facial paralysis auxiliary diagnosis method based on a generative model. A normal face image of a facial paralysis patient is generated through an image restoration model. The facial paralysis features are extracted by comparing the generated normal face image with the actual facial paralysis image, so as to solve the deficiencies of the existing methods in data and feature extraction.

[0009] To this end, the present invention provides a facial paralysis auxiliary diagnosis method based on a generative model, comprising: S1, constructing a framework of a normal face missing image restoration model to generate a predicted normal face image after masking the lesion part in the facial paralysis patient's image; S2, loading a portrait data set, adopting a two-stage training method of stepwise mask optimization, training the model until convergence, and saving the model weights to obtain a generative model; S3, collecting a digital image of the facial paralysis patient, normalizing it, masking the lesion part in the image, while retaining the normal area of ​​the face, and inputting the processed image into a computer program product; S4, performing a normalization process on the digital image of the facial paralysis patient, and performing a normalization process on the digital image of the facial paralysis patient. into the normal face missing image restoration model; S4, compare the generated normal face image with the original image, and use the structural similarity index SSIM and absolute difference absdiff to calculate the difference, obtain the lesion mask, and paste it back to the original image to obtain the facial paralysis marking map; S5, use the multi-scale statistical similarity index SWD to evaluate the similarity between the generated image and the input image, and use the obtained SWD evaluation result as a quantitative indicator to evaluate the severity of facial paralysis patients; S6, calculate the perceptual evaluation index FID and the local similarity evaluation index SSIM to evaluate the quality of the generated image.

[0010] The present invention also provides a computer-readable storage medium, comprising a computer program / instruction, which, when executed, is used to implement the steps of the above-mentioned facial paralysis auxiliary diagnosis method based on the generation model.

[0011] The present invention also provides a facial paralysis auxiliary diagnosis system based on a generation model, including a computer program / instruction, which is used to implement the steps of the above-mentioned facial paralysis auxiliary diagnosis method based on a generation model when executed.

[0012] The present invention also provides a computer program product, including a computer program / instruction, which, when executed, is used to implement the steps of the above-mentioned facial paralysis auxiliary diagnosis method based on the generation model.

[0013] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention generates a normal face image of a patient with facial paralysis by generating a model, compares it with the patient's real image, and calculates the total difference value, thereby providing a new and effective solution for auxiliary diagnosis of facial paralysis. This method can not only quantify the severity of facial paralysis, but also provide clinicians with a more intuitive diagnostic basis, thereby improving the accuracy and reliability of diagnosis.

[0014] (2) Although previous studies have made progress in generating facial images of patients with facial paralysis after recovery, existing models usually rely on datasets of facial information of patients with facial paralysis, which has significant limitations. First, the scarcity of samples of patients with facial paralysis makes data collection difficult and costly, and it is difficult to obtain a sufficient amount of high-quality data. Second, the collection of facial data of patients with facial paralysis may involve privacy issues, raise scientific research ethics issues, and further limit the collection and application of data. In contrast, the present invention proposes an innovative method that does not require direct collection of images of patients with facial paralysis as a training set. The method of the present invention uses a normal portrait dataset and analyzes normal facial information through a deep learning model to predict facial images of patients with facial paralysis after recovery. This method has significant advantages in generating facial images of patients with facial paralysis after recovery because it avoids the need to directly collect images of patients with facial paralysis and can achieve excellent image restoration effects using only normal facial image datasets, thereby reducing the difficulty and cost of data collection and protecting patient privacy.

[0015] (3) Although existing generative models based on image restoration perform well in restoring small areas of missing images, their performance drops significantly when dealing with large areas of occluded areas. These models are generally difficult to accurately reconstruct the features of large areas of missing facial areas. The method of the present invention involves masking a large area of ​​the affected side of a patient with facial paralysis and generating a normalized facial image, which requires the image restoration model to be able to accurately handle large areas of facial occlusion. Therefore, developing an image restoration model that can effectively handle such large areas of facial occlusion is of great significance for improving the accuracy and practicality of auxiliary diagnosis of facial paralysis.

[0016] (4) The present invention adopts a two-stage training method with gradual mask optimization. In the first stage, random mask processing is used to enable the model to learn the ability to repair faces in different missing situations, laying the foundation for subsequent more complex tasks. In the second stage, a specific half-occluded mask is used to further improve the model's adaptability and repair accuracy to specific missing patterns, thereby enhancing the model's pertinence and effectiveness in the task of auxiliary diagnosis of facial paralysis. This staged training method not only improves the generalization ability of the model, but also can more accurately generate normal face images when faced with new and unseen facial paralysis images, thereby improving the accuracy of diagnosis.

[0017] (5) The method of the present invention is not only innovative in theory, but also highly feasible in actual clinical application. By generating images of patients with facial paralysis after recovery, doctors can more intuitively assess the severity of facial paralysis and formulate more accurate treatment plans. In addition, the efficiency and accuracy of the method enable it to be quickly applied to large-scale clinical diagnosis, improve medical efficiency, and improve patient prognosis.

[0018] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings constituting a part of the present application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 A schematic diagram of a flow chart of a facial paralysis auxiliary diagnosis method based on a generative model according to the present invention; Figure 2 A model diagram is generated according to the present invention; Figure 3 A structural diagram of the Transformer main part of the model generated by the present invention; Figure 4 An example of a mask update strategy in the generation model of the present invention; Figure 5 This is a diagram showing the execution effect of the facial paralysis auxiliary diagnosis method based on the generative model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0021] See also Figure 1 , Figure 2 and Figure 5 The facial paralysis auxiliary diagnosis method based on the generation model of this embodiment includes the following steps S1-S6.

[0022] S1. Construct a model framework for normal face missing image restoration to generate a predicted normal face image after masking the lesion part in the image of a patient with facial paralysis.

[0023] A generative model framework based on image restoration is selected. The model needs to be specially designed for large-hole image restoration tasks, such as CoModGAN, LaMa, DeepFill v2, etc. Unlike the good performance of most image restoration models on small-area missing images, the existing models are not ideal when dealing with large-area occluded areas. After comparative analysis of a variety of large-hole image restoration models, the model framework constructed by the present invention has been carefully designed and optimized. It can not only more efficiently extract half of the facial features of patients with facial paralysis, but also generate highly realistic normal face images when half of the face is occluded. The architectural features of the framework include a convolution head, a Transformer body, and a convolution tail, such as Figure 2 shown.

[0024] Constructing the model specifically includes the following steps S101-S103.

[0025] S101, Construct the convolution head: This part contains five convolution layers. The first convolution layer is used to adjust the input dimension, expanding the number of input channels from 4 (3 channels for RGB images and 1 channel for masks) to 180. The next three convolution layers are used to downsample the resolution, with a step size of 2 for each convolution layer, gradually increasing the size of the feature map from Downsample to The fifth convolutional layer has 90 channels, which is used to extract half-face features to enhance the model's ability to capture half-face features of patients with facial paralysis, thereby improving diagnostic accuracy. The advantage of the fifth convolutional layer is that it improves the model's sensitivity to local features, making the generated normalized image closer to the real situation in half-face details, which is beneficial for subsequent diagnostic analysis; The convolution head receives an incomplete face image. And a given mask M:

[0026] Among them, I is the original image, M is the mask, Represents element-wise multiplication.

[0027] S102, build the Transformer main part: This part contains five stages of adjusted Transformer blocks. The adjusted Transformer blocks in each stage have been adjusted as follows: removing layer normalization, adopting feature interaction and fusion learning, applying optimized multi-head attention mechanism, performing window shifting, and adding global residual connection. Specifically, stage 1 contains two adjusted Transformer blocks with feature size ; Stage 2 contains three adjusted Transformer blocks with feature size ; Stage 3 contains four adjusted Transformer blocks with feature size ; Stage 4 contains three adjusted Transformer blocks with feature size ; Stage 5 contains two adjusted Transformer blocks with feature size ; Figure 3 and Figure 4 The structure of a single Transformer module after optimization and examples of mask update strategies in the multi-head contextual attention mechanism are shown respectively. These optimizations are aimed at improving the accuracy of distance dependency modeling. In particular, the multi-head contextual attention mechanism uses dynamic masks to effectively identify and utilize effective tags for distance dependency modeling. In addition, the improved Transformer module enhances the stability of the training process when dealing with complex masks.

[0028] In the traditional Transformer block, each submodule (such as the multi-head self-attention module and the MLP module) is preceded by layer normalization and then residual connections are applied. The specific structure is as follows:

[0029] Where x is the input feature tensor, represents the application of layer normalization operation to the input feature tensor x, Indicates applying a submodule to the input feature tensor x for processing. Represents applying a submodule to the layer-normalized input feature tensor for processing.

[0030] This structure helps stabilize training, but in the facial paralysis-assisted diagnosis task, layer normalization may amplify useless tokens, especially when dealing with large hole masks. In the facial images of patients with facial paralysis, there are significant differences between normal areas and damaged areas (i.e., areas affected by facial paralysis), and large hole masks usually cover the damaged areas.

[0031] During the training process, layer normalization uniformly normalizes all tokens (including those in normal and damaged areas), which may cause useless tokens (tokens carrying less useful information) in damaged areas to be over-amplified, thereby interfering with the model's learning of key features and making the training process unstable. Therefore, in order to better adapt to the facial paralysis auxiliary diagnosis task, the adjusted Transformer block removes layer normalization. This adjustment enables the model to focus more on learning and extracting information that is truly useful for facial paralysis diagnosis, avoiding interference from useless information, thereby improving the model's training stability and diagnostic accuracy in the facial paralysis auxiliary diagnosis task.

[0032] Instead of traditional residual learning, the adjusted Transformer block concatenates the output of the multi-head context attention module with the input features and performs fusion learning through the feature interaction module; the specific operations are as follows:

[0033] in, It is the kth stage l The output of the feature interaction module of each block, It means that it is the kth stage l -1 block of feature interaction module output, represents the fully connected layer, represents the multi-head contextual attention mechanism, It is a feature interaction module, which can be implemented as a multi-layer perceptron (MLP):

[0034] in, is the activation function, , are learnable weights, , is a learnable bias.

[0035] At the same time, the dropout mechanism is introduced in the feature interaction module to randomly discard some features and enhance the generalization ability of the model:

[0036] in, is the output feature tensor, It is a regularization technique used to prevent the model from overfitting the training data and improve the generalization ability of the model.

[0037] The multi-head contextual attention mechanism (MCA) uses a dynamic mask to select valid tokens and calculates the weighted sum of valid tokens; the following formula is used to calculate the attention in the multi-head contextual attention module :

[0038] Among them, Q, K, and V are query matrix, key matrix, and value matrix respectively. is a scaling factor used to prevent the dot product result from being too large and causing the gradient to disappear. Is a dynamically generated mask that represents valid tokens in the input image that are not covered by the mask. The mask is initialized by the input mask and automatically updated during the propagation process. Valid tokens will be used for attention calculation, while invalid tokens will be ignored.

[0039] Dynamic Mask It is expressed as:

[0040] Among them, the aggregation weight of invalid marks is basically 0.

[0041] The image restoration-based generative model proposed in the present invention adopts a dynamic mask updating strategy to optimize facial feature reconstruction, especially for processing large-area missing facial features in the auxiliary diagnosis of facial paralysis. Figure 4 The implementation process of this strategy is described in detail. The specific steps are as follows: the feature map is initially divided into 2×2 windows (shown in red in the figure), each of which contains several tags. Figure (a) shows the initial state, in which some tags are marked as invalid (the gray squares with diagonal lines represent invalid areas); Figure (b) shows that under the action of the attention mechanism, all tags in the upper left corner window first become valid (the squares not covered by diagonal lines represent valid areas), which marks the beginning of the mask update, and the valid tags will participate in the subsequent image reconstruction; Figure (c) shows the mask window moving to the lower right Pixels, ready to process the next window, the original upper left corner window becomes the middle window, and its mark state remains unchanged; Figure (d) shows the mask update of the mark in the new window to make it valid, repeating the process from Figure (a) to Figure (b), but applied to the new position; Figure (e) shows the mask window continues to move to the lower right pixels, ensuring that the 2×2 window in the initial state is processed to achieve global feature reconstruction; Figure (f) shows the last mask update of the original lower right corner window, ensuring that all marks eventually become valid and completing the mask update process.

[0042] After each attention operation, the mask is adjusted according to the following strategy: The mask will be automatically updated according to the results of the attention operation to reflect which areas have been successfully reconstructed during the calculation process. As long as there is at least one valid marker in the window, all markers in the window will be updated to valid after the attention. If all markers in the window are invalid, they will remain invalid after the attention. This update rule helps to gradually expand the valid pixel area, thereby incorporating more contextual information in subsequent processing. In this way, the model can more accurately reconstruct facial features that are missing in large areas, which is particularly important for auxiliary diagnosis of facial paralysis.

[0043] In addition, after each attention operation, Move the window size Pixels are connected across windows to achieve cross-window connections. This cross-window connection mechanism further enhances the model’s understanding of the global structure of the image, thereby improving the accuracy and efficiency of image restoration for large occluded areas.

[0044] After multiple Transformer blocks, the model described in S1 implements a global residual connection through a convolutional layer to maintain the integrity of the features. Specifically, the global residual connection uses a convolutional layer to transform the output features of the last Transformer block, and then adds the transformed features to the input features element by element to generate the final output features. This mechanism not only ensures the effective transfer of features between different levels, but also improves the optimization stability of the model and the quality of the generated images.

[0045] S103, construct the convolution tail part.

[0046] This part uses a convolution-based reconstruction module, which can restore the spatial resolution of the feature map to the size of the input image through upsampling operations. The output of the Transformer body is upsampled to match the size of the input image to generate a restored face image.

[0047] S2. Load the portrait dataset, adopt the two-stage training method of stepwise mask optimization, train the model until convergence, save the model weights, and obtain the generated model.

[0048] The CelebA-HQ portrait dataset used in the present invention is derived from the original CelebA dataset and contains more than 200,000 celebrity facial images. The dataset consists of high-definition normal face images, which are preprocessed in multiple steps to improve the quality. The specific preprocessing steps include: eliminating JPEG artifacts to reduce the impact of artifacts in the JPEG compression process; performing super-resolution processing to improve image resolution and enhance clarity; and high-quality resampling to ensure that the image maintains high quality during the sampling process. The high-quality images in the preprocessed CelebA-HQ dataset reduce noise and artifact interference, so that the trained generative model can effectively learn the details and structure of the face, thereby improving the quality and authenticity of the generated face image. The generative model trained using the CelebA-HQ dataset can effectively restore facial features, and can be well used to infer the normal face of patients with facial paralysis.

[0049] In this example, the dataset images are randomly divided into a training image set and a validation image set in a ratio of 7:3.

[0050] The two-stage training method using step-by-step mask optimization includes the following steps S201 - S202 .

[0051] S201. In the first stage, a mask is randomly generated on the training data set according to a preset rule to cover part of the image and simulate the missing content of the image. The training process is repeated until the generated model reaches the preset convergence standard on the validation set, and then the model weight is saved. The specific training process is as follows.

[0052] The CelebA-HQ dataset is divided into multiple batches, and the images in each batch are randomly masked and input into the model constructed by S1 for face image restoration; Traverse all batches of data sets, and in each training step, the model described in S1 generates face images and calculates the loss function; The adversarial loss is calculated using the following formula: ,

[0053] in, is the loss of the generator, is the loss of the discriminator, x and are real images and generated images respectively, and Represent the real image x and the generated image respectively Taking the average of all possible values ​​of and They represent the probability that the real image and the generated image are identified as real images respectively.

[0054] The generator aims to generate realistic images, making it difficult for the discriminator to distinguish between generated images and real images; the discriminator strives to accurately distinguish between the two.

[0055] The feature matching loss is calculated using the following formula: :

[0056] in, represents the activation value of the i-th layer of the pre-trained VGG-19 network, is the number of features of the layer, n is the number of layers selected, is the L1 norm.

[0057] The feature matching loss ensures that the generated images match the real images at multiple feature levels.

[0058] At the same time, the R1 regularization term is introduced into the loss function to improve the final performance of the generated model and generate higher quality images: in, is the output of the discriminator for image x, is the gradient of the discriminator to the real image x, Represents the L2 norm of the gradient.

[0059] The R1 regularization term prevents the discriminator from being overconfident by penalizing the discriminator for having too large a gradient with respect to the real image, thereby improving the stability of the generated model and the quality of the generated image.

[0060] The total loss is calculated using the following formula: in, , and are hyperparameters, which are set to 0.5, 20, and 0.2 in this example to balance the impact of different loss terms.

[0061] The loss of the model is calculated according to the loss function, and the parameters of the model are updated through back propagation.

[0062] S202. In the second stage, the generative model weights obtained by the first stage training are used to generate a half-masked mask for facial paralysis. Specifically, the mask is controlled to cover 50% of the image, that is, to cover the left or right half of the face image, so as to further improve the generalization ability of the model. Subsequently, the generative model is retrained using a data set with a specific mask. The training continues until the generative model meets the optimization criteria on the target task and the model weights are saved. The specific training steps are as follows.

[0063] Extract images from a dataset of patients with facial paralysis and apply the generated half-occluded masks; Input the masked image into the model constructed in step S1 to perform face image restoration; Traverse all batches of data sets. In each training step, the model generates restored face images and calculates the loss function. Calculate the loss of the model according to the total loss function described in S201, and update the parameters of the model through back propagation; Continue training until the generative model meets the optimization criteria on the target task and save the model weights.

[0064] The model training uses the Adam optimizer and sets the first-order moment estimation ( ) and the second moment estimate ( ), the batch size is 64, and the initial learning rate is set to .

[0065] S3. Collect digital images of patients with facial paralysis, normalize them, mask the diseased parts in the images, retain the normal areas of the face, and input the processed images into the normal face missing image repair model.

[0066] Step S3 specifically includes the following steps S301-S302.

[0067] S301, using a high-resolution camera to collect a facial image of a patient with facial paralysis, then performing a normalization process, and masking the abnormal side of the patient's face; Masking aims to simulate the absence of facial content, allowing the model to more accurately reconstruct those areas.

[0068] S302: Input the masked facial paralysis patient image into the generation model constructed in step S1, and set a random seed to generate multiple predicted normal face images.

[0069] S4. Compare the generated normal face image with the original image, and use the structural similarity index SSIM and absolute difference absdiff to calculate the difference, obtain the lesion mask, and paste it back to the original image to obtain the facial paralysis marking image.

[0070] Step S4 specifically includes the following steps S401-S407.

[0071] S401: Calculate the structural similarity index (SSIM) to quantify the structural similarity between the generated normal face image and the original image; SSIM is an indicator used to measure the structural similarity between two images, especially suitable for measuring the structural similarity of images. The calculation formula is as follows: Among them, x and y are two images to be compared, and are the means of images x and y, respectively. and are the variances of image x and image y, respectively. is the covariance of images x and y, and is a small constant used to avoid the denominator being zero, usually and , where L is the dynamic range of the image, and is a small constant. In this example, and .

[0072] S402, calculating the absolute difference (absdiff) to quantify the pixel-level difference between the generated normal face image and the original image; Absdiff is a simple pixel-level difference calculation method used to measure the pixel-level difference between two images. The calculation formula is as follows: Among them, x and y are the generated normal face image and the original image respectively. Represents the absolute difference between two images at each pixel position.

[0073] S403, generating a difference map to highlight the abnormal area between the normal face image and the original image; Specifically, the SSIM value is converted into a difference map ssim_diff, and 1-SSIM is used to represent the difference. The calculation result of the absolute difference absdiff is normalized to the range of [0, 1] for easy visualization, and the normalized absolute difference map is used as the difference map diff_map: , .

[0074] S404, using a weighted sum method, the SSIM difference map and the absdiff difference map are combined to generate a comprehensive difference map, and the calculation formula is as follows:

[0075] Among them, combined_diff represents the comprehensive difference map, is a weight parameter, usually between [0,1], used to balance the impact of SSIM and absdiff. The value is 0.5.

[0076] S405: combining the results of SSIM and absdiff, thresholding the comprehensive difference map to generate a lesion mask, which highlights the area in the original image that is significantly different from the generated normal face image; A threshold T is selected, and the selection of the threshold T can be carried out by a data-driven method or an adaptive method, and the part of the comprehensive difference map larger than T is marked as a lesion area: .

[0077] S406, repeating steps S401 to S405, comparing the multiple normal face images generated in step S3 with the original image, generating multiple different masks and fusing them to generate a more accurate and comprehensive lesion mask.

[0078] Vote for each mask and select the area that the majority of masks consider to be a lesion, reducing the misjudgment of a single mask:

[0079] Where N is the number of masks generated, Represents the i-th mask generated.

[0080] S407, pasting the generated lesion mask back to the original image to visualize the lesion area and assist doctors in diagnosis. Specifically, first read the original image y, and then use the following formula to apply the lesion mask final_mask to the original image y to highlight the lesion area:

[0081] Among them, highlighted_image represents a visualization image, color_mask is a color mask used to highlight the lesion area, and red is selected in this example.

[0082] S5. Use the multi-scale statistical similarity index SWD to evaluate the similarity between the generated image and the input image, and use the obtained SWD evaluation result as a quantitative indicator to evaluate the severity of facial paralysis patients.

[0083] In this step, the model that has been trained to convergence is used to input the images of patients with facial paralysis into the model to generate predicted normal face images. These images will be used for subsequent similarity evaluation.

[0084] The multi-scale statistical similarity index SWD is used to evaluate the similarity between the generated image and the input image.

[0085] The similarity between the generated normal face image and the original facial paralysis patient image is evaluated by the multi-scale statistical similarity index SWD. SWD can quantify the similarity between images and provide a numerical evaluation result.

[0086] The obtained similarity evaluation results are used as quantitative indicators to evaluate the severity of facial paralysis patients.

[0087] The SWD evaluation results are used as a quantitative indicator to evaluate the severity of facial paralysis patients. The lower the SWD value, the higher the similarity between the generated image and the input image, and the lower the severity of the disease; conversely, the higher the SWD value, the lower the similarity and the higher the severity of the disease.

[0088] The multi-scale statistical similarity index SWD is an indicator that measures the distance between two probability distributions, and is particularly suitable for high-dimensional data. SWD approximates the distance of high-dimensional distributions by projecting high-dimensional data into multiple random directions and then calculating the distance of the one-dimensional distribution on these projections. The specific calculation process is as follows.

[0089] First, the generated image and the real image are converted into Laplacian pyramid representation, starting from a low resolution and gradually increasing the resolution until the resolution of the original image is reached.

[0090] At each scale, local patches are randomly extracted from the generated image and the patient image. These patches can be windows of fixed size, such as 7x7 pixels. A large number of patches are extracted from each image to form two sets: the generated image patch set and the patient image patch set.

[0091] Then, the extracted image blocks are normalized to make their mean 0 and standard deviation 1 to eliminate the scale differences between different image blocks.

[0092] For each scale, the Sliced ​​Wasserstein Distance (SWD) between the generated image patch set and the real image patch set is calculated. Specifically, multiple random directions are selected, the image patches are projected onto these directions, and then the distance of the projected one-dimensional distribution is calculated. The SWD is calculated using the following formula:

[0093] in, and are the sets of the i-th generated image patch and the j-th patient image patch at the s-th scale, and are the inner products of the projection in the kth random direction and the i-th generated image patch and the j-th patient image patch at the s-th scale, are K random directions, K is the number of random directions, and N is the number of image patches.

[0094] Next, the SWD values ​​at different scales are aggregated by weighted average to obtain a comprehensive multi-scale statistical similarity index. When aggregating the SWD values ​​at different scales, the weight factor , the weight factor is adjusted according to the symmetry difference of the image blocks at each scale. Specifically, if the symmetry difference of the image blocks at a certain scale is large, it means that the features at this scale are more important for the diagnosis of facial paralysis, so a higher weight is given. Features at different scales contribute differently to the diagnosis of facial paralysis. By introducing weight factors, the importance of features at each scale can be more accurately reflected, thereby improving the diagnostic value of the comprehensive index. The specific formula is as follows:

[0095] in, is the weight factor of the sth scale, is the SWD value at the sth scale. The weight factor Adjustments are made based on the symmetry differences at each scale. The specific calculation formula is as follows: in, is the symmetry difference at the sth scale, calculated as follows:

[0096] in, and are the left and right image blocks of the ith image block at the sth scale, is the Euclidean norm.

[0097] This index can be used to quantitatively assess the severity of facial paralysis. Specifically, the lower the SWD value, the higher the similarity between the generated image and the patient's image at multiple scales, and thus the lower the severity of the disease; conversely, the higher the SWD value, the lower the similarity between the generated image and the patient's image at multiple scales, and thus the higher the severity of the disease.

[0098] S6. Calculate the perceptual evaluation index FID and the local similarity evaluation index SSIM to evaluate the quality of the generated image.

[0099] Step S6 specifically includes the following steps S601-S603.

[0100] S601: Calculate the FID perception evaluation index.

[0101] The FID is calculated using the following formula:

[0102] in, represents the squared Euclidean distance between mean vectors, represents the trace of the matrix (i.e., the sum of the diagonal elements of the matrix), Represents the square root of the product of two covariance matrices.

[0103] FID evaluates the overall distribution of the image set and can more comprehensively measure the quality of the generated images.

[0104] S602: Calculate the SSIM evaluation index.

[0105] SSIM is calculated using the following formula:

[0106] Among them, x and y are two images to be compared, and are the means of image x and image y respectively, and are the variances of image x and image y, respectively. is the covariance of images x and y, and is a small constant used to avoid the denominator being zero.

[0107] SSIM evaluates the local similarity of images and is robust to image noise and small disturbances, and can more accurately evaluate image quality.

[0108] S603: Calculate the comprehensive quantization error.

[0109] The comprehensive quantization error TotalError is calculated using the following formula:

[0110] in, and is a weight parameter, usually between [0,1], used to balance the impact of FID and SSIM.

[0111] In this example and The average value is 0.5.

[0112] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A facial paralysis auxiliary diagnosis method based on a generative model, characterized in that: include: S1. Construct a framework of a normal face missing image restoration model to generate a predicted normal face image after masking the lesion part in the image of a patient with facial paralysis; S2, load the portrait dataset, adopt the two-stage training method of stepwise mask optimization, train the model until convergence, save the model weights, and obtain the generated model; S3, collecting digital images of patients with facial paralysis, normalizing them, masking the diseased parts in the images, retaining the normal areas of the face, and inputting the processed images into a normal face missing image restoration model; S4, comparing the generated normal face image with the original image, and jointly using the structural similarity index SSIM and the absolute difference absdiff to calculate the difference, obtain the lesion mask, and paste it back to the original image to obtain the facial paralysis marking image; S5. Use the multi-scale statistical similarity index SWD to evaluate the similarity between the generated image and the input image, and use the obtained SWD evaluation result as a quantitative indicator to evaluate the severity of facial paralysis patients; S6. Calculate the perceptual evaluation index FID and the local similarity evaluation index SSIM to evaluate the quality of the generated image.

2. The facial paralysis auxiliary diagnosis method based on the generative model according to claim 1 is characterized in that: The framework of the normal face missing image restoration model in step S1 introduces an improved Transformer structure and a dynamic mask update strategy after each attention calculation. Step S1 includes: S101, the model architecture consists of three parts: a convolution head, an improved Transformer body and a convolution tail. The steps of constructing a normal face missing image restoration model framework are as follows: Construct the convolution head part, which includes five convolution layers. The first convolution layer is used to adjust the input dimension and expand the number of input channels from 4 channels to 180 channels. The next three convolution layers are used to downsample the resolution. The step size of each convolution layer is 2, gradually increasing the size of the feature map from Downsample to ; The fifth convolutional layer has 90 channels, which is used to extract half-face features; Construct the main part of Transformer, which includes five processing stages. In stage 1, the feature size of the Transformer block is ; The feature size of the Transformer block in stage 2 is ; The feature size of the Transformer block in stage 3 is ; The feature size of the Transformer block in stage 4 is ; The feature size of the Transformer block in stage 5 is ; Construct the convolution tail part, which uses a convolution-based reconstruction module to restore the spatial resolution of the feature map to the size of the input image through upsampling operations, thereby generating a repaired normal face image; S102, the improved Transformer blocks all use a non-normalized structure, which includes a multi-head context attention module MCA and a feature interaction module, wherein the output and input features of the multi-head context attention module are concatenated and then fused and learned through the feature interaction module, wherein the MCA module uses a dynamic mask to select valid tokens and calculates the weighted sum of valid tokens, and a dropout mechanism is introduced in the feature interaction module; S103. The dynamic mask update strategy follows the following steps: after each attention operation, the mask is automatically updated to reflect the successfully reconstructed area. If there is at least one valid mark in the window, all marks in the window are updated to be valid; if all marks in the window are originally invalid, they continue to remain invalid after the attention operation. Through continuous window movement and repeated application of the attention mechanism, the mask will be gradually updated until it is completely valid. This process helps to gradually expand the effective pixel area and integrate more contextual information, thereby improving the accuracy of facial feature reconstruction in auxiliary diagnosis of facial paralysis.

3. The facial paralysis auxiliary diagnosis method based on the generative model according to claim 2 is characterized in that: The method by which the MCA module calculates the weighted sum of valid tokens is as follows: ， in, is a dynamically generated mask used to represent valid tokens in the input image that are not covered by the mask. It is expressed as: ， in, is the attention in the multi-head contextual attention module, Q, K, and V are the query matrix, key matrix, and value matrix respectively. is the scaling factor.

4. The facial paralysis auxiliary diagnosis method based on the generative model according to claim 1 is characterized in that: In step S2, the two-stage training method comprises the following steps: S201, the first stage: randomly generating a mask on the training data set to cover part of the face image and simulate the missing content of the face image. The training process is repeated until the generated model reaches a preset convergence standard on the validation set, and then the model weight is saved; S202, second stage: using the weights of the generative model obtained from the first stage training, generate a half-face mask, that is, mask the left or right half of the face image. The training process is repeated until the generative model reaches a preset convergence standard on the validation set, and then the model weights are saved.

5. The facial paralysis auxiliary diagnosis method based on the generative model according to claim 1 is characterized in that: Step S3 includes: S301, using a high-resolution camera device to collect a facial image of a patient with facial paralysis, and selecting the abnormal side of the face for masking; S302: Input the masked facial paralysis patient image into the trained model, and set a random seed to generate a predicted normal face image.

6. The facial paralysis auxiliary diagnosis method based on generative model according to claim 1 is characterized in that: Step S4 includes: S401, calculating the structural similarity index SSIM to quantify the structural similarity between the generated normal face image and the original image, wherein SSIM measures the structural similarity of the image, and the calculation formula is as follows: ， Among them, x and y are two images to be compared, and are the means of images x and y, respectively. and are the variances of image x and image y, respectively. is the covariance of images x and y, and is a small constant used to avoid the denominator being zero, usually and , where L is the dynamic range of the image, and is a small constant; S402, calculating the absolute difference absdiff to quantify the pixel-level difference between the generated normal face image and the original image, wherein absdiff is used to measure the pixel-level difference between two images, and the calculation formula is as follows: ， Among them, x and y are the generated normal face image and the original image respectively. Represents the absolute difference between the two images at each pixel position, S403, generating a difference map to highlight the abnormal area between the normal face image and the original image, wherein the SSIM value is converted into a difference map ssim_diff, 1-SSIM is used to represent the difference, the calculation result of the absolute difference absdiff is normalized to the range of [0, 1] for visualization, and the normalized absolute difference map is used as the difference map diff_map: ， ， S404, using a weighted sum method, the SSIM difference map and the absdiff difference map are combined to generate a comprehensive difference map, and the calculation formula is as follows: ， Among them, combined_diff represents the comprehensive difference map, It is a weight parameter, usually between [0,1], used to balance the influence of SSIM and absdiff; S405. Combine the results of SSIM and absdiff, perform thresholding on the comprehensive difference map, generate a lesion mask, the mask highlights the area in the original image that is significantly different from the generated normal face image, select a threshold T, the selection of the threshold T can be a data-driven method or an adaptive method, and mark the part of the comprehensive difference map that is greater than T as a lesion area: ， S406. Repeat steps S401 to S405 to generate multiple different masks and fuse them to generate a more accurate and comprehensive lesion mask. Vote for each mask and select the area that the majority of masks consider to be a lesion to reduce the misjudgment of a single mask. ， Where N is the number of masks generated, represents the i-th mask generated; S407, pasting the generated lesion mask back to the original image to visualize the lesion area and assist doctors in diagnosis, wherein the original image y is first read, and then the lesion mask final_mask is applied to the original image y using the following formula to highlight the lesion area: ， Among them, highlighted_image represents a visualization image, and color_mask is a color mask used to highlight the lesion area.

7. The facial paralysis auxiliary diagnosis method based on generative model according to claim 1 is characterized in that: Step S5 includes: Convert the generated image and the real image into a Laplacian pyramid representation, starting from a low resolution and gradually increasing the resolution until the resolution of the original image is reached; At each scale, local image patches are randomly extracted from the generated images and patient images. These image patches are windows of fixed size. A large number of image patches are extracted from each image to form two sets: a generated image patch set and a patient image patch set. The extracted image blocks are normalized to make their mean value 0 and standard deviation 1, eliminating the scale differences between different image blocks; For each scale, the SWD between the generated image patch set and the real image patch set is calculated, where the SWD is calculated using the following formula: ， in, is the set of the i-th generated image patch at the s-th scale, is the set of the jth patient image patch at the sth scale, is the inner product of the projection in the kth random direction and the i-th generated image block at the s-th scale, is the inner product of the projection in the kth random direction and the jth patient image patch at the sth scale, are K random directions, K is the number of random directions, N is the number of image patches, The SWD values ​​at different scales are aggregated by weighted averaging to obtain a comprehensive multi-scale statistical similarity index. When aggregating the SWD values ​​at different scales, a weight factor is introduced. , the weight factor is adjusted according to the symmetry difference of the image block at each scale. The specific formula is as follows: ， in, is the weight factor of the sth scale, is the SWD value at the sth scale, and the weight factor Adjustments are made based on the symmetry differences at each scale. The specific calculation formula is as follows: ， in, is the symmetry difference at the sth scale, calculated as follows: ， in, and are the left and right image blocks of the ith image block at the sth scale, is the Euclidean norm.

8. A computer-readable storage medium comprising a computer program / instructions, characterized in that: When executed, the program / instruction is used to implement the steps of the facial paralysis auxiliary diagnosis method based on the generative model according to any one of claims 1-7.

9. A facial paralysis auxiliary diagnosis system based on a generative model, comprising a computer program / instruction, characterized in that: When executed, the program / instruction is used to implement the steps of the facial paralysis auxiliary diagnosis method based on the generative model according to any one of claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that When executed, the program / instruction is used to implement the steps of the facial paralysis auxiliary diagnosis method based on the generative model according to any one of claims 1-7.

Images