Face image refinement method based on multi-modal flaw reasoning

Abstract

The invention discloses a face image refinement method based on multi-mode flaw reasoning, and the method comprises the steps: obtaining and preprocessing a face flaw image and a user demand text, and carrying out the feature coding of the face flaw image and the user demand text, and obtaining a face flaw tensor and a text condition evidence vector; processing the face flaw tensor through a prior extraction convolutional network to obtain a flaw prior vector; performing forward uncertainty reasoning on the text condition evidence vector and the flaw prior vector through a mask flaw reasoning layer to obtain a refined posteriori vector; processing the refinement posterior vector through a refinement mask deconvolution network to obtain a refinement mask; performing further fine trimming on the face flaw tensor based on the fine trimming mask to obtain a face fine trimming tensor; and performing feature decoding on the face refinement tensor to obtain a final face refinement image. According to the method, the problems of insufficient repairing quality and high data labeling difficulty of fine repairing of the face image are solved, and personalized face fine repairing with low labeling cost and high detail retention is realized.

Description

Technical Field

[0001] This invention relates to the technical field of computer vision and image editing, and in particular to a method for refining facial images based on multimodal defect reasoning. Background Technology

[0002] Fine-grained facial image restoration is an important technique for improving image quality by repairing imperfections in a given facial photograph. It is widely used in image processing, digital entertainment, and other fields. As a typical deep learning-based image editing task, its core is to achieve precise repair of imperfections while preserving facial details.

[0003] Currently, existing technologies for facial image retouching mainly fall into two categories: one is to directly refine facial images using convolutional neural networks constructed through computer vision deep learning techniques. This approach relies on large-scale data pre-training, resulting in high costs and insufficient retouching quality. The other approach involves providing fine-grained blemish templates from facial images, using these templates as conditions for indirect retouching by the convolutional neural network. However, this approach overly relies on high-quality blemish masks annotated by human experts, leading to significant data annotation difficulties and potential performance degradation in practical applications due to insufficient granularity of the blemish masks. In summary, existing technologies struggle to balance retouching cost, detail preservation, and retouching accuracy, necessitating a facial image retouching solution that efficiently utilizes multimodal information and reduces reliance on large-scale annotated data. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and propose a face image retouching method based on multimodal defect reasoning. By fusing multimodal information of face defect images and user requirement text, it achieves personalized face retouching with low annotation cost and high detail preservation.

[0005] To achieve the above objectives, the technical solution provided by this invention is: a method for refining facial images based on multimodal defect reasoning, comprising the following steps: S1: Acquire multimodal data, including images of facial imperfections and user request text, and preprocess the images of facial imperfections and user request text to obtain preprocessed images of facial imperfections of uniform size and preprocessed user request text with punctuation removed. S2: Perform feature encoding on the preprocessed facial defect image and the preprocessed user requirement text, encode the preprocessed facial defect image into a facial defect tensor, and encode the preprocessed user requirement text into a text conditional evidence vector. S3: Design a prior extraction convolutional network that includes downsampling convolutional layers and global max pooling layers. The face blemish tensor is downsampled and convolved through the downsampling convolutional layers to extract prior features of the face blemish image. The non-maximum blemish features of the prior features of the face blemish image are suppressed by the global max pooling layer to obtain the blemish prior vector. S4: Design a mask flaw inference layer that includes a face refinement evidence matrix generation mechanism and a face refinement evidence matrix inference mechanism. By performing forward uncertainty inference on the aforementioned text conditional evidence vector and flaw prior vector, the text conditional evidence vector and flaw prior vector are fused in a multimodal manner, thereby reducing the uncertainty of the flaw prior vector and obtaining the refinement posterior vector and the amount of uncertainty reduction in the refinement posterior. Specifically, the face refinement evidence matrix generation mechanism is used to perform uncertainty inference on the flaw prior vector and text conditional evidence vector to obtain a face refinement evidence matrix. Each element in the face refinement evidence matrix represents the confidence level of the text conditional evidence vector corresponding to a certain dimension of the flaw prior vector. The face refinement evidence matrix inference mechanism is used to perform a cross product calculation on the face refinement evidence matrix and the flaw prior vector to obtain a refinement posterior vector. The refinement posterior vector is then subtracted element-wise from the flaw prior vector, and the average value is calculated to obtain the amount of uncertainty reduction in the refinement posterior. S5: Design a fine-tuning mask deconvolution network that includes shape transformation operation and deconvolution layer. First, the fine-tuning posterior vector is processed by the shape transformation operation to obtain fine-tuning mask features. Then, the fine-tuning mask features are decoded by the deconvolution layer to obtain the fine-tuning mask. S6: Based on the refined mask obtained in step S5, a pixel-level gating attention mechanism is used to further refine the face blemish tensor obtained in step S2. That is, according to the features of the face blemish tensor that are suppressed and enhanced by the refined mask, the features of the face blemish tensor without blemishes are suppressed, and the features of the face blemish tensor with blemishes are enhanced, and finally the face refined tensor is obtained. S7: Perform feature decoding on the face refinement tensor to obtain the final face refinement image.

[0006] Furthermore, in step S1, the facial blemish image is an 8-bit JPEG or PNG compressed digital image in the RGB color space, such as 512*512, and the user request text is rich text format ASCII encoded English text.

[0007] Further, in step S2, the preprocessed facial blemish image is normalized 512*512*3 single-precision floating-point image data, with normalized mean values ​​of [0.5, 0.5, 0.5] for the red, green, and blue channels, and standard deviations of [0.5, 0.5]. [0.5, 0.5]; The preprocessed user request text filters out ASCII punctuation marks, including commas, periods, and quotation marks, and converts them to lowercase; The preprocessed facial blemish image is encoded into a facial blemish tensor using an SD-ft-MSE variational encoder, wherein the SD-ft-MSE variational encoder compresses the preprocessed facial blemish image from a 512*512*3 image pixel space to a low-dimensional 64*64*4 facial blemish tensor latent space to reduce computation; The preprocessed user request text is encoded into text conditional evidence using a byte-pair encoding word segmenter, wherein the byte-pair encoding word segmenter has a vocabulary size of 49408, splits the text into no more than 20 tokens, and inserts special words <|startoftext|> and <|endoftext|> at the beginning and end of sentences, and finally encodes it into a dense text conditional evidence vector of dimension 512.

[0008] Furthermore, step S3 includes the following steps: S31: Using the facial blemish tensor as the input to the prior extraction convolutional network, coarse-grained blemish feature recognition is performed through a downsampling convolutional layer with a kernel size of 7*7 and a stride of 2 to reduce the interference of non-blemish noise features in the facial blemish tensor. Subsequently, the blemish features are normalized and irrelevant features are suppressed by a batch normalization layer and a ReLU activation function layer, respectively, to obtain coarse-grained prior features of the facial blemish image. The coarse-grained prior features of the facial blemish image are in the form of a 32*32*256 single-precision floating-point tensor, which stores the prior feature information of the facial blemish region in the facial blemish image. The batch normalization layer and the ReLU activation function layer map the value range of the coarse-grained prior features of the facial blemish image to [0, +∞), where the larger the value, the higher the probability that the coarse-grained prior features of the facial blemish image contain blemishes in the facial blemish image. S32: Using the coarse-grained prior features of the facial blemish image as input, fine-grained blemish feature recognition is performed through a downsampling convolutional layer with a kernel size of 3*3 and a stride of 1 to obtain fine-grained prior features of the facial blemish image. These fine-grained prior features are represented as single-precision floating-point tensors in the form of 32*32*256, used to further suppress non-critical blemish patterns in the coarse-grained prior features. These non-critical blemish patterns are sub-precedence features of 3*3*256 sampled from the coarse-grained prior features, divided into non-face region blemish patterns and dense face blemish patterns. The non-face region blemish patterns are background patterns that do not exist in the face region of the facial blemish image. The dense face blemish patterns are generated by significant face blemish patterns, characterized by a large number of blemish features representing the same facial blemish in the facial blemish image. Suppressing the non-critical defect patterns reduces the defect feature values ​​of the 3*3*256 region of the coarse-grained facial defect image prior features corresponding to the non-critical defect patterns. By performing non-maximum defect feature suppression on the fine-grained facial defect image prior features, single-precision format facial defect image prior features are obtained. The non-maximum defect feature suppression includes a ReLU activation function layer, a 3*3 kernel size max pooling operation, and a bilinear interpolation operation performed sequentially. The ReLU activation function layer is used to suppress negative feature values ​​in the fine-grained facial defect image prior features to ensure the effectiveness of the max pooling operation. The max pooling operation compares the defect feature values ​​of each element in the fine-grained facial defect image prior features with the elements in its 8-neighborhood, and retains the element with the largest defect feature value to obtain 11*11*256 single-precision format facial defect image prior features. S33: Using the prior features of the single-precision format face blemish image as input, global average pooling is performed to average the first and second dimensions of the prior features of the single-precision format face blemish image, which is in the form of 11*11*256. This realizes the mapping of the prior features of the face blemish image from pixels to vector space, resulting in a 1*256 global face blemish prior vector. The global face blemish prior vector is normalized through a Sigmoid activation function layer to obtain the first round of blemish prior vector with a value range of [0,1]. Each element in the blemish prior vector is the initial inference confidence of different channel dimensions in the face blemish tensor.

[0009] Furthermore, step S4 includes the following steps: S41: Using the defect prior vector as the input to the mask defect inference layer, in the face retouching evidence matrix generation mechanism, the defect prior vector is sequentially passed through a query fully connected layer and a ReLU activation function layer to obtain an evidence query vector with a shape of 1*256, and then converted into an evidence query of the form 4*64 through a scaling operation; the text conditional evidence vector is sequentially passed through a key fully connected layer and a ReLU activation function layer to obtain an evidence key vector with a shape of 1*256, and then converted into an evidence key of the form 4*64 through a scaling operation; wherein, the weight initialization method of the above query fully connected layer and key fully connected layer is He normal initialization, with an input dimension of 256 and an output dimension of 256; S42: Perform a cross product calculation on the evidence query and the evidence key to achieve flawed prior vector reasoning based on text-conditional evidence vectors, resulting in a 4*64*64 face refinement attention matrix. Normalize the face refinement attention matrix using the Softmax function to obtain a face refinement evidence matrix with a shape of 4*64*64. The Softmax function performs probability normalization on the first dimension of the face refinement attention matrix to ensure that the sum of the evidence confidence scores in the second and third dimensions of the face refinement evidence matrix is ​​1, thereby quantifying the contribution of different text-conditional evidence vectors to flawed prior vector reasoning. Each piece of evidence in the face refinement evidence matrix has a shape of 4*1*1, representing the confidence score of different dimensions of the flawed prior vector in the presence of different pieces of evidence in the text-conditional evidence vector. S43: The defect prior vector is passed sequentially through a value fully connected layer and a ReLU activation function layer to obtain an evidence value vector. The weights of the value fully connected layer are initialized using He normal initialization, with an input dimension of 256 and an output dimension of 256. The evidence value vector has a shape of 1*256 and is converted into evidence values ​​of the form 4*64 through a scaling operation. The face refinement evidence matrix is ​​cross-multiplied with the evidence values ​​to obtain a refinement posterior of the form 4*64. Then, the shape is transformed to obtain a refinement posterior vector of the form 1*256. Each element in the refinement posterior vector represents the inference confidence of the defect features of different channel dimensions in the face defect tensor after inference in the mask defect inference layer. S44: Calculate the element-wise average of the refined posterior vector to obtain the inference confidence of the refined posterior. Compare the inference confidence of the refined posterior vector with the convergence threshold. When the inference confidence of the refined posterior vector is higher than the convergence threshold, use the refined posterior vector as the flawed prior vector for the next round, and repeat steps S41~S43 for inference with the flawed prior vector. When the inference confidence of the refined posterior vector is lower than the convergence threshold, use the refined posterior vector as the input of the subsequent refined mask deconvolution network.

[0010] Further, in step S5, the deconvolution layer consists of a convolutional layer with a kernel size of 3*3 and a stride of 1, a ReLU activation function layer, a bilinear interpolation layer, a convolutional layer with a kernel size of 1*1 and a stride of 1, and a Sigmoid activation function layer. The bilinear interpolation layer doubles the length and width of the refined mask feature, the 1*1 convolutional layer compresses the number of output channels to 1, thus converting the refined mask feature into a single-channel feature map, and the Sigmoid activation function layer performs normalization, converting the single-channel feature map into a refined mask.

[0011] Furthermore, in step S6, the pixel-level gating attention mechanism is implemented using the Hadamard product operator.

[0012] Furthermore, in step S7, the face refinement tensor is decoded using an SD-ft-MSE variational decoder to obtain the final face refinement image.

[0013] Compared with the prior art, the present invention has the following advantages and beneficial effects: 1. This invention reduces the reliance on manually labeled defective templates by using text and image multimodal fusion reasoning, thereby reducing the difficulty and cost of data labeling.

[0014] 2. Employing coarse and fine-grained tiered blemish feature extraction improves the accuracy and fineness of blemish recognition, effectively preserving the original details of the face.

[0015] 3. The multi-round iteration based on uncertainty reasoning enhances the matching degree between the refinement results and user needs.

[0016] 4. Pixel-level gating attention mechanism enables targeted fine-tuning, ensuring the repair effect of flawed areas while avoiding excessive modification of flawless areas. Attached Figure Description

[0017] Figure 1 is an architectural diagram of the method of the present invention. Detailed Implementation

[0018] The present invention will be further described below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0019] As shown in Figure 1, this embodiment discloses a face image retouching method based on multimodal defect reasoning, which is used to accurately repair defects in face images. The specific implementation process is as follows: 1) Acquire multimodal data, including images of facial imperfections and user request text, and perform data preprocessing on the aforementioned images of facial imperfections and user request text to obtain preprocessed images of facial imperfections of uniform size and preprocessed user request text with punctuation removed; wherein, the images of facial imperfections are 8-bit JPEG or PNG compressed digital images in RGB color space with a format of 512*512, and the user request text is rich text format ASCII encoded English text.

[0020] 2) Perform feature encoding on the preprocessed facial defect image and the preprocessed user requirement text, encode the preprocessed facial defect image into a facial defect tensor, and encode the preprocessed user requirement text into a text conditional evidence vector.

[0021] The preprocessed facial blemish image is normalized 512*512*3 single-precision floating-point image data, with normalized mean values ​​of [0.5, 0.5, 0.5] and standard deviations of [0.5, 0.5, 0.5]. [0.5]; The preprocessed user requirement text filters out ASCII punctuation marks, including commas, periods, and quotation marks, and converts them to lowercase; The preprocessed facial blemish image is encoded into a facial blemish tensor using an SD-ft-MSE variational encoder, wherein the SD-ft-MSE variational encoder compresses the preprocessed facial blemish image from a 512*512*3 image pixel space to a low-dimensional 64*64*4 facial blemish tensor latent space to reduce computation; The preprocessed user requirement text is encoded into text conditional evidence using a byte-pair encoding word segmenter, wherein the byte-pair encoding word segmenter has a vocabulary size of 49408, splits the text into no more than 20 tokens, and inserts special words <|startoftext|> and <|endoftext|> at the beginning and end of sentences, and finally encodes it into a dense text conditional evidence vector of dimension 512.

[0022] 3) Design a prior extraction convolutional network containing downsampling convolutional layers and global max pooling layers. The facial blemish tensor is downsampled and convolved through the downsampling convolutional layers to extract prior features of the facial blemish image. The non-maximum blemish features of the prior features are then suppressed by the global max pooling layer to obtain the blemish prior vector. This includes the following steps: 3.1) Using the facial blemish tensor as the input to the prior extraction convolutional network, coarse-grained blemish feature recognition is performed through a downsampling convolutional layer with a kernel size of 7*7 and a stride of 2 to reduce the interference of non-blemish noise features in the facial blemish tensor. Subsequently, the blemish features are normalized and irrelevant features are suppressed by a batch normalization layer and a ReLU activation function layer, respectively, to obtain coarse-grained prior features of the facial blemish image. The coarse-grained prior features of the facial blemish image are in the form of a 32*32*256 single-precision floating-point tensor, which stores the prior feature information of the facial blemish region in the facial blemish image. The batch normalization layer and the ReLU activation function layer map the value range of the coarse-grained prior features of the facial blemish image to [0, +∞), where the larger the value, the higher the probability that the coarse-grained prior features of the facial blemish image contain blemishes in the facial blemish image. 3.2) Using the coarse-grained prior features of the facial blemish image as input, fine-grained blemish feature recognition is performed through a downsampling convolutional layer with a kernel size of 3*3 and a stride of 1 to obtain fine-grained prior features of the facial blemish image. These fine-grained prior features are represented as single-precision floating-point tensors in the form of 32*32*256, used to further suppress non-critical blemish patterns in the coarse-grained prior features. These non-critical blemish patterns are sub-precedent features of 3*3*256 sampled from the coarse-grained prior features, divided into non-face region blemish patterns and dense face blemish patterns. The non-face region blemish patterns are background patterns that do not exist in the face region of the facial blemish image. The dense face blemish patterns are generated by significant face blemish patterns, characterized by a large number of blemish features representing the same facial blemish in the facial blemish image. Suppressing the non-critical defect patterns reduces the defect feature values ​​of the 3*3*256 region of the coarse-grained facial defect image prior features corresponding to the non-critical defect patterns. By performing non-maximum defect feature suppression on the fine-grained facial defect image prior features, single-precision format facial defect image prior features are obtained. The non-maximum defect feature suppression includes a ReLU activation function layer, a 3*3 kernel size max pooling operation, and a bilinear interpolation operation performed sequentially. The ReLU activation function layer is used to suppress negative feature values ​​in the fine-grained facial defect image prior features to ensure the effectiveness of the max pooling operation. The max pooling operation compares the defect feature values ​​of each element in the fine-grained facial defect image prior features with the elements in its 8-neighborhood, and retains the element with the largest defect feature value to obtain 11*11*256 single-precision format facial defect image prior features. 3.3) Using the prior features of the single-precision format face blemish image as input, global average pooling is performed to average the first and second dimensions of the prior features of the single-precision format face blemish image, which is in the form of 11*11*256. This realizes the mapping of the prior features of the face blemish image from pixels to vector space, resulting in a 1*256 global face blemish prior vector. The global face blemish prior vector is normalized through a Sigmoid activation function layer to obtain the first round of blemish prior vector with a value range of [0,1]. Each element in the blemish prior vector is the initial inference confidence of different channel dimensions in the face blemish tensor.

[0023] 4) Design a mask flaw inference layer that includes a face refinement evidence matrix generation mechanism and a face refinement evidence matrix inference mechanism. By performing forward uncertainty inference on the aforementioned text conditional evidence vector and flaw prior vector, the text conditional evidence vector and flaw prior vector are fused in a multimodal manner to reduce the uncertainty of the flaw prior vector and obtain the refinement posterior vector and the amount of uncertainty reduction in the refinement posterior. Specifically, the face refinement evidence matrix generation mechanism is used to perform uncertainty inference on the flaw prior vector and text conditional evidence vector to obtain the face refinement evidence matrix. Each element in the face refinement evidence matrix represents the confidence level of the text conditional evidence vector corresponding to a certain dimension of the flaw prior vector. The face refinement evidence matrix inference mechanism is used to perform a cross product calculation on the face refinement evidence matrix and the flaw prior vector to obtain the refinement posterior vector. The refinement posterior vector is then subtracted element-wise from the flaw prior vector, and the average value is calculated to obtain the amount of uncertainty reduction in the refinement posterior. This includes the following steps: 4.1) Using the aforementioned defect prior vector as input to the mask defect inference layer, in the face retouching evidence matrix generation mechanism, the defect prior vector is sequentially passed through a query fully connected layer and a ReLU activation function layer to obtain an evidence query vector with a shape of 1*256, and then converted into an evidence query of the form 4*64 through a scaling operation; the text conditional evidence vector is sequentially passed through a key fully connected layer and a ReLU activation function layer to obtain an evidence key vector with a shape of 1*256, and then converted into an evidence key of the form 4*64 through a scaling operation; wherein, the weight initialization method of the aforementioned query fully connected layer and key fully connected layer is He normal initialization, with an input dimension of 256 and an output dimension of 256; 4.2) Perform a cross product calculation on the evidence query and the evidence key to achieve flawed prior vector reasoning based on text-conditional evidence vectors, resulting in a 4*64*64 face refinement attention matrix. Normalize the face refinement attention matrix using the Softmax function to obtain a face refinement evidence matrix with a shape of 4*64*64. The Softmax function performs probability normalization on the first dimension of the face refinement attention matrix to ensure that the sum of the evidence confidence scores in the second and third dimensions of the face refinement evidence matrix is ​​1, thereby quantifying the contribution of different text-conditional evidence vectors to flawed prior vector reasoning. Each piece of evidence in the face refinement evidence matrix has a shape of 4*1*1, representing the confidence score of different dimensions of the flawed prior vector in the presence of different pieces of evidence in the text-conditional evidence vector. 4.3) The defect prior vector is passed sequentially through a value fully connected layer and a ReLU activation function layer to obtain an evidence value vector. The weights of the value fully connected layer are initialized using He normal initialization, with an input dimension of 256 and an output dimension of 256. The evidence value vector has a shape of 1*256 and is converted into evidence values ​​of the form 4*64 through a scaling operation. The face refinement evidence matrix and the evidence values ​​are cross-multiplied to obtain a refinement posterior of the form 4*64. The shape is then transformed to obtain a refinement posterior vector of the form 1*256. Each element in the refinement posterior vector represents the inference confidence of the defect features of different channel dimensions in the face defect tensor after inference in the mask defect inference layer. 4.4) Calculate the element-wise average of the refined posterior vector to obtain the inference confidence of the refined posterior. Compare the inference confidence of the refined posterior vector with a convergence threshold (e.g., 0.85). When the inference confidence of the refined posterior vector is higher than the convergence threshold, use the refined posterior vector as the flawed prior vector for the next round, and repeat steps 4.1) to 4.3) for inference. When the inference confidence of the refined posterior vector is lower than the convergence threshold, use the refined posterior vector as the input of the subsequent refined mask deconvolution network.

[0024] 5) Design a refinement mask deconvolution network that includes shape transformation operations and deconvolution layers. First, the refinement posterior vector is transformed to obtain refinement mask features. Then, the refinement mask features are decoded by the deconvolution layers to obtain the refinement mask. The deconvolution layers consist of a 3*3 kernel convolution layer with a stride of 1, a ReLU activation function layer, a bilinear interpolation layer, a 1*1 kernel convolution layer with a stride of 1, and a Sigmoid activation function layer. The bilinear interpolation layer doubles the length and width of the refinement mask features. The 1*1 kernel convolution layer compresses the number of output channels to 1, thus converting the refinement mask features into a single-channel feature map. The Sigmoid activation function layer normalizes the single-channel feature map and converts it into the refinement mask.

[0025] 6) Based on the refined mask obtained in step 5), a pixel-level gated attention mechanism is used to further refine the face blemish tensor obtained in step 2). That is, according to the features of the face blemish tensor suppressed and enhanced by the refined mask, the features of the face blemish tensor without blemishes are suppressed, and the features of the face blemish tensor with blemishes are enhanced, and finally the face refined tensor is obtained; wherein, the pixel-level gated attention mechanism is implemented using the Hadamard product operator.

[0026] 7) Use the SD-ft-MSE variational decoder to perform feature decoding on the face refinement tensor to obtain the final face refinement image.

[0027] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for refining facial images based on multimodal defect reasoning, characterized in that, Includes the following steps: S1: Acquire multimodal data, including images of facial imperfections and user request text, and preprocess the images of facial imperfections and user request text to obtain preprocessed images of facial imperfections of uniform size and preprocessed user request text with punctuation removed. S2: Perform feature encoding on the preprocessed facial defect image and the preprocessed user requirement text, encode the preprocessed facial defect image into a facial defect tensor, and encode the preprocessed user requirement text into a text conditional evidence vector. S3: Design a prior extraction convolutional network that includes downsampling convolutional layers and global max pooling layers. The face blemish tensor is downsampled and convolved through the downsampling convolutional layers to extract prior features of the face blemish image. The non-maximum blemish features of the prior features of the face blemish image are suppressed by the global max pooling layer to obtain the blemish prior vector. S4: Design a mask flaw inference layer that includes a face refinement evidence matrix generation mechanism and a face refinement evidence matrix inference mechanism. By performing forward uncertainty inference on the aforementioned text conditional evidence vector and flaw prior vector, the text conditional evidence vector and flaw prior vector are fused in a multimodal manner, thereby reducing the uncertainty of the flaw prior vector and obtaining the refinement posterior vector and the amount of uncertainty reduction in the refinement posterior. Specifically, the face refinement evidence matrix generation mechanism is used to perform uncertainty inference on the flaw prior vector and text conditional evidence vector to obtain a face refinement evidence matrix. Each element in the face refinement evidence matrix represents the confidence level of the text conditional evidence vector corresponding to a certain dimension of the flaw prior vector. The face refinement evidence matrix inference mechanism is used to perform a cross product calculation on the face refinement evidence matrix and the flaw prior vector to obtain a refinement posterior vector. The refinement posterior vector is then subtracted element-wise from the flaw prior vector, and the average value is calculated to obtain the amount of uncertainty reduction in the refinement posterior. S5: Design a fine-tuning mask deconvolution network that includes shape transformation operation and deconvolution layer. First, the fine-tuning posterior vector is processed by the shape transformation operation to obtain fine-tuning mask features. Then, the fine-tuning mask features are decoded by the deconvolution layer to obtain the fine-tuning mask. S6: Based on the refined mask obtained in step S5, a pixel-level gating attention mechanism is used to further refine the face blemish tensor obtained in step S2. That is, according to the features of the face blemish tensor that are suppressed and enhanced by the refined mask, the features of the face blemish tensor without blemishes are suppressed, and the features of the face blemish tensor with blemishes are enhanced, and finally the face refined tensor is obtained. S7: Perform feature decoding on the face refinement tensor to obtain the final face refinement image.

2. The face image retouching method based on multimodal defect reasoning according to claim 1, characterized in that, In step S1, the image of facial imperfections is an 8-bit JPEG or PNG compressed digital image in the RGB color space, with a format of 512*512, and the user request text is rich text format ASCII encoded English text.

3. The face image retouching method based on multimodal defect reasoning according to claim 1, characterized in that, In step S2, the preprocessed facial blemish image is normalized 512*512*3 single-precision floating-point image data, with normalized mean values ​​of [0.5, 0.5, 0.5] and standard deviations of [0.5, 0.5, 0.5]. [0.5]; The preprocessed user requirement text filters out ASCII punctuation marks, including commas, periods, and quotation marks, and converts them to lowercase; The preprocessed facial blemish image is encoded into a facial blemish tensor using an SD-ft-MSE variational encoder, wherein the SD-ft-MSE variational encoder compresses the preprocessed facial blemish image from a 512*512*3 image pixel space to a low-dimensional 64*64*4 facial blemish tensor latent space to reduce computation; The preprocessed user requirement text is encoded into text conditional evidence using a byte-pair encoding word segmenter, wherein the byte-pair encoding word segmenter has a vocabulary size of 49408, splits the text into no more than 20 tokens, and inserts special words <|startoftext|> and <|endoftext|> at the beginning and end of sentences, and finally encodes it into a dense text conditional evidence vector of dimension 512.

4. The face image retouching method based on multimodal defect reasoning according to claim 1, characterized in that, Step S3 includes the following steps: S31: Using the facial blemish tensor as the input to the prior extraction convolutional network, coarse-grained blemish feature recognition is performed through a downsampling convolutional layer with a kernel size of 7*7 and a stride of 2 to reduce the interference of non-blemish noise features in the facial blemish tensor. Subsequently, the blemish features are normalized and irrelevant features are suppressed by a batch normalization layer and a ReLU activation function layer, respectively, to obtain coarse-grained prior features of the facial blemish image. The coarse-grained prior features of the facial blemish image are in the form of a 32*32*256 single-precision floating-point tensor, which stores the prior feature information of the facial blemish region in the facial blemish image. The batch normalization layer and the ReLU activation function layer map the value range of the coarse-grained prior features of the facial blemish image to [0, +∞), where the larger the value, the higher the probability that the coarse-grained prior features of the facial blemish image contain blemishes in the facial blemish image. S32: Using the coarse-grained prior features of the facial blemish image as input, fine-grained blemish feature recognition is performed through a downsampling convolutional layer with a kernel size of 3*3 and a stride of 1 to obtain fine-grained prior features of the facial blemish image. These fine-grained prior features are represented as single-precision floating-point tensors in the form of 32*32*256, used to further suppress non-critical blemish patterns in the coarse-grained prior features. These non-critical blemish patterns are sub-precedence features of 3*3*256 sampled from the coarse-grained prior features, divided into non-face region blemish patterns and dense face blemish patterns. The non-face region blemish patterns are background patterns that do not exist in the face region of the facial blemish image. The dense face blemish patterns are generated by significant face blemish patterns, characterized by a large number of blemish features representing the same facial blemish in the facial blemish image. Suppressing the non-critical defect patterns reduces the defect feature values ​​of the 3*3*256 region of the coarse-grained facial defect image prior features corresponding to the non-critical defect patterns. By performing non-maximum defect feature suppression on the fine-grained facial defect image prior features, single-precision format facial defect image prior features are obtained. The non-maximum defect feature suppression includes a ReLU activation function layer, a 3*3 kernel size max pooling operation, and a bilinear interpolation operation performed sequentially. The ReLU activation function layer is used to suppress negative feature values ​​in the fine-grained facial defect image prior features to ensure the effectiveness of the max pooling operation. The max pooling operation compares the defect feature values ​​of each element in the fine-grained facial defect image prior features with the elements in its 8-neighborhood, and retains the element with the largest defect feature value to obtain 11*11*256 single-precision format facial defect image prior features. S33: Using the prior features of the single-precision format face blemish image as input, global average pooling is performed to average the first and second dimensions of the prior features of the single-precision format face blemish image, which is in the form of 11*11*256. This realizes the mapping of the prior features of the face blemish image from pixels to vector space, resulting in a 1*256 global face blemish prior vector. The global face blemish prior vector is normalized through a Sigmoid activation function layer to obtain the first round of blemish prior vector with a value range of [0,1]. Each element in the blemish prior vector is the initial inference confidence of different channel dimensions in the face blemish tensor.

5. The face image retouching method based on multimodal defect reasoning according to claim 1, characterized in that, Step S4 includes the following steps: S41: Using the defect prior vector as the input to the mask defect inference layer, in the face retouching evidence matrix generation mechanism, the defect prior vector is sequentially passed through a query fully connected layer and a ReLU activation function layer to obtain an evidence query vector with a shape of 1*256, and then converted into an evidence query of the form 4*64 through a scaling operation; the text conditional evidence vector is sequentially passed through a key fully connected layer and a ReLU activation function layer to obtain an evidence key vector with a shape of 1*256, and then converted into an evidence key of the form 4*64 through a scaling operation; wherein, the weight initialization method of the above query fully connected layer and key fully connected layer is He normal initialization, with an input dimension of 256 and an output dimension of 256; S42: Perform a cross product calculation on the evidence query and the evidence key to achieve flawed prior vector reasoning based on text-conditional evidence vectors, resulting in a 4*64*64 face refinement attention matrix. Normalize the face refinement attention matrix using the Softmax function to obtain a face refinement evidence matrix with a shape of 4*64*64. The Softmax function performs probability normalization on the first dimension of the face refinement attention matrix to ensure that the sum of the evidence confidence scores in the second and third dimensions of the face refinement evidence matrix is ​​1, thereby quantifying the contribution of different text-conditional evidence vectors to flawed prior vector reasoning. Each piece of evidence in the face refinement evidence matrix has a shape of 4*1*1, representing the confidence score of different dimensions of the flawed prior vector in the presence of different pieces of evidence in the text-conditional evidence vector. S43: The defect prior vector is passed sequentially through a value fully connected layer and a ReLU activation function layer to obtain an evidence value vector. The weights of the value fully connected layer are initialized using He normal initialization, with an input dimension of 256 and an output dimension of 256. The evidence value vector has a shape of 1*256 and is converted into evidence values ​​of the form 4*64 through a scaling operation. The face refinement evidence matrix is ​​cross-multiplied with the evidence values ​​to obtain a refinement posterior of the form 4*64. Then, the shape is transformed to obtain a refinement posterior vector of the form 1*256. Each element in the refinement posterior vector represents the inference confidence of the defect features of different channel dimensions in the face defect tensor after inference in the mask defect inference layer. S44: Calculate the element-wise average of the refined posterior vector to obtain the inference confidence of the refined posterior. Compare the inference confidence of the refined posterior vector with the convergence threshold. When the inference confidence of the refined posterior vector is higher than the convergence threshold, use the refined posterior vector as the flawed prior vector for the next round, and repeat steps S41~S43 for inference with the flawed prior vector. When the inference confidence of the refined posterior vector is lower than the convergence threshold, use the refined posterior vector as the input of the subsequent refined mask deconvolution network.

6. The face image retouching method based on multimodal defect reasoning according to claim 1, characterized in that, In step S5, the deconvolution layer consists of a convolutional layer with a kernel size of 3*3 and a stride of 1, a ReLU activation function layer, a bilinear interpolation layer, a convolutional layer with a kernel size of 1*1 and a stride of 1, and a Sigmoid activation function layer. The bilinear interpolation layer doubles the length and width of the refined mask feature, the 1*1 convolutional layer compresses the number of output channels to 1, thus converting the refined mask feature into a single-channel feature map, and the Sigmoid activation function layer normalizes the single-channel feature map, converting it into a refined mask.

7. The face image retouching method based on multimodal defect reasoning according to claim 1, characterized in that, In step S6, the pixel-level gating attention mechanism is implemented using the Hadamard product operator.

8. The face image retouching method based on multimodal defect reasoning according to claim 1, characterized in that, In step S7, the face refinement tensor is decoded using an SD-ft-MSE variational decoder to obtain the final face refinement image.

Images