Blind person face image restoration method and device, electronic equipment and storage medium

Abstract

The invention provides a blind person face image restoration method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining a to-be-restored image of a blind person face; inputting the to-be-restored image into the image restoration model to obtain a target restoration image output by the image restoration model; wherein the image restoration model comprises a degradation prediction network, a text prompt module, a visual prompt module and a decoding module; according to the method, a degradation prediction network can effectively evaluate the degradation degree of an input to-be-restored image and provide a corresponding degradation probability weight so as to enhance the prompt generation capability of a subsequent task; moreover, the introduction of the degradation prediction network improves the adaptability of the image restoration model, and enables the model to adjust a prompt strategy according to the degradation degree, thereby optimizing the final image restoration effect of the face of the blind person, and improving the fidelity of image restoration.

Description

Technical Field

[0001] This invention relates to the field of image restoration technology, and in particular to a method, apparatus, electronic device, and storage medium for restoring images of blind people's faces. Background Technology

[0002] With the continuous development of facial image restoration technology, various restoration methods have emerged, including denoising, deblurring, super-resolution, and facial structure completion. These methods have achieved certain results in restoration tasks targeting specific degrees of degradation. However, these single-task restoration methods often struggle to effectively handle the superposition of multiple degradation factors when faced with facial images of varying degrees of degradation, thus limiting their adaptability and robustness in practical applications. Furthermore, because the models lack awareness of the degradation itself, the restoration process may result in overfitted smooth facial planes that lose facial details, or underfitted blurred faces that fail to effectively restore clear facial structures.

[0003] To address this issue, unified face inpainting methods capable of adapting to varying degrees of degradation have garnered increasing attention. These methods attempt to process face images ranging from mild to severe degradation through a unified framework. However, these methods typically rely on joint training on different degradation levels, which can lead to inter-task interference, affecting inpainting quality and limiting the model's generalization ability in real-world scenarios. Particularly when degradation levels are unevenly distributed, the model tends to favor optimization for specific degradation levels, making it difficult to maintain consistent inpainting quality globally.

[0004] Stable Diffusion (SD) models have gained widespread attention in image inpainting due to their ability to generate high-quality images and perform well under various degradation conditions. This model guides the restoration of degraded face images by utilizing high-quality image priors. While it has made some progress in inpainting tasks, its performance still relies on carefully designed cue priors. Especially when dealing with face images with complex and diverse degradation levels, constructing cue priors that accurately capture degradation information remains a challenge. If the model lacks precise perception of the degree of degradation, the restoration result may exhibit an overly smooth facial surface lacking realistic texture, or, in cases of severe degradation, an inability to effectively restore facial features, leaving the face still blurry and distorted. Summary of the Invention

[0005] This invention provides a method, apparatus, electronic device, and storage medium for restoring blind face images, which addresses the shortcomings of existing stable diffusion models that rely on high-quality image priors to guide the restoration of degraded face images. Although these models have made some progress in restoration tasks, their performance still depends on carefully designed cue priors.

[0006] This invention provides a method for restoring facial images of blind people, comprising the following steps:

[0007] Obtain the image of the blind person's face to be repaired;

[0008] The image to be repaired is input into the image repair model to obtain the target repaired image output by the image repair model;

[0009] The image restoration model includes a degradation prediction network, a text prompting module, a visual prompting module, and a decoding module. The degradation prediction network is used to determine degradation probability weights based on the first face image features of the image to be restored. The text prompting module is used to determine text prompts based on the first face image features and to determine text prompt priors based on the text prompts and the degradation probability weights. The visual prompting module is used to determine visual prompts based on the second face image features of the image to be restored and to determine visual prompt priors based on the visual prompts and the degradation probability weights. The decoding module is used to determine multimodal prompts based on the text prompt priors and the visual prompt priors, and to perform image restoration on the image to be restored based on the multimodal prompts to obtain the target restored image.

[0010] According to a method for restoring a blind person's face image provided by the present invention, the step of determining the textual prompt of the first face image features includes:

[0011] Based on the cross-modal mapping network in the text prompt module, the features of the first face image are cross-modal mapped to obtain a cross-modal representation of the text; the cross-modal mapping network is constructed based on a state-space model;

[0012] A set of text prompt parameters is extracted from a preset text prompt pool, and the cross-modal representation of the text is concatenated with its corresponding text prompt parameters to obtain the text prompt.

[0013] According to a method for restoring a blind person's face image provided by the present invention, the step of determining the visual cues of the second face image features of the image to be restored includes:

[0014] The image to be repaired is projected into the latent embedding space to obtain the latent feature representation corresponding to the image to be repaired;

[0015] The latent feature representation is decomposed into image spatial latent features at different scales.

[0016] Based on the cross-modal mapping network in the visual cueing module, cross-modal mapping is performed on the latent features of the image space at different scales to obtain the visual cross-modal representations at each level;

[0017] The visual cues are determined based on the visual cross-modal representations of each level.

[0018] According to a method for restoring blind facial images provided by the present invention, determining the visual cues based on the visual cross-modal representations at each level includes:

[0019] A set of visual cue parameters is extracted from a preset visual cue pool; the visual cue parameters include learnable parameters at different degradation levels;

[0020] The degree of matching between the visual cross-modal representations of each level and the visual cue parameters is determined by a cross-attention network, and the fused cue features are determined based on the degree of matching.

[0021] The visual cue is obtained by concatenating the fused cue features with the latent features of the image space.

[0022] According to a method for restoring blind facial images provided by the present invention, the training steps of the image restoration model include:

[0023] An initial image restoration model, a sample image to be restored, and a label image of the sample image to be restored are obtained; the sample image to be restored is obtained by performing degradation operations on the label image, including Gaussian blur, downsampling, noise addition, and JPEG compression.

[0024] Extract the sample face image features of the sample image to be repaired, and input the sample face image features into the initial image repair model to obtain the predicted repaired image output by the initial image repair model;

[0025] Based on the difference between the predicted restored image and the labeled image, a target loss is determined, and the initial image restoration model is iterated based on the target loss to obtain the image restoration model.

[0026] According to the present invention, a method for restoring a blind face image is provided, wherein the first face image feature is obtained by inputting the image to be restored into an image encoder in a contrastive language image pre-training model;

[0027] The second facial image feature is obtained by inputting the image to be repaired into a variational autoencoder.

[0028] The present invention also provides a device for restoring facial images of blind people, comprising the following units:

[0029] The acquisition unit is used to acquire the image of the blind face to be restored;

[0030] An input unit is used to input the image to be repaired into the image repair model to obtain the target repaired image output by the image repair model;

[0031] The image restoration model includes a degradation prediction network, a text prompting module, a visual prompting module, and a decoding module. The degradation prediction network is used to determine degradation probability weights based on the first face image features of the image to be restored. The text prompting module is used to determine text prompts based on the first face image features and to determine text prompt priors based on the text prompts and the degradation probability weights. The visual prompting module is used to determine visual prompts based on the second face image features of the image to be restored and to determine visual prompt priors based on the visual prompts and the degradation probability weights. The decoding module is used to determine multimodal prompts based on the text prompt priors and the visual prompt priors, and to perform image restoration on the image to be restored based on the multimodal prompts to obtain the target restored image.

[0032] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described methods for restoring blind face images.

[0033] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the blind face image restoration method as described above.

[0034] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described methods for restoring blind face images.

[0035] This invention provides a method, apparatus, electronic device, and storage medium for restoring blind faces. The method involves acquiring an image of a blind face to be restored; inputting the image to be restored into an image restoration model to obtain a target restored image output by the model; wherein the image restoration model includes a degradation prediction network, a text prompt module, a visual prompt module, and a decoding module; the degradation prediction network determines degradation probability weights based on the first face image features of the image to be restored; the text prompt module determines text prompts based on the first face image features and determines text prompt priors based on the text prompts and degradation probability weights; the visual prompt module determines visual prompts based on the second face image features of the image to be restored and determines visual prompt priors based on the visual prompts and degradation probability weights; and the decoding module determines multimodal prompts based on the text prompt priors and visual prompt priors, and performs image restoration on the image to be restored based on the multimodal prompts to obtain the target restored image. In this method, the degradation prediction network can effectively evaluate the degree of degradation of the input image to be restored and provide corresponding degradation probability weights to enhance the prompt generation ability of subsequent tasks. Furthermore, the introduction of the degradation prediction network improves the adaptability of the image restoration model, enabling it to adjust the prompt strategy according to the degree of degradation, thereby optimizing the final image restoration effect of blind faces and improving the fidelity of image restoration. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 This is one of the flowcharts illustrating the blind face image restoration method provided by the present invention.

[0038] Figure 2 This is the second flowchart of the blind face image restoration method provided by the present invention.

[0039] Figure 3 This is a schematic diagram of the degradation prediction network provided by the present invention.

[0040] Figure 4 This is a schematic diagram of the process for determining prior text prompts provided by the present invention.

[0041] Figure 5 This is a schematic diagram of the state space feature extraction and transfer module based on the state space model provided by the present invention.

[0042] Figure 6 This is a schematic diagram of the structure of the blind face image restoration device provided by the present invention.

[0043] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0045] In recent years, with the continuous advancement of facial image restoration technology, various restoration methods have been widely proposed, including facial denoising and deblurring techniques. These methods have shown significant effectiveness in handling single degrees of degradation. However, when faced with facial images of varying degrees of degradation, these single-task-focused restoration techniques often struggle to effectively address the superposition and dynamic changes of different degradation factors, limiting their adaptability and stability in practical applications. To address this limitation, research has gradually shifted towards training and fine-tuning models using pre-generated degraded faces, assisting the models in learning the dynamic changes of degradation factors.

[0046] While this approach enhances the model's adaptability to different degrees of degradation, it also has significant limitations. Because the training data is based on synthetically generated degraded faces, the model is prone to overfitting to this data, resulting in overly smoothed facial surfaces in the restoration results, lacking realistic texture. Furthermore, it performs poorly on real-world face degradation datasets, struggling to accurately reconstruct actual degraded facial images, thus limiting its generalization ability and impacting its feasibility in real-world applications.

[0047] To address the above problems, this invention provides a method for restoring facial images of blind people. Figure 1 This is one of the flowcharts illustrating the method for restoring blind facial images provided by the present invention. Figure 2 This is the second flowchart illustrating the blind face image restoration method provided by the present invention, as shown below. Figure 1 , Figure 2 As shown, the method includes steps 110 and 120.

[0048] Step 110: Obtain the image of the blind person's face to be repaired.

[0049] Specifically, the image to be repaired of a blind person's face can be obtained. Here, the image to be repaired is the image that needs to be repaired. It can be pre-acquired by an image acquisition device, captured in real time, or downloaded or scanned by the Internet. This embodiment of the invention does not make specific limitations in this regard.

[0050] Step 120: Input the image to be repaired into the image repair model to obtain the target repaired image output by the image repair model;

[0051] The image restoration model includes a degradation prediction network, a text prompting module, a visual prompting module, and a decoding module. The degradation prediction network is used to determine degradation probability weights based on the first face image features of the image to be restored. The text prompting module is used to determine text prompts based on the first face image features and to determine text prompt priors based on the text prompts and the degradation probability weights. The visual prompting module is used to determine visual prompts based on the second face image features of the image to be restored and to determine visual prompt priors based on the visual prompts and the degradation probability weights. The decoding module is used to determine multimodal prompts based on the text prompt priors and the visual prompt priors, and to perform image restoration on the image to be restored based on the multimodal prompts to obtain the target restored image.

[0052] Specifically, after obtaining the image to be repaired, the image to be repaired can be input into the image repair model to obtain the target repaired image output by the image repair model.

[0053] Here, the image restoration model includes a degradation prediction network, a text prompting module, a visual prompting module, and a decoding module.

[0054] Figure 3 This is a schematic diagram of the degradation prediction network provided by the present invention, as shown below. Figure 3 As shown, the degradation prediction network is used to determine degradation probability weights based on the first face image features of the image to be restored. The degradation prediction network can sequentially include a linear layer, an activation layer, a linear layer, and a degradation probability predictor. The linear layer can be a Linear layer, the activation layer can be a ReLU layer, and finally, the text cue prior is determined based on the degradation probability weights output by the degradation probability predictor, and the visual cue prior is determined based on the degradation probability weights output by the degradation probability predictor.

[0055] Here, the first face image feature is obtained by inputting the image to be repaired into the image encoder of the CLIP (Contrastive Language-Image Pre-training) model.

[0056] The training steps for the degradation prediction network include:

[0057] Step S1: Obtain a high-resolution face image, crop and adjust it to a 512×512 image size, and apply various degradation operations such as blurring, noise addition, downsampling, and JPEG compression to synthesize a low-quality image. Then, use the CLIP pre-trained model to extract the low-resolution image features of the low-quality image.

[0058] Step S2: Adjust the hyperparameters of the degradation equation and divide the low-quality images into three training datasets according to the different degrees of degradation, corresponding to mild, moderate and severe degradation categories respectively;

[0059] Step S3: Construct a degradation probability predictor in the degradation prediction network, using a fully connected layer structure and pre-training based on the cross-entropy loss function, so that the degradation probability predictor can correctly classify the degree of degradation of the input low-resolution image features.

[0060] Step S4: After pre-training, use the degradation probability predictor to process any low-resolution input features and output a set of probability values ​​p. i ={p1, p2, ..., p N}, where N is the number of degradation levels, and the generated probability values ​​are used as dynamic weights (degradation probability weights) to guide the subsequent text and visual cue generation process;

[0061] The various degradation operations mainly employ Gaussian blur, downsampling, noise addition, and JPEG compression, which can be specifically represented as follows:

[0062]

[0063] Where x represents a high-resolution face image, I represents a low-quality image, and kσ is a Gaussian blur kernel with standard deviation σ, controlling the degree of image blur. r This represents a downsampling operation with a scaling factor of r, where n δ This represents the addition of random noise with an amplitude of δ in JPEG. q This indicates that image compression with a quality factor of q is applied to the image, simulating the quality degradation caused by compression.

[0064] In the process of degraded image synthesis, different combinations of hyperparameters such as Gaussian blur, downsampling, noise factor, and JPEG quality factor are used to generate low-resolution degraded face image data with different degrees of degradation, which are divided into mild, moderate, and severe degradation categories. These are then used to train a degradation probability predictor using cross-entropy loss. The optimization objective formula is described as follows:

[0065]

[0066] Among them, L DPP Represents the cross-entropy loss, yo，c A binary indicator used to mark whether an observed sample o belongs to category c; p o，c This represents the probability predicted by the model.

[0067] Understandably, degradation prediction networks can effectively assess the degree of degradation of the input image to be restored and provide corresponding degradation probability weights to enhance the prompt generation capability of subsequent tasks. Since blind face restoration tasks may generate multiple high-quality results from the same low-resolution image, and image restoration models lack sufficient sensitivity to different degrees of degradation, the introduction of this degradation prediction network improves the adaptability of the image restoration model, enabling it to adjust the prompt strategy according to the degree of degradation, thereby optimizing the final image restoration effect for blind faces.

[0068] It should be noted that the degradation prediction network uses multi-level processing and the cross-entropy loss function to optimize the network. By analyzing the degradation features of the input blind face image to be restored, it can accurately identify common degradation levels such as noise, blur, and low light, and output the corresponding degradation probability distribution.

[0069] Here, the text prompting module is used to determine text prompts for the features of the first face image and to determine the text prompt prior based on the text prompt and degradation probability weights. Specifically, the cross-modal mapping network in the text prompting module can be used to perform cross-modal mapping on the features of the first face image to obtain a cross-modal representation of the text. Then, a set of text prompt parameters extracted from a preset text prompt pool is concatenated with the cross-modal representation to determine the text prompt. The text prompting module includes an image encoder of the CLIP pre-trained model, a text encoder of the CLIP pre-trained model, linear layers, and a cross-modal mapping network. The cross-modal mapping network is a Mamba network implemented based on a state-space model for state-space modeling, and the linear layers can be linear layers.

[0070] Cross-modal feature generation based on state-space networks utilizes the long-range dependency modeling capability of state-space models to translate low-quality feature embeddings into cross-modal feature representations. The state-space network model, combined with linear layers, convolutional layers, and activation functions, can efficiently extract deep degradation information from face images.

[0071] The visual cue module is used to determine visual cues for the second face image features of the image to be restored. Based on the visual cues and degradation probability weights, a prior visual cue is determined. The second face image features are obtained by inputting the image to be restored into a Variational Autoencoder (VAE). The visual cue module includes a VAE encoder, ResBlock, Mamba, a cross-attention mechanism, and ResBlock.

[0072] The decoding module is used to determine multimodal cues based on textual and visual cues priors, and to perform image inpainting on the image to be repaired based on the multimodal cues, thereby obtaining the target repaired image.

[0073] Here, the decoding module can be a decoder for a stable diffusion model, such as a U-shaped denoising network. It can input multimodal cue priors into a pre-trained stable diffusion model, perform cue-guided diffusion denoising and restoration through the U-shaped denoising network, and decode the output of the stable diffusion model into a high-definition restored face image, i.e., the target restored image.

[0074] Understandably, the stable diffusion model uses multimodal cues as prior information to guide the generation activities of the sampling process, enabling the network to dynamically adjust and accurately repair different degrees of degradation.

[0075] In some preferred embodiments, the face image restoration model based on stable diffusion of different degradation levels decomposes multimodal prompts into two parts: local features and global features, which guide the generation path in the diffusion process respectively. This significantly improves the effect and efficiency of face restoration under different degradation levels, effectively utilizes dynamic weights and conditional information, and gradually reduces prediction noise, thereby achieving accurate restoration for specific degradation levels.

[0076] In some preferred embodiments, a CLIP pre-trained image encoder can be used to extract low-quality image features, and a state-space model can be used for cross-modal transformation to generate text prompts suitable for text encoding. A dynamic weighting mechanism is employed to adjust the contribution ratio of different prompts in the text prompt vocabulary, enabling the text prompts to match image features at different degradation levels. The CLIP text encoder is then used to further optimize the text prompts, enabling them to accurately describe facial details and improve the restoration effect of the stable diffusion model in different degradation scenarios.

[0077] In some preferred embodiments, the low-quality face image feature extraction process achieves collaborative learning of multimodal features by jointly optimizing the parameters of the CLIP encoder and VAE, which not only enhances the visual representation ability of the image, but also improves the sensitivity of feature extraction to different degrees of degradation.

[0078] The method provided in this invention acquires an image of a blind face to be restored; inputs the image to be restored into an image restoration model to obtain a target restored image output by the image restoration model; wherein, the image restoration model includes a degradation prediction network, a text prompt module, a visual prompt module, and a decoding module; the degradation prediction network is used to determine degradation probability weights based on the first face image features of the image to be restored; the text prompt module is used to determine text prompts based on the first face image features, and to determine text prompt priors based on the text prompts and degradation probability weights; the visual prompt module is used to determine visual prompts based on the second face image features of the image to be restored, and to determine visual prompt priors based on the visual prompts and degradation probability weights; the decoding module is used to determine multimodal prompts based on the text prompt priors and visual prompt priors, and to perform image restoration on the image to be restored based on the multimodal prompts to obtain the target restored image. In this method, the degradation prediction network can effectively evaluate the degree of degradation of the input image to be restored and provide corresponding degradation probability weights to enhance the prompt generation ability of subsequent tasks. Furthermore, the introduction of the degradation prediction network improves the adaptability of the image restoration model, enabling it to adjust the prompt strategy according to the degree of degradation, thereby optimizing the final image restoration effect of blind faces and improving the fidelity of image restoration.

[0079] The method provided in this invention proposes a face image restoration method for different degradation severity by combining cue word generation learning guided by state space networks and diffusion restoration guided by multimodal cue word priors. By providing fine-grained visual and textual modal cue word priors containing sufficient high-definition prior information to a stable diffusion model, and leveraging the advantages of state space networks in long sequence feature modeling and transfer, as well as the superiority of diffusion models in high-quality face image generation and restoration, this method enables the acquisition of degradation probability weights for different degradation degrees and guides the generation of high-quality cue words for face restoration.

[0080] Based on the above embodiments, the step of determining the text prompt of the first facial image features includes:

[0081] Step 210: Based on the cross-modal mapping network in the text prompt module, perform cross-modal mapping on the features of the first face image to obtain a cross-modal representation of the text; the cross-modal mapping network is constructed based on a state-space model;

[0082] Step 220: Extract a set of text prompt parameters from a preset text prompt pool, and concatenate the cross-modal representation of the text with its corresponding text prompt parameters to obtain the text prompt.

[0083] Specifically, the cross-modal mapping network in the text prompt module can be used to perform cross-modal mapping on the features of the first face image to obtain a cross-modal representation of the text, so as to adapt to different degrees of degradation.

[0084] Then, a set of text cue parameters is extracted from a pre-defined text cue pool. This set of parameters contains text features tailored to different levels of degradation severity to capture detailed information at different quality levels. Furthermore, the cross-modal representation of the text is concatenated with its corresponding text cue parameters to obtain a text cue applicable to different degradation levels.

[0085] The preset text cue pool comes from a multimodal cue lexicon, which consists of a set of learnable parameters. High-quality text and visual embeddings are generated for each degree of degradation. After cross-modal concatenation, the text is processed by a text encoder or cross-attention module to form accurate cue. Based on the degradation degree probability weights obtained from the degradation prediction network, the cue lexicon is weighted and fused to construct different cue lexicons for different predicted degradation degrees, ensuring the accuracy of the model's perception of degradation degree and the computational efficiency of inference during the inference stage.

[0086] Finally, the concatenated text prompts are encoded using the CLIP text encoder and combined with dynamic weights calculated by the degradation probability predictor, so that information from different degradation levels can be reasonably contributed to the final text prompt generation, thereby optimizing the performance of Stable Diffusion in the repair task.

[0087] Figure 4 This is a schematic diagram of the process for determining prior text prompts provided by the present invention. Figure 5 This is a schematic diagram of the state-space feature extraction and transfer module based on the state-space model provided by the present invention, as shown below. Figure 4 and Figure 5 As shown, the cross-modal mapping network is a Mamba-based cross-modal mapping network, which can use a state-space model as the basis for its network architecture. The specific process can be represented as follows:

[0088] Mamba(·)=Lin((σ(Lin(·)))×SSM(Lin(·),σ(Conv(Lin(·))))

[0089]

[0090] Where Lin(·) represents a linear transformation, Conv(·) represents a convolution operation, σ(·) is the Sigmoid activation function, and SSM(·) represents the state-space modeling of the Mamba structure, used for feature mapping at different degradation levels. This represents cross-modal text representation. This represents the features of the first face image.

[0091] The text suggestion parameters are learnable parameters, which are taken from a preset text suggestion pool, and are represented as follows: Where N is the number of categories representing the severity of degradation, L is the number of text tokens, and C is the embedding dimension of the CLIP text encoder. By connecting the cross-modal representation with the text cue parameters, it can adapt to different degradation scenarios.

[0092] Here, the dynamic weights (degradation probability weights) are calculated by the degradation probability predictor and used to adjust the contribution ratio of different degradation levels, making the final generated text prompts more accurate, as shown below:

[0093]

[0094] Where, p i E represents the degradation probability weights generated by the degradation probability predictor. txt Represents CLIP text encoder, e txt This indicates prior information in the text prompt.

[0095] Understandably, the text prompt generation function extracts visual information from low-quality images and combines it with cross-modal mapping and learnable text embeddings to construct high-quality text descriptions. This function not only adapts to different degrees of image degradation but also utilizes dynamic weights provided by a degradation probability predictor to guide the text encoding process, making the generated prompts more consistent with the true features of the image. Ultimately, the optimized text prompt prior effectively enhances StableDiffusion's text-to-image generation capabilities, improving the accuracy and realism of the restoration results.

[0096] Here, the state space feature extraction and transfer module based on Mamba is implemented based on a state space network. Combined with the extension in the vision domain, it can realize the transfer from CLIP visual features to multimodal features, and assist the model in realizing feature extraction from visual modality to text-visual alignment modality.

[0097] Structured state-space sequence models transform one-dimensional functions or sequences x(t)∈R into continuous systems. L Mapped to output y(t)∈R L And introduce an intermediate hidden state h(t)∈R N This process uses ordinary differential equations to represent the input data, where the changes in the hidden states are described by the following formula:

[0098] h'(t)=Ah(t)+Bx(t), y(t)=Ch(t),

[0099] Where, matrix A∈R N×N Let B represent the evolution matrix of the system, where B ∈ R. N×L and C∈R L×NIt is the projection matrix. To transition from a continuous model to a discretized model, the Mamba framework utilizes the time scale parameter Δ to transform matrices A and B into discrete equivalent forms using the zero-order preservation method. and The discrete update equation is obtained:

[0100]

[0101] And ultimately expressed as:

[0102]

[0103] To enable traditional structured state-space models to adapt to face image restoration tasks with varying degrees of degradation, this module uses different Mamba modules based on state-space networks to align the input image features from visual to multimodal features for different degrees of degradation. This ensures that the model can acquire fine-grained multimodal face features on images with varying degrees of degradation, thereby generating degradation-aware textual prior prompts in subsequent steps.

[0104] Based on the above embodiments, the step of determining the visual cues of the second face image features of the image to be repaired includes:

[0105] Step 310: Project the image to be repaired into the latent embedding space to obtain the latent feature representation corresponding to the image to be repaired;

[0106] Step 320: Perform multi-scale decomposition on the latent feature representation to obtain image spatial latent features at different scales;

[0107] Step 330: Based on the cross-modal mapping network in the visual cueing module, perform cross-modal mapping on the image spatial latent features at different levels to obtain visual cross-modal representations at each level;

[0108] Step 340: Determine the visual cue based on the visual cross-modal representations of each level.

[0109] Specifically, in order to extract the latent features of the image to be repaired more precisely, a VAE encoder can be used to project the image to be repaired into a latent embedding space to obtain the latent feature representation corresponding to the image to be repaired. Where, x LQ This represents the input image to be repaired. It represents the corresponding latent features.

[0110] Subsequently, the latent features were represented. Multi-scale decomposition is performed to extract information from different levels of the image, resulting in... This represents the latent spatial features of the image at the k-th scale.

[0111] It should be noted that lower-scale features have a larger receptive field, which is used to preserve global structural information, while higher-scale features capture finer local details.

[0112] Furthermore, based on the cross-modal mapping network in the visual cueing module, cross-modal mapping is performed on the latent features of image space at different scales to obtain visual cross-modal representations at each level.

[0113] Similar to the feature extraction and transfer process in the text prompt generation section, by inputting the latent features of image space at various scales into a Mamba-based cross-modal mapping network for transformation, a more suitable cross-modal embedding for subsequent visual prompt generation can be obtained. That is, the visual cross-modal representation of each level is as follows:

[0114]

[0115] Here, Conv(·) represents a convolutional layer, RB(·) represents a residual block, and Mamba(·) uses the same state-space network structure as the text prompt word generation network to model feature changes under different degradation levels. Indicates the features of the second face image. This represents the visual cross-modal representation at each level.

[0116] Understandably, by decomposing the original input image to be repaired into spatial scales, the image inpainting model can better improve its cognitive performance in capturing the global structural information and fine local details of the image to be repaired under different receptive field conditions. This can optimize the expression of the subsequently generated visual cues, making them more consistent with the features of the original image.

[0117] Finally, visual cues are determined based on the cross-modal visual representations at each level.

[0118] It is understandable that the cross-modal feature generation process based on state-space networks, by introducing the concept of multi-scale feature extraction, performs hierarchical processing on low-quality features, further enhancing the detailed representation ability of cross-modal features, thereby improving the model's adaptability to different degrees of degradation.

[0119] Based on the above embodiments, step 340 includes:

[0120] Step 341: Extract a set of visual cue parameters from a preset visual cue pool; the visual cue parameters include learnable parameters at different degradation levels;

[0121] Step 342: Determine the degree of matching between the visual cross-modal representations of each level and the visual cue parameters through a cross-attention network, and determine the fused cue features based on the degree of matching;

[0122] Step 343: The fused cue features are spliced ​​with the latent features in the image space to obtain the visual cue.

[0123] Specifically, visual cross-modal representations can be input into the spatial cue word generation and learning module. This module extracts degraded perceptual information from latent features at different scales and integrates it into a spatial cue embedding suitable for Stable Diffusion. This module adopts a structure similar to the text cue word generation network and introduces a multi-head cross-attention mechanism to improve the adaptability of spatial cues. Its processing flow includes multi-scale feature extraction, visual cue pool retrieval, spatial cue concatenation, and dynamic weight adjustment, as detailed below:

[0124] A set of visual cue parameters is extracted from a pre-defined visual cue pool. These visual cue parameters include learnable parameters at different degradation levels to provide spatial cue embedding at different scales.

[0125] Then, a multi-head cross-attention mechanism is used to determine the degree of matching between the visual cross-modal representations at each level and the visual cue parameters, and based on the degree of matching, the fused cue features are determined.

[0126] Finally, the fused cue features are stitched together with the latent features of the image space to obtain visual cues, forming complete spatial cue information that can more effectively guide the reconstruction task of Stable Diffusion.

[0127] Furthermore, the degradation probability weights calculated by the degradation probability predictor can be used, and the information contribution ratios of different degradation levels can be adjusted to ensure the expressive power of spatial cues.

[0128] The acquisition of visual cues and text cues adopts a similar construction method, and for each spatial size k, a construction is performed. Where N is the number of degradation severity categories, M is the capacity hyperparameter of visual cues, and C k This represents the channel dimension under scale k.

[0129] Here, the multi-head cross-attention mechanism is used to calculate the degree of matching between the visual cross-modal representations at each level and the visual cue parameters, and to generate the final spatial cue embedding (fusion of cue features), specifically represented as follows:

[0130]

[0131] Among them, MHCAk (q, k, v) represents the k-th level multi-head cross-attention layer, p i The dynamic weights are calculated by the degradation probability predictor. e represents the visual cross-modal representation at each level vis Indicates a visual cue.

[0132] Understandably, the spatial cue word generation function achieves accurate modeling of degradation information in the image to be restored by performing feature transformation and dynamic fusion on cross-modal visual representations at various levels, combined with a learnable visual cue pool. This method can adapt to different degradation conditions and optimize visual cue information using a degradation probability predictor, enabling it to better guide Stable Diffusion to generate high-quality restoration results. Ultimately, the optimized visual cue effectively improves the model's adaptability to different degraded images, enhancing the visual consistency and detail fidelity of the generated results.

[0133] Based on the above embodiments, the training steps of the image restoration model include:

[0134] Step 410: Obtain an initial image restoration model, a sample image to be restored, and a label image of the sample image to be restored; the sample image to be restored is obtained by performing degradation operations on the label image, including Gaussian blur, downsampling, noise addition, and JPEG compression.

[0135] Step 420: Extract the sample face image features of the sample image to be repaired, and input the sample face image features into the initial image repair model to obtain the predicted repaired image output by the initial image repair model;

[0136] Step 430: Based on the difference between the predicted image to be restored and the labeled image, determine the target loss, and perform parameter iteration on the initial image restoration model based on the target loss to obtain the image restoration model.

[0137] Specifically, firstly, an initial image restoration model, a sample image to be restored, and a label image of the sample image to be restored are obtained. Here, the sample image to be restored is obtained by performing a degradation operation on the label image. The degradation operation includes at least one of Gaussian blur, downsampling, noise addition, and JPEG compression.

[0138] It is understandable that the labeled image is a high-resolution face image, while the sample image to be repaired is a low-resolution face image corresponding to the high-resolution face image.

[0139] The sample images to be restored can be obtained from datasets with different degrees of degradation. It integrates multiple datasets such as CelebA-HQ, FFHQ, WebFace, and LFW, and combines data augmentation techniques to cover different degrees of noise, blur, low light, resolution degradation, and compression distortion, ensuring that the model comprehensively learns the features of low-quality face images with different degrees of degradation and improves the generalization performance of restoration.

[0140] Then, the sample face image features of the image to be restored are extracted, and these features are input into the initial image restoration model to obtain the predicted restored image output by the initial image restoration model. Here, the extraction of sample face image features, i.e., low-quality face image feature extraction, is achieved by using a CLIP encoder and VAE to encode the low-quality face image, generating a high-dimensional feature representation that contains the core visual information of the face image, providing effective support for degradation prediction.

[0141] The degradation prediction network analyzes the features of the input sample face image through multi-layer processing and optimization with cross-entropy loss, and accurately identifies common degradation levels such as mild noise, severe blur and extremely low light.

[0142] Finally, after obtaining the predicted restored image based on the image restoration model, the target loss can be determined based on the difference between the predicted restored image and the label image, and the parameters of the initial image restoration model can be iterated based on the target loss. The initial image restoration model after parameter iteration is then used as the image restoration model.

[0143] Understandably, the greater the difference between the predicted restored image and the labeled image, the greater the target loss; conversely, the smaller the difference between the predicted restored image and the labeled image, the smaller the target loss.

[0144] Based on any of the above embodiments, the degradation probability fusion adopts a dynamic weight allocation method, which can more accurately control the contribution of prompt words with different degradation levels and significantly reduce the interference effect between tasks with different degradation levels.

[0145] A joint optimization strategy is introduced during the text prompt generation process. By co-training with visual prompts, the text prompts' ability to perceive degradation features is further improved, thereby enhancing the robustness of the generated prompts in guiding face image restoration.

[0146] In this embodiment of the invention, a prior learning method for visual branching cue words based on multi-scale feature extraction and transfer of state space modules is proposed, similar to a text branching model. The method includes multi-scale feature generation based on state space networks, construction of a visual cue word library, degradation probability fusion, and visual cue word generation.

[0147] The multi-scale feature generation based on state-space networks utilizes a pre-trained variational autoencoder (VAE) to map low-quality face images to a latent space representation, decomposes it into latent feature embeddings at multiple scales, processes the features at each scale through a state-space network model, and extracts cross-modal features by combining residual blocks and convolutional layers, ensuring that the features at each scale can retain global and local details of degradation information.

[0148] Visual cue word library construction: It consists of multiple sets of learnable parameters related to the degree of degradation. Each set of cue words corresponds to a specific degree of degradation. These cue words are fused with multi-scale cross-modal feature embeddings through a multi-head cross-attention mechanism during the cue generation process to provide targeted guidance for face restoration of different degrees of degradation.

[0149] Based on the degradation probability generated by the degradation predictor, the weights of different prompt words are dynamically adjusted during the visual prompt generation process, which significantly reduces the interference between different degradation levels during multi-scale feature fusion and ensures that the prompt information accurately reflects the main degradation features of the face image.

[0150] Visual cue word generation generates visual cues through cross-attention processing of multi-scale feature embedding and cue word library. The cue word results at each scale are finally fused with dynamic weights through residual processing to form a unified multi-scale visual cue embedding, which serves as prior information for a stable diffusion model to guide face image restoration with different degrees of degradation.

[0151] In some preferred embodiments, the multi-scale feature generation process based on state-space networks, combined with a multi-resolution feature decomposition strategy, can further enhance the network's ability to express facial image features with different degrees of degradation.

[0152] In some preferred embodiments, the degradation probability fusion employs an attention enhancement module, which enables fine-grained optimization of cue word weight adjustment for different degrees of degradation, reducing performance interference.

[0153] The blind face image restoration device provided by the present invention is described below. The blind face image restoration device described below can be referred to in correspondence with the blind face image restoration method described above.

[0154] Based on any of the above embodiments, the present invention provides a device for restoring facial images of blind people. Figure 6 This is a schematic diagram of the structure of the blind face image restoration device provided by the present invention, as shown below. Figure 6 As shown, the device includes:

[0155] The acquisition unit 610 is used to acquire the image of the blind face to be repaired;

[0156] The input unit 620 is used to input the image to be repaired into the image repair model to obtain the target repaired image output by the image repair model;

[0157] The image restoration model includes a degradation prediction network, a text prompting module, a visual prompting module, and a decoding module. The degradation prediction network is used to determine degradation probability weights based on the first face image features of the image to be restored. The text prompting module is used to determine text prompts based on the first face image features and to determine text prompt priors based on the text prompts and the degradation probability weights. The visual prompting module is used to determine visual prompts based on the second face image features of the image to be restored and to determine visual prompt priors based on the visual prompts and the degradation probability weights. The decoding module is used to determine multimodal prompts based on the text prompt priors and the visual prompt priors, and to perform image restoration on the image to be restored based on the multimodal prompts to obtain the target restored image.

[0158] The apparatus provided in this invention acquires an image of a blind face to be repaired; inputs the image to be repaired into an image repair model to obtain a target repaired image output by the image repair model; wherein, the image repair model includes a degradation prediction network, a text prompt module, a visual prompt module, and a decoding module; the degradation prediction network is used to determine degradation probability weights based on the first face image features of the image to be repaired; the text prompt module is used to determine text prompts based on the first face image features, and to determine text prompt priors based on the text prompts and degradation probability weights; the visual prompt module is used to determine visual prompts based on the second face image features of the image to be repaired, and to determine visual prompt priors based on the visual prompts and degradation probability weights; the decoding module is used to determine multimodal prompts based on the text prompt priors and visual prompt priors, and to perform image repair on the image to be repaired based on the multimodal prompts to obtain the target repaired image. In this method, the degradation prediction network can effectively evaluate the degree of degradation of the input image to be restored and provide corresponding degradation probability weights to enhance the prompt generation ability of subsequent tasks. Furthermore, the introduction of the degradation prediction network improves the adaptability of the image restoration model, enabling it to adjust the prompt strategy according to the degree of degradation, thereby optimizing the final image restoration effect of blind faces and improving the fidelity of image restoration.

[0159] Based on any of the above embodiments, a first determining unit is further included, the first determining unit being specifically used for:

[0160] Based on the cross-modal mapping network in the text prompt module, the features of the first face image are cross-modal mapped to obtain a cross-modal representation of the text; the cross-modal mapping network is constructed based on a state-space model;

[0161] A set of text prompt parameters is extracted from a preset text prompt pool, and the cross-modal representation of the text is concatenated with its corresponding text prompt parameters to obtain the text prompt.

[0162] Based on any of the above embodiments, a second determining unit is further included, the second determining unit specifically including:

[0163] A latent feature representation unit is determined to project the image to be repaired into a latent embedding space to obtain the latent feature representation corresponding to the image to be repaired;

[0164] A multi-scale decomposition unit is used to perform multi-scale decomposition on the latent feature representation to obtain image spatial latent features at different levels of scale.

[0165] The cross-modal mapping unit is used to perform cross-modal mapping on the image spatial latent features at different levels and scales based on the cross-modal mapping network in the visual cueing module, so as to obtain the visual cross-modal representation at each level;

[0166] A visual cue unit is defined for determining the visual cue based on the visual cross-modal representation of each level.

[0167] Based on any of the above embodiments, the visual cue determination unit is specifically used for:

[0168] A set of visual cue parameters is extracted from a preset visual cue pool; the visual cue parameters include learnable parameters at different degradation levels;

[0169] The degree of matching between the visual cross-modal representations of each level and the visual cue parameters is determined by a cross-attention network, and the fused cue features are determined based on the degree of matching.

[0170] The visual cue is obtained by concatenating the fused cue features with the latent features of the image space.

[0171] Based on any of the above embodiments, a training unit is further included, wherein the training unit is specifically used for:

[0172] An initial image restoration model, a sample image to be restored, and a label image of the sample image to be restored are obtained; the sample image to be restored is obtained by performing degradation operations on the label image, including Gaussian blur, downsampling, noise addition, and JPEG compression.

[0173] Extract the sample face image features of the sample image to be repaired, and input the sample face image features into the initial image repair model to obtain the predicted repaired image output by the initial image repair model;

[0174] Based on the difference between the predicted restored image and the labeled image, a target loss is determined, and the initial image restoration model is iterated based on the target loss to obtain the image restoration model.

[0175] Based on any of the above embodiments, the first face image feature is obtained by inputting the image to be repaired into the image encoder in the contrastive language image pre-training model;

[0176] The second facial image feature is obtained by inputting the image to be repaired into a variational autoencoder.

[0177] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logic instructions in the memory 730 to execute a method for restoring a blind face image. The method includes: acquiring an image of a blind face to be restored; inputting the image to be restored into an image restoration model to obtain a target restored image output by the image restoration model; the image restoration model includes a degradation prediction network, a text prompt module, a visual prompt module, and a decoding module; the degradation prediction network is used to determine degradation probability weights based on first face image features of the image to be restored; the text prompt module is used to determine text prompts based on the first face image features and to determine text prompt priors based on the text prompts and the degradation probability weights; the visual prompt module is used to determine visual prompts based on second face image features of the image to be restored and to determine visual prompt priors based on the visual prompts and the degradation probability weights; the decoding module is used to determine multimodal prompts based on the text prompt priors and the visual prompt priors, and to perform image restoration on the image to be restored based on the multimodal prompts to obtain the target restored image.

[0178] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0179] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the blind face image restoration method provided by the above methods. The method includes: acquiring a blind face image to be restored; inputting the image to be restored into an image restoration model to obtain a target restored image output by the image restoration model; the image restoration model includes a degradation prediction network, a text prompt module, a visual prompt module, and a decoding module; the degradation prediction network is used to determine degradation probability weights based on a first face image feature of the image to be restored; the text prompt module is used to determine text prompts based on the first face image feature and to determine a text prompt prior based on the text prompt and the degradation probability weight; the visual prompt module is used to determine visual prompts based on a second face image feature of the image to be restored and to determine a visual prompt prior based on the visual prompt and the degradation probability weight; the decoding module is used to determine multimodal prompts based on the text prompt prior and the visual prompt prior, and to perform image restoration on the image to be restored based on the multimodal prompts to obtain the target restored image.

[0180] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the blind face image restoration method provided by the above methods. The method includes: acquiring a blind face image to be restored; inputting the image to be restored into an image restoration model to obtain a target restored image output by the image restoration model; the image restoration model includes a degradation prediction network, a text prompt module, a visual prompt module, and a decoding module; the degradation prediction network is used to determine degradation probability weights based on first face image features of the image to be restored; the text prompt module is used to determine text prompts based on the first face image features and to determine text prompt priors based on the text prompts and the degradation probability weights; the visual prompt module is used to determine visual prompts based on second face image features of the image to be restored and to determine visual prompt priors based on the visual prompts and the degradation probability weights; the decoding module is used to determine multimodal prompts based on the text prompt priors and the visual prompt priors, and to perform image restoration on the image to be restored based on the multimodal prompts to obtain the target restored image.

[0181] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0182] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for restoring facial images of blind people, characterized in that, include: Obtain the image of the blind person's face to be repaired; The image to be repaired is input into the image repair model to obtain the target repaired image output by the image repair model; The image restoration model includes a degradation prediction network, a text prompting module, a visual prompting module, and a decoding module; the degradation prediction network is used to determine degradation probability weights based on the first face image features of the image to be restored; The text prompt module is used to determine text prompts for the features of the first face image, and to determine text prompt priors based on the text prompts and the degradation probability weights; the visual prompt module is used to determine visual prompts for the features of the second face image of the image to be repaired, and to determine visual prompt priors based on the visual prompts and the degradation probability weights. The decoding module is used to determine multimodal cues based on the textual and visual cues priors, and to perform image restoration on the image to be restored based on the multimodal cues, thereby obtaining the target restored image.

2. The method for restoring blind face images according to claim 1, characterized in that, The steps for determining the textual prompts based on the features of the first facial image include: Based on the cross-modal mapping network in the text prompt module, the features of the first face image are cross-modal mapped to obtain a cross-modal representation of the text; the cross-modal mapping network is constructed based on a state-space model; A set of text prompt parameters is extracted from a preset text prompt pool, and the cross-modal representation of the text is concatenated with its corresponding text prompt parameters to obtain the text prompt.

3. The method for restoring blind face images according to claim 1, characterized in that, The steps for determining the visual cues of the second facial image features of the image to be repaired include: The image to be repaired is projected into the latent embedding space to obtain the latent feature representation corresponding to the image to be repaired; The latent feature representation is decomposed into image spatial latent features at different scales. Based on the cross-modal mapping network in the visual cueing module, cross-modal mapping is performed on the latent features of the image space at different scales to obtain the visual cross-modal representations at each level; The visual cues are determined based on the visual cross-modal representations of each level.

4. The method for restoring blind face images according to claim 3, characterized in that, The determination of the visual cues based on the cross-modal visual representations of each level includes: A set of visual cue parameters is extracted from a preset visual cue pool; the visual cue parameters include learnable parameters at different degradation levels; The degree of matching between the visual cross-modal representations of each level and the visual cue parameters is determined by a cross-attention network, and the fused cue features are determined based on the degree of matching. The visual cue is obtained by concatenating the fused cue features with the latent features of the image space.

5. The method for restoring blind face images according to any one of claims 1 to 4, characterized in that, The training steps of the image restoration model include: An initial image restoration model, a sample image to be restored, and a label image of the sample image to be restored are obtained; the sample image to be restored is obtained by performing degradation operations on the label image, including Gaussian blur, downsampling, noise addition, and JPEG compression. Extract the sample face image features of the sample image to be repaired, and input the sample face image features into the initial image repair model to obtain the predicted repaired image output by the initial image repair model; Based on the difference between the predicted restored image and the labeled image, a target loss is determined, and the initial image restoration model is iterated based on the target loss to obtain the image restoration model.

6. The method for restoring blind face images according to any one of claims 1 to 4, characterized in that, The first facial image feature is obtained by inputting the image to be repaired into the image encoder in the contrastive language image pre-training model; The second facial image feature is obtained by inputting the image to be repaired into a variational autoencoder.

7. A device for restoring facial images of blind people, characterized in that, include: The acquisition unit is used to acquire the image of the blind face to be restored; An input unit is used to input the image to be repaired into the image repair model to obtain the target repaired image output by the image repair model; The image restoration model includes a degradation prediction network, a text prompting module, a visual prompting module, and a decoding module; the degradation prediction network is used to determine degradation probability weights based on the first face image features of the image to be restored; The text prompt module is used to determine text prompts for the features of the first face image, and to determine text prompt priors based on the text prompts and the degradation probability weights; the visual prompt module is used to determine visual prompts for the features of the second face image of the image to be repaired, and to determine visual prompt priors based on the visual prompts and the degradation probability weights. The decoding module is used to determine multimodal cues based on the textual and visual cues priors, and to perform image restoration on the image to be restored based on the multimodal cues, thereby obtaining the target restored image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the blind face image restoration method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the blind face image restoration method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the blind face image restoration method as described in any one of claims 1 to 6.

Images