Face blindness repair method, device, equipment, and storage medium
Through the forward multi-step noise addition and reverse multi-step noise removal processing method, combined with low-pass filter and exchange low-frequency components, the problem of low-quality facial image repair in the existing technology is solved, and high-quality Face blind repair and reduce calculation consumption.
Patent Information
- Application Number
- CN202411204104.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-30
AI Technical Summary
The prior art is difficult to effectively repair low-quality face images, especially to improve the visual clarity and detail quality of the image while maintaining the facial properties.
A facial blind repair method is adopted, and image quality is gradually improved through forward multi-step noise addition and reverse multi-step noise removal, combined with low-pass filter and exchange low-frequency components. The method includes acquiring a low-resolution image, performing multi-step noise addition processing, using a low-pass filter for denoising processing, and adjusting the processing steps according to preset conditions and mean square error.
It significantly improves the repair quality of face images, maintains the accuracy of face attributes, and reduces the time consumption of the repair process.
Smart Images

Figure CN118710558B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment, and storage medium for blind face repair. Background Art
[0002] With the development of computer vision technology and the Internet, images have become an indispensable information carrier in people's social life. However, in the process of image acquisition, storage, and transmission, there are various environmental factors that cause various image degradations such as noise pollution, blur effects, compression artifacts, etc., which make it difficult to use images. Image restoration technology is a technology that repairs image degradation to improve image quality.
[0003] Blind face restoration is an important branch of image restoration. Face images are widely used in modern society. They play a key role in personal identity authentication, public security monitoring, social networking platforms and other fields. However, due to technical or environmental limitations, poor image quality is often encountered. Blind face restoration (BFR) technology aims to solve this problem and improve the practicality and reliability of low-quality face images. The ultimate goal of blind face restoration technology is not only to improve the visual clarity and detail quality of the image, but also to accurately maintain the original facial attributes during the restoration process without pre-identifying the specific causes of degradation. Summary of the invention
[0004] In order to solve one of the above-mentioned technical defects, the present application provides a face blindness repair method, device, equipment, and storage medium.
[0005] In a first aspect, the present application provides a method for repairing face blindness, the method comprising:
[0006] Get the image to be repaired ; Among them, the image to be repaired For low-resolution images;
[0007] Treat the repaired image , starting from step 1, perform forward multi-step noise processing to obtain the Step Image ;in, is the target number of steps for adding noise to the image to be repaired;
[0008] Based on low pass filter ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image ;in, is the reverse denoising step number identifier, , are the parameters of the low-pass filter, The initial value of is determined according to the degree of preservation of face identity information. For The denoising target number of steps, ;
[0009] according to Predict the first image ;
[0010] right , starting from step 1, perform forward multi-step noise processing to obtain the Step Image ;in, For The number of noise-added target steps;
[0011] Based on low pass filter ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image ;in, is the reverse denoising step number identifier, , For The denoising target number of steps, ;
[0012] If the current The preset conditions are not met and the current If the mean square error between the low-frequency component of the image to be repaired is greater than the preset threshold, the image is enlarged. , repeatedly performing a low-pass filter based ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image the steps and subsequent steps;
[0013] If the current Satisfy the preset conditions, or the current If the mean square error between the low-frequency component of the image to be repaired and the low-frequency component of the image to be repaired is not greater than the preset threshold, then the low-pass filter is used to ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image ;according to Predict the second image ,based on Get the restored image; where, is the reverse denoising step number identifier, , For The denoising target number of steps, .
[0014] Optionally, in each step of the noise addition process to be performed on the repaired image, Get the image of this step;
[0015] in, is the number of noise adding steps for the repaired image. , For the Step image, is the noise drawn from a standard Gaussian distribution, For the The noise processing parameters of the step, , is the step number indicator, For the The noise factor of the step, , For the Diffusion speed when adding noise.
[0016] Optionally, From Starting from the step, after each step of the reverse denoising process, Exchange the low-frequency components of the denoised image with The low-frequency component of
[0017] in, For the After the denoising process, for The low-frequency component of for The low-frequency component of For the Step image;
[0018] right From Starting from the step, after each step of the reverse denoising process, Exchange the low-frequency components of the denoised image with The low-frequency component of
[0019] in, For the After the denoising process, for The low-frequency component of for The low-frequency component of For the Step image;
[0020] right From Starting from the step, after each step of the reverse denoising process, Exchange the low-frequency components of the denoised image with The low-frequency component of
[0021] in, For the After the denoising process, for The low-frequency component of for The low-frequency component of For the Step image.
[0022] Optionally, according to Predict the first image ;
[0023] in, For the The noise processing parameters of the step, , is the step number indicator, For the The diffusion speed when adding noise, is the noise drawn from a standard Gaussian distribution;
[0024] according to Predict the second image ;
[0025] in, For the The noise processing parameters of the step, , is the step number indicator, No. Diffusion speed when adding noise.
[0026] Optionally, based on Get the restored image, including:
[0027] Will Compressed into latent space by encoder Then, starting from step 1, forward multi-step noise addition is performed to obtain the Step Image ;in, To compress into the latent space The target number of steps for adding noise to the image after denoising;
[0028] right From Step 1 is started, and the reverse multi-step denoising process is performed to obtain the latent variable ;
[0029] based on The restored image is obtained through the decoder.
[0030] Optionally, obtaining an image to be repaired includes:
[0031] Obtain an input image; the input image is a low-resolution image;
[0032] Convert the input image tensor from [H, W, C] to [C, H, W], where H is the height, W is the width, and C is the number of channels;
[0033] Normalize the values of the converted tensor in each dimension to [0, 1];
[0034] Normalize the values of the normalized tensor in each dimension to a mean of 0 and a variance of 1;
[0035] The standardized image is used as the image to be repaired.
[0036] Optionally, , , The initial value of is 2. , , ;
[0037] expand ,include: ;
[0038] The default condition is , the preset threshold is .
[0039] In a third aspect of the present application, a face blindness repair device is provided, the device comprising:
[0040] Acquisition module, used to obtain the image to be repaired ; Among them, the image to be repaired For low-resolution images;
[0041] The first noise adding module is used to repair the image , starting from step 1, perform forward multi-step noise processing to obtain the Step Image ;in, is the target number of steps for adding noise to the image to be repaired;
[0042] The first denoising module is used based on a low-pass filter ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image ;in, is the reverse denoising step number identifier, , are the parameters of the low-pass filter, The initial value of is determined according to the degree of preservation of face identity information. For The denoising target number of steps, ;
[0043] The first prediction module is used to Predict the first image ;
[0044] The second noise adding module is used to , starting from step 1, perform forward multi-step noise processing to obtain the Step Image ;in, For The number of noise-added target steps;
[0045] The second denoising module is used based on a low-pass filter ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image ;in, is the reverse denoising step number identifier, , For The denoising target number of steps, ;
[0046] Loop module, used if the current The preset conditions are not met and the current If the mean square error between the low-frequency component of the image to be repaired is greater than the preset threshold, the image is enlarged. , repeat the second denoising module and the loop module; if the current Satisfy the preset conditions, or the current If the mean square error between the low-frequency component of the image to be repaired is not greater than a preset threshold, the processing module is executed;
[0047] Processing module for low-pass filter based ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image ;according to Predict the second image ,based on Get the restored image; where, is the reverse denoising step number identifier, , For The denoising target number of steps, .
[0048] In a third aspect of the present application, an electronic device is provided, including:
[0049] Memory;
[0050] Processor; and
[0051] Computer programs;
[0052] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect above.
[0053] In a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored; the computer program is executed by a processor to implement the method described in the first aspect above.
[0054] The present application provides a method, device, equipment, and storage medium for blind face restoration. The method obtains an image to be restored. ; Treat the repaired image Perform forward multi-step noise processing to obtain the Step Image ; Based on low-pass filter Perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image ;according to Predict the first image ;right Perform forward multi-step noise processing to obtain the Step Image ; Based on low-pass filter Perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image According to the current Relationship to preconditions, and The restored image is obtained by calculating the mean square error between the low frequency component of the image to be restored and the image to be restored. The method of the present application can improve the quality of face image restoration and reduce the time consumption of the restoration process. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0056] Figure 1 A flowchart of a method for blind face repair provided in an embodiment of the present application;
[0057] Figure 2 is a schematic diagram of a denoising architecture provided in an embodiment of the present application;
[0058] Figure 3 is a SNR curve diagram of different resolution models provided in the embodiments of the present application;
[0059] Figure 4 A schematic diagram of the structure of a face blindness repair device provided in an embodiment of the present application;
[0060] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0061] In order to make the technical solutions and advantages in the embodiments of the present application more clearly understood, the exemplary embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.
[0062] This embodiment provides a method for repairing face blindness. Figure 1 , the implementation process of this method is as follows:
[0063] 101. Obtain the image to be repaired .
[0064] The image to be repaired For low-resolution images.
[0065] This step can be implemented in the following steps in specific applications:
[0066] 101-1, obtain an input image.
[0067] Among them, the input image is a low-resolution image.
[0068] 101-2, convert the input image tensor tensor from [H, W, C] to [C, H, W].
[0069] Among them, H is the height, W is the width, and C is the number of channels.
[0070] 101-3, normalize the values of the converted tensor in each dimension to [0, 1].
[0071] 101-4, normalize the values of the normalized tensor in each dimension to a mean of 0 and a variance of 1.
[0072] 101-5, the standardized image is used as the image to be repaired.
[0073] 102. Treat the repaired image , starting from step 1, perform forward multi-step noise processing to obtain the Step Image .
[0074] in, is the target number of steps for adding noise to the image to be repaired, for example .
[0075] In addition, in each step of the noise addition process for the repaired image, The image after noise processing in this step is obtained.
[0076] in, is the number of noise adding steps for the repaired image. , For the Step image, is the noise drawn from a standard Gaussian distribution, For the The noise processing parameters of the step, , is the step number indicator, For the The noise factor of the step, , For the Diffusion speed when adding noise.
[0077] like , then in step 102, the repaired image will be denoised for 100 steps, and the image after the 100th step of denoising That is Step Image .
[0078] 103, based on low-pass filter ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image .
[0079] in, are the parameters of the low-pass filter, The initial value of is determined according to the degree of preservation of face identity information, such as The initial value of is 2. is the reverse denoising step number identifier, , are the parameters of the low-pass filter, The initial value of is determined according to the degree of preservation of face identity information. For The denoising target number of steps, .
[0080] In addition, From Starting from the step, after each step of the reverse denoising process, Exchange the low-frequency components of the denoised image with low frequency components.
[0081] in, For the After the denoising process, for The low-frequency component of for The low-frequency component of For the Step image.
[0082] The low-pass filter can be used to filter the input A filtering operation that downsamples by a factor of 1 and then upsamples.
[0083] like , , then in step 102, it will start from step 1 and perform 100 steps of noise addition forward to obtain the image after the 100th step of noise addition. (Right now ). In step 103, starting from step 100, multiple steps of denoising and swapping are performed backward to obtain an image denoised and swapped to step 50. (Right now ).
[0084] 104, according to Predict the first image .
[0085] Among them, the first image is the predicted 0-step output image, .
[0086] in, For the The noise processing parameters of the step, , is the step number indicator, For the The diffusion speed when adding noise, is the noise drawn from a standard Gaussian distribution.
[0087] By executing steps 102 to 104, the image to be repaired is subjected to the first denoising and denoising process, wherein the number of denoising steps is greater than the number of denoising steps, and the output image of step 0 is predicted based on the denoised image. .
[0088] 105, yes , starting from step 1, perform forward multi-step noise processing to obtain the Step Image .
[0089] in, For The target number of noise steps is .
[0090] The denoising process in this step is similar to the denoising process in step 102, except that the denoising object and the number of denoising steps when the denoising is finally stopped are different. The denoising object in step 102 is the image to be repaired obtained in step 101. The noise object in this step is the one predicted in step 104. Step 102 is to add noise Stop after step 1, and finally get Step Image , step 105 adds noise Stop after step 1, and finally get Step Image .
[0091] In addition, in In each step of the noise addition process, Get the image of this step.
[0092] in, For The number of noise adding steps is marked, , For the Step image, For the The noise processing parameters of the step, , is the step number indicator, For the The noise factor of the step, , For the Diffusion speed when adding noise.
[0093] It should be noted that The value can be The same value as The values of are different, and this embodiment and subsequent embodiments do not limit The value of The relationship between the values.
[0094] like 250, then in step 105 Add noise for 250 steps to get the image after adding noise at the 250th step .
[0095] 106, based on low-pass filter ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image .
[0096] in, is the reverse denoising step number identifier, , For The denoising target number of steps, .
[0097] The denoising process in this step is similar to the denoising process in step 103, except that the denoising object and the number of denoising steps when the denoising is finally stopped are different. The denoising object in step 103 is the denoising object obtained in step 102. The denoising object in this step is the one obtained in step 105. Step 103: After adding noise in the first step, the image is denoised and exchanged in multiple steps. After the denoising and exchange are completed, stop and get the Step Image , step 106 from Step 1, after multiple steps of denoising, in step After the denoising and exchange are completed, stop and get the Step Image .
[0098] In addition, From Starting from the step, after each step of the reverse denoising process, Exchange the low-frequency components of the denoised image with low frequency components.
[0099] in, For the After the denoising process, for The low-frequency component of for The low-frequency component of For the Step image.
[0100] It should be noted that The value can be The same value as The values of are different, and this embodiment and subsequent embodiments do not limit The value of The relationship between the values.
[0101] like , , this step will Denoising from step 250 to step 150, we get .
[0102] The second denoising process is completed through step 105 , and the first stage of the second denoising process is performed through step 106 .
[0103] 107, according to the current Relationship to preconditions, and The repaired image is obtained by calculating the mean square error between the low-frequency component of the image to be repaired and the image to be repaired.
[0104] If the current The preset conditions are not met and the current If the mean square error between the low-frequency component of the image to be repaired is greater than the preset threshold, the image is enlarged. , repeatedly performing a low-pass filter based ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image and subsequent steps (i.e., repeatedly executing steps 106 and 107).
[0105] Among them, the expansion Can be .
[0106] like The initial value of is 2, so the preset condition can be , the preset threshold is .
[0107] That is, if the current The preset conditions are not met and the current If the mean square error between the low-frequency components of the image to be repaired is greater than a preset threshold, the first stage of the second denoising process will be repeated.
[0108] If the current Satisfy the preset conditions, or the current If the mean square error between the low-frequency component of the image to be repaired and the image to be repaired is not greater than the preset threshold, then:
[0109] 1. Based on low-pass filter ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image .
[0110] in, is the reverse denoising step number identifier, , For The denoising target number of steps, .
[0111] In addition, From Starting from the step, after each step of the reverse denoising process, Exchange the low-frequency components of the denoised image with low frequency components.
[0112] in, For the After the denoising process, for The low-frequency component of for The low-frequency component of For the Step image.
[0113] At this point, the second stage of the second denoising process is completed, and the entire second denoising process is also completed.
[0114] 2. According to Get the restored image.
[0115] A basis The implementation method to obtain the repaired image is:
[0116] 1.1, according to Predict the second image .
[0117] Among them, the second image is the predicted 0-step output image, .
[0118] in, For the The noise processing parameters of the step, , is the step number indicator, No. Diffusion speed when adding noise.
[0119] 1.2, based on Get the restored image.
[0120] This step can be based on ,A large-scale Wensheng graph diffusion model is used to generate fine facial details and obtain the ,restored image.
[0121] For example,
[0122] 1) Compressed into latent space by encoder Then, starting from step 1, forward multi-step noise addition is performed to obtain the Step Image .
[0123] in, To compress into the latent space The target number of steps for adding noise to the image after , such as .
[0124] 2) Yes From Step 1 is started, and the reverse multi-step denoising process is performed to obtain the latent variable .
[0125] For example, the noise reduction principle based on the diffusion model From Step 1 is started, and the reverse multi-step denoising process is performed to obtain the latent variable .
[0126] 3) Based on The restored image is obtained through the decoder.
[0127] We can connect new channels on the channel dimension of the latent space of the Stable Diffusion Model to expand from [1, 4, 64, 64] to [1, 7, 64, 64], so that the input image can be connected to its latent space for guided generation.
[0128] In specific implementation, you can first Upsample to 512 512 resolution, and then encoded into a latent variable by the SDM (spatial diffusion model) encoder, and noise is added to the latent space to the 760th step according to the SNR (Signal-to-Noise Ratio) attribute, and then compared with the original input 64 The 64 images are concatenated in the channel dimension, denoised by DDIM (Dense Depth Image Matting) in the fine-tuned SDM network until step 0, and finally decoded into pixel space by the SDM decoder to obtain 512 The final repair result of 512.
[0129] Below , , The initial value of is 2. , , ,expand for , the default condition is , the preset threshold is For example, in step 102, the image to be repaired is denoised for 100 steps, and the image of the 100th step is obtained. (That is, the image after 100 steps of noise addition). In step 103, based on the low-pass filter ,right Starting from step 100, reverse multi-step denoising is performed, and after each step of denoising, the low-frequency components of the denoised image are exchanged with The low-frequency component of the image at step 50 is obtained (i.e., the image after noise reduction and exchange in step 50). In step 104, according to Predicted Image In step 105, Add noise for 250 steps and get the image at step 250 (That is, the image after 250 steps of noise addition). In step 106, based on the low-pass filter ,right Starting from step 250, reverse multi-step denoising is performed, and after each step of denoising, the low-frequency components of the denoised image are exchanged with The low-frequency component of the image at step 150 is obtained (That is, the image after denoising and swapping at step 150).
[0130] At this time, the low-pass filter of , the preset conditions are not met.
[0131] A. If The mean square error between the low-frequency component of the image to be repaired obtained in step 101 is no greater than , then the second stage of the second denoising process is performed.
[0132] B. If The mean square error between the low-frequency component of the image to be repaired obtained in step 101 is greater than , then Updated to 4 (i.e. ), and re-execute step 106.
[0133] When step 106 is executed again, Therefore, after executing step 106 Still 4, which does not meet the preset conditions.
[0134] b1, if step 106 is executed again The mean square error between the low-frequency component of the image to be repaired obtained in step 101 is no greater than , then the second stage of the second denoising process is performed.
[0135] b2, if step 106 is executed again The mean square error between the low-frequency component of the image to be repaired obtained in step 101 is greater than , then Updated to 8 (i.e. ), and re-execute step 106.
[0136] When step 106 is executed again, , and so on, until After 16, the second stage of the second denoising process is performed.
[0137] In the second stage of the second denoising process, a low-pass filter is used to , for the current Starting from step 150, reverse multi-step denoising is performed, and after each step of denoising, the low-frequency components of the denoised image are exchanged with The low-frequency component of the image at step 50 is obtained ;according to Predict the second image ;based on Get the restored image.
[0138] The face blindness restoration method provided in this embodiment can face extensive and serious degradation, including resolution degradation caused by various degradation factors, and repair it to obtain a better restoration effect, and can significantly reduce the computational consumption during reasoning.
[0139] The face blindness restoration method provided in this embodiment is implemented through two denoising processes, wherein the second denoising process is divided into two stages.
[0140] Before the first denoising process, obtain a low-resolution (e.g., 64*64) image to be repaired (e.g., the image format is numpy array), and then convert the input image tensor from [H, W, C] to [C, H, W], and then normalize the data to a mean of 0 and a variance of 1 to facilitate subsequent reasoning.
[0141] In the first noise addition process, Add noise to the image to be repaired to 100 steps.
[0142] In the first denoising process, the parameters The filter performs a filtering operation of downsampling the 100-step image by 2 times and then upsampling to obtain the low-frequency component of the 100-step image. At the same time, from step 100 to step 50, the parameter The filter pair performs a filtering operation of downsampling 2 times and then upsampling the image obtained after each denoising process to obtain the low-frequency component of the image after each denoising process. The low-frequency components of the denoised image are swapped with the low-frequency components of the 100-step image. The first denoising process stops at 50 steps.
[0143] Based on the 50-step image obtained by the first denoising, predict the 0-step output image , To obtain the preliminary restoration result, it is used as the input for the second denoising process.
[0144] In the second noise adding process, first Upsample to 256 256 resolution, and add noise to 250 steps according to the SNR attribute curve of the diffusion model with different resolutions.
[0145] In the first stage of the second denoising, the In the diffusion model of 256, denoising starts from step 250, and N=2 low-frequency replacement operations are still performed at each step. When denoising reaches step 150, the mean square error between the low-frequency components of the current denoised image and the original input image is calculated to measure the degree of degradation removal. If it is greater than the threshold 5e-3, it is considered that the degradation removal is insufficient, and N is replaced. 2 Enhance the de-degradation effect of the low-frequency filter, and denoise again from 250 steps to 150 steps, and so on. When N=16 or the mean square error is less than the threshold, the second stage of the second denoising is performed, that is, denoising from 150 steps to 50 steps, and the second denoising process stops when it reaches 50 steps. Figure 2 The structure shown completes this section.
[0146] The goals of face image restoration are to remove degradation and to keep the face identity unchanged. However, when the degradation is severe, completely removing the degradation will inevitably lead to changes in the face identity. Therefore, the second denoising process can balance the effects of removing degradation and preserving the face.
[0147] Based on the 50-step image obtained by the second denoising, the 0-step output image is predicted again ,based on Get the restored image.
[0148] Another basis The method for obtaining the restored image is as follows: the restored image is obtained through a stable diffusion model after injection.
[0149] Among them, the implementation process of the stable diffusion model after injection is as follows:
[0150] 201, obtain high-quality reference images.
[0151] The image to be restored and the reference image correspond to the same identity information. That is, the image to be restored and the reference image are images of the same face, but with different resolutions. This is like two images of a person at different times and / or in different locations, and / or two images of different resolutions at the same time and location.
[0152] 202 , obtaining identity information based on the reference image, and injecting the identity information into the stable diffusion model to obtain the injected stable diffusion model.
[0153] The specific implementation process of obtaining identity information based on the reference image is as follows:
[0154] 1. Project the reference image through MLP (Multilayer Perceptron) to obtain the projected identity features.
[0155] 2. Use the projected identity features as the basis for the Q-Former (query transformer) to query the CLIP (Contrastive Language–Image Pre-training, image encoder) image features in the reference image.
[0156] 3. Use shortcut structure to combine the projected identity features with CLIP image features to obtain identity information.
[0157] In practice, the identity features are projected through a multi-layer perceptron (MLP) as the query basis for Q-Former to extract identity-related CLIP image features, and a shortcut structure is used to combine the projected identity features with the identity-related CLIP image features to form the final identity features. This combination ensures that identity information can be more fully extracted from the reference facial image than relying solely on CLIP image features.
[0158] 203, will , the reference image and identity information are input into the trained identity-specific restoration model to obtain the restored image.
[0159] Among them, the loss function of the restoration model of a specific identity when obtaining the restored image is .
[0160] in, The loss function for obtaining the restored image for the restoration model of a specific identity, For the The latent space variables of the denoised image. For text prompt conditions, is the number of denoising steps, is the noise drawn from a standard Gaussian distribution, is the image to be repaired, is the reference image, is the noise prediction parameter, For the The prediction noise of the step, is the expected function.
[0161] By injecting identity information, the blindly repaired identity features can be maintained. When obtaining identity information based on the reference image, CLIP (image encoder) and Q-Former are integrated to extract identity information from high-quality reference images. In practice, the identity features are projected through a multi-layer perceptron (MLP) as a query basis for Q-Former to extract identity-related CLIP image features. At the same time, a shortcut structure is used to combine the projected identity features with the identity-related CLIP image features to form the final identity features. This combination ensures that identity information can be more fully extracted from the reference facial image than relying solely on CLIP image features.
[0162] The identity-specific inpainting models not only fully exploit the contents in the low-quality image (i.e., the image to be inpainted), but also extract sufficient identity information, and their direct collaboration is usually able to maintain faithful identity and visual quality simultaneously.
[0163] However, the general inpainting model has only been trained on the image to be inpainted and the corresponding high-resolution image pair, and has no ability to extract information from the reference image. If the identity information in the reference image is directly injected into the U-Net model, it cannot correctly generate a high-resolution face with identity preservation. In order to fully and accurately utilize the identity information in the reference image, the following loss function is adopted when the inpainting model of a specific identity obtains the inpainted image .
[0164] In addition, the identity-specific inpainting model adopts the structure and parameters of the stable diffusion model (such as Stable Diffusion v1.5) to ensure the generation ability. Therefore, a trainable blind inpainting control network (BFRControlNet) is created, which replicates the encoder and intermediate blocks of the denoising U-Net model in the stable diffusion model.
[0165] Specifically, the training process of the identity-specific repair model is as follows:
[0166] 401, use the real sample images and the corresponding low-quality sample images to train BFRControlNet and obtain a general repair model.
[0167] In the general inpainting model, BFRControlNet concatenates the channels of the corresponding low-quality image samples with the channels of the noise variables of the stable diffusion model.
[0168] That is to say, BFRControlNet can splice the low-quality image to be repaired in the pixel domain with the noise latent variable of the original input in the stable diffusion model on the channel axis. For example, the number of channels of the noise latent variable of the original input is 4, and the number of channels of the low-quality image to be repaired in the pixel domain is 3. After splicing, the number of input channels of BFRControlNet increases to 7. Therefore, the number of channels of the BFRControlNet input convolutional layer is also 7 to adapt to the spliced input.
[0169] In addition, the loss function of BFRControlNet .
[0170] in, is the loss function of BFRControlNet, For the The latent space variables of the denoised image. For text prompt conditions, is the number of denoising steps, is the noise drawn from a standard Gaussian distribution, For low quality images, is the noise prediction parameter, For the The prediction noise of the step, is the expected function.
[0171] During the training process, is a low-quality sample image; when repairing based on the trained general repair model, The low-quality image to be restored.
[0172] For example, during training, the content injection module injects the corresponding low-quality sample images into the stable diffusion model including BFRControlNet to obtain a universal repair model. When repairing the material to be repaired based on the trained universal repair model, the content injection module injects the low-quality image to be repaired into the universal repair model. Whether in the training phase or the repair phase of the material to be repaired, the universal repair model does not save specific identity information, but only repairs based on low-quality images (LQ).
[0173] The universal restoration model can make full use of the spatial structure and text information in the image to be restored during the face restoration process, avoid serious color shift problems, and the pixel-level stitching can avoid information loss when encoding the image to be restored into the latent space.
[0174] In the training of BFRControlNet, it can be initialized from the pre-trained stable diffusion model, and the parameters added by channel splicing are initialized to zero.
[0175] 402, inject the corresponding sample identity information into the general repair model.
[0176] For example, by decoupling the cross-attention mechanism, the corresponding sample identity information is injected into the general repair model. That is, the decoupling cross-attention mechanism is used to inject the sample identity information into the denoising U-Net model of the general repair model.
[0177] Among them, the decoupled cross attention mechanism is implemented using the following formula:
[0178] .
[0179] in, is the cross attention mechanism, For query, A key for a generic repair model, is the value of the general repair model, is a key based on the corresponding sample identity information, is the value based on the corresponding sample identity information, is the normalization function, for The dimension of for The transpose of for The transpose of .
[0180] 403 , training the injected general restoration model based on the real sample image, the corresponding low-quality sample image and the corresponding sample identity information to obtain a restoration model for a specific identity.
[0181] Among them, in each layer of the U-Net model of the specific identity restoration model, the corresponding low-quality sample image and the corresponding sample identity information are involved in the calculation.
[0182] In addition, in order to improve the detail generation capability of the method provided in this embodiment, the U-Net model is used to predict noise in the process of generating higher resolution images. The receptive field of the model is improved by using dilated convolutions on the convolutional layers of the U-Net intermediate layers of the model, thereby further improving the image generation details while ensuring the original prior of the generation process.
[0183] Specifically, the U-Net model is implemented using the following convolution formula:
[0184] ( .
[0185] in, is the feature map, is the preset expansion rate, is the convolution kernel, is the convolution parameter, is the spatial position in the feature map, is the spatial position within the convolution kernel.
[0186] The images generated at higher resolutions appear blurry and semantically biased, which can be attributed to the limited receptive field of the U-Net layer. The convolution kernel of the U-Net denoiser used in the method provided in this embodiment is a dilated convolution with a specific dilation rate δ. This convolution allows the high-resolution image generation method provided in this embodiment to cover a wider range of features without additional adjustment, while retaining the original prior at a higher resolution to sample nearby features to enhance local details, thereby conveniently combining comprehensive information in the generation process. The convolution process diagram is shown in FIG. Figure 3 shown.
[0187] In addition, the same noise level has different degrees of damage to images of different resolutions, which leads to the need to modify the noise level at the junction of different resolutions to improve the quality of image generation. In order to add different degrees of noise to images of different resolutions to improve the quality and fidelity of the generated images, the U-Net model controls the noise amplitude of each denoising through the following equation:
[0188] .
[0189] in, Adjustment coefficient, is the signal-to-noise ratio of the low-resolution image before denoising, is the signal-to-noise ratio of the high-resolution image after denoising.
[0190] The signal-to-noise ratio of any image , is the denoising step number indicator, For the The noise processing parameters of the step, , is the step number indicator, For the The noise factor of the step, , For the Diffusion speed when adding noise.
[0191] By adjusting By adjusting the value of , a new noise coefficient can be obtained under different resolution image generation tasks to adapt to the generation capability of the original model and improve the quality and fidelity of image generation.
[0192] The SNR (Signal-to-Noise Ratio) curves of different models are as follows: Figure 3 As shown, for images with high-quality original images, the final repaired image can use PSNR (Peak Signal-to-Noise Ratio) and SSIM (Mean Structural Similarity Index Measure) to measure its pixel-level quality. The higher the PSNR and SSIM, the closer the repair result is to the original image. FID (Frechet Inception Distance) can also be used to measure its perceptual quality. The lower the FID, the better the quality of the repair result. In addition, ArcfaceID indicates the degree of similarity between the repair result and the original image. The higher the ArcfaceID, the higher the similarity. The face blind repair method provided in this embodiment has excellent performance in these indicators and more indicators. Regarding time efficiency, the face blind repair method provided in this embodiment not only has comparable or better performance, but also improves the inference speed by at least 30%.
[0193] This embodiment provides a method for blind face restoration, obtaining an image to be restored ; Treat the repaired image Perform forward multi-step noise processing to obtain the Step Image ; Based on low-pass filter Perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image ;according to Predict the first image ;right Perform forward multi-step noise processing to obtain the Step Image ; Based on low-pass filter Perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image According to the current Relationship to preconditions, and The restored image is obtained by calculating the mean square error between the low frequency component of the image to be restored and the image to be restored. The method of this embodiment can improve the quality of face image restoration and reduce the time consumption of the restoration process.
[0194] Based on the same inventive concept of the face blindness repair method, this embodiment provides a face blindness repair device, see Figure 4 , the device comprises:
[0195] Acquisition module 401, used to acquire the image to be repaired Among them, the image to be repaired For low-resolution images.
[0196] The first noise adding module 402 is used to repair the image , starting from step 1, perform forward multi-step noise processing to obtain the Step Image .in, is the target number of steps for adding noise to the image to be repaired.
[0197] The first denoising module 403 is used to remove noise based on a low-pass filter. ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image .in, is the reverse denoising step number identifier, , are the parameters of the low-pass filter, The initial value of is determined according to the degree of preservation of face identity information. For The denoising target number of steps, .
[0198] The first prediction module 404 is used to Predict the first image .
[0199] The second noise adding module 405 is used to , starting from step 1, perform forward multi-step noise processing to obtain the Step Image .in, For The target number of noise steps.
[0200] The second denoising module 406 is used to remove noise based on a low-pass filter. ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image .in, is the reverse denoising step number identifier, , For The denoising target number of steps, .
[0201] Loop module 407, used if the current The preset conditions are not met and the current If the mean square error between the low-frequency component of the image to be repaired is greater than the preset threshold, the image is enlarged. , repeat the second denoising module 406 and the loop module 407. Satisfy the preset conditions, or the current If the mean square error between the low-frequency component of the image to be repaired and the low-frequency component is not greater than a preset threshold, the processing module 408 is executed.
[0202] Processing module 408 is used to process ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image .according to Predict the second image ,based on Get the restored image. Among them, is the reverse denoising step number identifier, , For The denoising target number of steps, .
[0203] Optionally, the first denoising module 402 performs denoising on the image to be repaired by Get the image of this step.
[0204] in, is the number of noise adding steps for the repaired image. , For the Step image, is the noise drawn from a standard Gaussian distribution, For the The noise processing parameters of the step, , is the step number indicator, For the The noise factor of the step, , For the Diffusion speed when adding noise.
[0205] Optionally, the first denoising module 403 From Starting from the step, after each step of the reverse denoising process, Exchange the low-frequency components of the denoised image with low frequency components.
[0206] in, For the After the denoising process, for The low-frequency component of for The low-frequency component of For the Step image.
[0207] The second denoising module 406 is From Starting from the step, after each step of the reverse denoising process, Exchange the low-frequency components of the denoised image with low frequency components.
[0208] in, For the After the denoising process, for The low-frequency component of for The low-frequency component of For the Step image.
[0209] Processing module 408 From Starting from the step, after each step of the reverse denoising process, Exchange the low-frequency components of the denoised image with low frequency components.
[0210] in, For the After the denoising process, for The low-frequency component of for The low-frequency component of For the Step image.
[0211] Optionally, the first prediction module 404 is based on Predict the first image .
[0212] in, For the The noise processing parameters of the step, , is the step number indicator, For the The diffusion speed when adding noise, is the noise drawn from a standard Gaussian distribution.
[0213] Processing module 408 Predict the second image .
[0214] in, For the The noise processing parameters of the step, , is the step number indicator, No. Diffusion speed when adding noise.
[0215] The processing module 408 is used to Compressed into latent space by encoder Then, starting from step 1, forward multi-step noise addition is performed to obtain the Step Image .in, To compress into the latent space The target number of steps for adding noise to the image after . From Step 1 is started, and the reverse multi-step denoising process is performed to obtain the latent variable .based on The restored image is obtained through the decoder.
[0216] The acquisition module 401 is used to acquire an input image. The input image is a low-resolution image. The tensor of the input image is converted from [H, W, C] to [C, H, W], where H is the height, W is the width, and C is the number of channels. The values of the converted tensor in each dimension are normalized to [0, 1]. The values of the normalized tensor in each dimension are normalized to have a mean of 0 and a variance of 1. The normalized image is used as the image to be repaired.
[0217] Optionally, , , The initial value of is 2. , , .
[0218] expand ,include: .
[0219] The default condition is , the preset threshold is .
[0220] The device provided in this embodiment can improve the quality of facial image restoration and reduce the time consumption of the restoration process.
[0221] Based on the same inventive concept of the method for repairing blind faces, this embodiment provides an electronic device, such as Figure 5 As shown, it includes: a memory 501, a processor 502, and a computer program.
[0222] The computer program is stored in the memory 501 and is configured to be executed by the processor 502 to implement the above-mentioned face blindness repair method.
[0223] Specifically,
[0224] Get the image to be repaired Among them, the image to be repaired For low-resolution images.
[0225] Treat the repaired image , starting from step 1, perform forward multi-step noise processing to obtain the Step Image .in, is the target number of steps for adding noise to the image to be repaired.
[0226] Based on low pass filter ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image .in, is the reverse denoising step number identifier, , are the parameters of the low-pass filter, The initial value of is determined according to the degree of preservation of face identity information. For The denoising target number of steps, .
[0227] according to Predict the first image .
[0228] right , starting from step 1, perform forward multi-step noise processing to obtain the Step Image .in, For The target number of noise steps.
[0229] Based on low pass filter ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image .in, is the reverse denoising step number identifier, , For The denoising target number of steps, .
[0230] If the current The preset conditions are not met and the current If the mean square error between the low-frequency component of the image to be repaired is greater than the preset threshold, the image is enlarged. , repeatedly performing a low-pass filter based ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image and subsequent steps.
[0231] If the current Satisfy the preset conditions, or the current If the mean square error between the low-frequency component of the image to be repaired and the low-frequency component of the image to be repaired is not greater than the preset threshold, then the low-pass filter is used to ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image .according to Predict the second image ,based on Get the restored image. Among them, is the reverse denoising step number identifier, , For The denoising target number of steps, .
[0232] Optionally, in each step of the noise addition process to be performed on the repaired image, Get the image of this step.
[0233] in, is the number of noise adding steps for the repaired image. , For the Step image, is the noise drawn from a standard Gaussian distribution, For the The noise processing parameters of the step, , is the step number indicator, For the The noise factor of the step, , For the Diffusion speed when adding noise.
[0234] Optionally, From Starting from the step, after each step of the reverse denoising process, Exchange the low-frequency components of the denoised image with low frequency components.
[0235] in, For the After the denoising process, for The low-frequency component of for The low-frequency component of For the Step image.
[0236] right From Starting from the step, after each step of the reverse denoising process, Exchange the low-frequency components of the denoised image with low frequency components.
[0237] in, For the After the denoising process, for The low-frequency component of for The low-frequency component of For the Step image.
[0238] right From Starting from the step, after each step of the reverse denoising process, Exchange the low-frequency components of the denoised image with low frequency components.
[0239] in, For the After the denoising process, for The low-frequency component of for The low-frequency component of For the Step image.
[0240] Optionally, according to Predict the first image .
[0241] in, For the The noise processing parameters of the step, , is the step number indicator, For the The diffusion speed when adding noise, is the noise drawn from a standard Gaussian distribution.
[0242] according to Predict the second image .
[0243] in, For the The noise processing parameters of the step, , is the step number indicator, No. Diffusion speed when adding noise.
[0244] Optionally, based on Get the restored image, including:
[0245] Will Compressed into latent space by encoder Then, starting from step 1, forward multi-step noise addition is performed to obtain the Step Image .in, To compress into the latent space The target number of steps for adding noise to the image after denoising.
[0246] right From Step 1 is started, and the reverse multi-step denoising process is performed to obtain the latent variable .
[0247] based on The restored image is obtained through the decoder.
[0248] Optionally, obtaining an image to be repaired includes:
[0249] Get the input image. The input image is a low-resolution image.
[0250] Convert the input image tensor from [H, W, C] to [C, H, W], where H is the height, W is the width, and C is the number of channels.
[0251] Normalize the values of the converted tensor in each dimension to [0, 1].
[0252] The normalized tensor is normalized in each dimension to have a mean of 0 and a variance of 1.
[0253] The standardized image is used as the image to be repaired.
[0254] Optionally, , , The initial value of is 2. , , .
[0255] expand ,include: .
[0256] The default condition is , the preset threshold is .
[0257] The electronic device provided by this embodiment has a computer program executed by a processor to improve the quality of facial image restoration and reduce the time consumption of the restoration process.
[0258] Based on the same inventive concept of the method for restoring blind faces, this embodiment provides a computer-readable storage medium, on which a computer program is stored. The computer program is executed by a processor to implement the method for restoring blind faces.
[0259] Specifically,
[0260] Get the image to be repaired Among them, the image to be repaired For low-resolution images.
[0261] Treat the repaired image , starting from step 1, perform forward multi-step noise processing to obtain the Step Image .in, is the target number of steps for adding noise to the image to be repaired.
[0262] Based on low pass filter ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image .in, is the reverse denoising step number identifier, , are the parameters of the low-pass filter, The initial value of is determined according to the degree of preservation of face identity information. For The denoising target number of steps, .
[0263] according to Predict the first image .
[0264] right , starting from step 1, perform forward multi-step noise processing to obtain the Step Image .in, For The target number of noise steps.
[0265] Based on low pass filter ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image .in, is the reverse denoising step number identifier, , For The denoising target number of steps, .
[0266] If the current The preset conditions are not met and the current If the mean square error between the low-frequency component of the image to be repaired is greater than the preset threshold, the image is enlarged. , repeatedly performing a low-pass filter based ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image and subsequent steps.
[0267] If the current Satisfy the preset conditions, or the current If the mean square error between the low-frequency component of the image to be repaired and the low-frequency component of the image to be repaired is not greater than the preset threshold, then the low-pass filter is used to ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image .according to Predict the second image ,based on Get the restored image. Among them, is the reverse denoising step number identifier, , For The denoising target number of steps, .
[0268] Optionally, in each step of the noise addition process to be performed on the repaired image, Get the image of this step.
[0269] in, is the number of noise adding steps for the repaired image. , For the Step image, is the noise drawn from a standard Gaussian distribution, For the The noise processing parameters of the step, , is the step number indicator, For the The noise factor of the step, , For the Diffusion speed when adding noise.
[0270] Optionally, From Starting from the step, after each step of the reverse denoising process, Exchange the low-frequency components of the denoised image with low frequency components.
[0271] in, For the After the denoising process, for The low-frequency component of for The low-frequency component of For the Step image.
[0272] right From Starting from the step, after each step of the reverse denoising process, Exchange the low-frequency components of the denoised image with low frequency components.
[0273] in, For the After the denoising process, for The low-frequency component of for The low-frequency component of For the Step image.
[0274] right From Starting from the step, after each step of the reverse denoising process, Exchange the low-frequency components of the denoised image with low frequency components.
[0275] in, For the After the denoising process, for The low-frequency component of for The low-frequency component of For the Step image.
[0276] Optionally, according to Predict the first image .
[0277] in, For the The noise processing parameters of the step, , is the step number indicator, For the The diffusion speed when adding noise, is the noise drawn from a standard Gaussian distribution.
[0278] according to Predict the second image .
[0279] in, For the The noise processing parameters of the step, , is the step number indicator, No. Diffusion speed when adding noise.
[0280] Optionally, based on Get the restored image, including:
[0281] Will Compressed into latent space by encoder Then, starting from step 1, forward multi-step noise addition is performed to obtain the Step Image .in, To compress into the latent space The target number of steps for adding noise to the image after denoising.
[0282] right From Step 1 is started, and the reverse multi-step denoising process is performed to obtain the latent variable .
[0283] based on The restored image is obtained through the decoder.
[0284] Optionally, obtaining an image to be repaired includes:
[0285] Get the input image. The input image is a low-resolution image.
[0286] Convert the input image tensor from [H, W, C] to [C, H, W], where H is the height, W is the width, and C is the number of channels.
[0287] Normalize the values of the converted tensor in each dimension to [0, 1].
[0288] The normalized tensor is normalized in each dimension to have a mean of 0 and a variance of 1.
[0289] The standardized image is used as the image to be repaired.
[0290] Optionally, , , The initial value of is 2. , , .
[0291] expand ,include: .
[0292] The default condition is , the preset threshold is .
[0293] The computer-readable storage medium provided in this embodiment has a computer program on it which is executed by a processor to improve the quality of facial image restoration and reduce the time consumption of the restoration process.
[0294] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present application may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.
[0295] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0296] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0297] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0298] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0299] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A method for repairing face blindness, characterized in that: The method comprises: Get the image to be repaired ; Wherein, the image to be repaired For low-resolution images; For the image to be repaired , starting from step 1, perform forward multi-step noise processing to obtain the Step Image ;in, is the target number of steps for adding noise to the image to be repaired; Based on low pass filter ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image ;in, is the reverse denoising step number identifier, , are the parameters of the low-pass filter, The initial value of is determined according to the degree of preservation of face identity information. For The denoising target number of steps, ; For the Step image; according to Predict the first image ; right , starting from step 1, perform forward multi-step noise processing to obtain the Step Image ;in, For The number of noise-added target steps; Based on low pass filter ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image ;in, is the reverse denoising step number identifier, , For The denoising target number of steps, ; For the Step image; If the current The preset conditions are not met and the current The mean square error between the low-frequency component of the image to be repaired and the image to be repaired is greater than a preset threshold, then the image is enlarged. , repeatedly performing a low-pass filter based ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image the steps and subsequent steps; If the current Satisfy the preset conditions, or the current The mean square error between the low-frequency component of the image to be repaired and the low-frequency component of the image to be repaired is not greater than a preset threshold, then based on the low-pass filter ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image ;according to Predict the second image ,based on Get the restored image; where, is the reverse denoising step number identifier, , For The denoising target number of steps, For the Step image.
2. The method according to claim 1, characterized in that In each step of the noise addition process on the image to be repaired, Get the image of this step; in, is the number of steps for adding noise to the image to be repaired, , For the Step image, is the noise drawn from a standard Gaussian distribution, For the The noise processing parameters of the step, , is the step number indicator, For the The noise factor of the step, , For the Diffusion speed when adding noise.
3. The method according to claim 1, characterized in that right From Starting from the step, after each step of denoising, the Exchange the low-frequency components of the denoised image with The low-frequency component of in, For the After the denoising process, for The low-frequency component of for The low-frequency component of right From Starting from the step, after each step of denoising, the Exchange the low-frequency components of the denoised image with The low-frequency component of in, For the After the denoising process, for The low-frequency component of for The low-frequency component of right From Starting from the step, after each step of denoising, the Exchange the low-frequency components of the denoised image with The low-frequency component of in, For the After the denoising process, for The low-frequency component of for low frequency components.
4. The method according to claim 1, characterized in that: according to Predict the first image ; in, For the The noise processing parameters of the step, , is the step number indicator, For the The diffusion speed when adding noise, is the noise drawn from a standard Gaussian distribution; according to Predict the second image ; in, For the The noise processing parameters of the step, , is the step number indicator, No. Diffusion speed when adding noise.
5. The method according to claim 1, characterized in that The basis Get the restored image, including: Will Compressed into latent space by encoder Then, starting from step 1, forward multi-step noise addition is performed to obtain the Step Image ;in, To compress into the latent space The target number of steps for adding noise to the image after denoising; right From Step 1 is started, and the reverse multi-step denoising process is performed to obtain the latent variable ; based on The restored image is obtained through the decoder.
6. The method according to claim 1, characterized in that The step of obtaining the image to be repaired comprises: Acquire an input image; the input image is a low-resolution image; Convert the input image tensor from [H, W, C] to [C, H, W], where H is the height, W is the width, and C is the number of channels; Normalize the values of the converted tensor in each dimension to [0, 1]; Normalize the values of the normalized tensor in each dimension to a mean of 0 and a variance of 1; The standardized image is used as the image to be repaired.
7. The method according to claim 1, characterized in that , , The initial value of is 2. , , ; The expansion ,include: ; The pre-condition is , the preset threshold is .
8. A face blindness repair device, characterized in that: The device comprises: Acquisition module, used to obtain the image to be repaired ; Wherein, the image to be repaired For low-resolution images; The first noise adding module is used to add noise to the image to be repaired. , starting from step 1, perform forward multi-step noise processing to obtain the Step Image ;in, is the target number of steps for adding noise to the image to be repaired; The first denoising module is used based on a low-pass filter ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image ;in, is the reverse denoising step number identifier, , are the parameters of the low-pass filter, The initial value of is determined according to the degree of preservation of face identity information. For The denoising target number of steps, ; For the Step image; The first prediction module is used to Predict the first image ; The second noise adding module is used to , starting from step 1, perform forward multi-step noise processing to obtain the Step Image ;in, For The number of noise-added target steps; The second denoising module is used based on a low-pass filter ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image ;in, is the reverse denoising step number identifier, , For The denoising target number of steps, ; For the Step image; Loop module, used if the current The preset conditions are not met and the current The mean square error between the low-frequency component of the image to be repaired and the image to be repaired is greater than a preset threshold, then the image is enlarged. , repeat the second denoising module and the loop module; if the current Satisfy the preset conditions, or the current If the mean square error between the low-frequency component of the image to be repaired and the low-frequency component is not greater than a preset threshold, the processing module is executed; Processing module for low-pass filter based ,right From Step 1 is used to perform reverse multi-step denoising, and each After the denoising process, the low-frequency components of the denoised image are exchanged with The low-frequency component of Step Image ;according to Predict the second image ,based on Get the restored image; where, is the reverse denoising step number identifier, , For The denoising target number of steps, For the Step image.
9. An electronic device, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: A computer program is stored thereon; the computer program is executed by a processor to implement the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Unified image repair and enhancement method generating diffusion priori
CN116071256A
Human blind face image restoration system and method based on potential diffusion model
CN116402719A
