Face image super-resolution method and device of diffusion model based on residual prediction

Abstract

The invention discloses a face image super-resolution method and device of a diffusion model based on residual prediction, and relates to the field of image super-resolution, and the method comprises the steps: obtaining a to-be-reconstructed low-resolution image; a diffusion model based on residual prediction and a conditional noise predictor are constructed and trained, and a trained diffusion model and a trained conditional noise predictor are obtained; gaussian noise is sampled from standard Gaussian distribution to serve as residual noise images, a to-be-reconstructed low-resolution image, the residual noise image of each step and the corresponding step length are input into a trained conditional noise predictor, prediction noise of each step is obtained through prediction, and the low-resolution image is reconstructed. Iteratively executing a de-noising process of T steps through a reverse de-noising network in the trained diffusion model to obtain a restored residual image; and adding the restored residual image and the up-sampled low-resolution image to be reconstructed to obtain a restored high-resolution image. The image super-resolution method solves the problems that an existing image super-resolution method is complex in generation process and limited in detail recovery.

Description

Technical Field

[0001] The present invention relates to the field of image super-resolution, and in particular to a facial image super-resolution method and device based on a diffusion model of residual prediction. Background Art

[0002] Current image super-resolution methods based on diffusion models mainly map low-resolution images to high-resolution space by gradually adding and removing noise. Among them, the standard diffusion model achieves the gradual restoration of image details through a large number of sampling steps, but the computational cost is high and the generation process is slow; the fast diffusion model reduces the number of steps by improving the sampling strategy, but may sacrifice some generation quality under low step conditions; the conditional diffusion model guides the diffusion process by introducing conditional information (such as the features of the input low-resolution image), thereby improving the stability and accuracy of the generation. However, these methods usually directly model high-resolution images, resulting in a complex generation process and limited restoration of details. Summary of the invention

[0003] The purpose of this application is to propose a facial image super-resolution method and device based on a diffusion model of residual prediction in response to the above-mentioned technical problems.

[0004] In a first aspect, the present invention provides a facial image super-resolution method based on a diffusion model of residual prediction, comprising the following steps:

[0005] Acquire a low-resolution image to be reconstructed;

[0006] A diffusion model and a conditional noise predictor based on residual prediction are constructed. The diffusion model includes a forward denoising network and a reverse denoising network. During the training process of the diffusion model and the conditional noise predictor, the residual image of the real high-resolution image and the low-resolution image is input into the forward denoising network, and Gaussian noise of t steps is iteratively added to obtain the residual noise image of the t-th step; the low-resolution image, the residual noise image of the t-th step and the number of steps t are input into the conditional noise predictor to obtain the predicted noise of the t-th step; the residual noise image of the t-th step, the predicted noise of the t-th step and the number of steps t are input into the reverse denoising network to obtain the restored residual image of the t-th step; a mean square error loss function is constructed according to the added Gaussian noise and the predicted noise, and the diffusion model and the conditional noise predictor are trained using the mean square error loss function to obtain a trained diffusion model and a trained conditional noise predictor;

[0007] Gaussian noise is sampled from a standard Gaussian distribution as a residual noise image. The low-resolution image to be reconstructed, the residual noise image of each step and the corresponding step size are input into a trained conditional noise predictor to predict the predicted noise of each step. The denoising process of T steps is iteratively performed through a reverse denoising network in a trained diffusion model to obtain a restored residual image. The restored residual image is added to the upsampled low-resolution image to be reconstructed to obtain a restored high-resolution image.

[0008] Preferably, the conditional noise predictor includes a conditional guidance branch and a noise prediction network based on Unet; the low-resolution image is input into the conditional guidance branch to obtain a coded feature map; the coded feature map, the residual noise image of the tth step and the number of steps t are input into the noise prediction network to obtain the predicted noise of the tth step.

[0009] Preferably, the conditional guidance branch includes an input layer, a feature extraction module, a global feature aggregation module and an output layer connected in sequence, the input layer includes a convolution layer with a convolution kernel size of 3×3, the low-resolution image is input to the input layer, and a shallow feature coding map is extracted; the feature extraction module includes a plurality of dense residual blocks connected in sequence, the shallow feature coding map is input to the feature extraction module, and a deep feature coding map is extracted; the global feature aggregation module includes a global average pooling layer and a fully connected layer connected in sequence, the deep feature coding map is input to the global feature aggregation module, and a global feature coding map is generated; the output layer includes a convolution layer with a convolution kernel size of 1×1, the global feature coding map is input to the output layer, and a coding feature map is generated.

[0010] Preferably, the dense residual block includes a first dense block and a second dense block, the input features of the dense residual block are input into the first dense block, and the first intermediate features are output, the input features of the dense residual block are locally residually connected with the first intermediate features to obtain the second intermediate features, the second intermediate features are input into the second dense block, and the third intermediate features are output, the second intermediate features are locally residually connected with the third intermediate features, and are globally residually connected with the input features of the dense residual block to obtain the output features of the dense residual block.

[0011] Preferably, the first dense block and the second dense block both adopt a dense structure, the dense structure includes a first convolutional layer, a second convolutional layer and a third convolutional layer connected in sequence, the convolution kernel sizes of the first convolutional layer, the second convolutional layer and the third convolutional layer are all 3×3, the first convolutional layer and the second convolutional layer both use a Leaky ReLU activation function, the input features of the dense structure are concatenated with the output features of the first convolutional layer using dense jump connections to obtain the input features of the second convolutional layer, and the input features of the dense structure and the input features of the second convolutional layer are concatenated with the output features of the second convolutional layer using dense jump connections as the input features of the third convolutional layer.

[0012] Preferably, the noise prediction network includes a feature extraction path, a feature restoration path and a result generation module; the feature extraction path includes a primary feature extraction module, an intermediate feature extraction module and an advanced feature extraction module connected in sequence; the feature restoration path includes an advanced restoration module, an intermediate restoration module and a primary restoration module connected in sequence; the primary feature extraction module, the intermediate feature extraction module and the advanced feature extraction module all include a convolution layer, a cross attention layer and a downsampling layer connected in sequence; the advanced restoration module includes a convolution layer, a cross attention layer and a normalization layer connected in sequence; the primary feature extraction module, the intermediate feature extraction module and the result generation module all include a convolution layer, a cross attention layer and an upsampling layer connected in sequence; the encoded feature map and the step number t are spliced ​​and input into each cross attention layer. The residual noise image of the t-th step is input into the primary feature extraction module to obtain a primary compressed feature map, the primary compressed feature map is input into the intermediate feature extraction module to obtain an intermediate compressed feature map, the intermediate compressed feature map is input into the advanced feature extraction module to obtain an advanced compressed feature map; the advanced compressed feature map is input into the advanced restoration module to obtain an advanced restored feature map; the advanced compressed feature map is spliced ​​with the advanced restored feature map through a jump connection and input into the intermediate restoration module to obtain an intermediate restored feature map, the intermediate compressed feature map is spliced ​​with the intermediate restored feature map through a jump connection and input into the primary restoration module to obtain a primary restored feature map; the primary compressed feature map is spliced ​​with the primary restored feature map through a jump connection and input into the result generation module to obtain the predicted noise of the t-th step.

[0013] In a second aspect, the present invention provides a facial image super-resolution device based on a diffusion model of residual prediction, comprising:

[0014] An image acquisition module, configured to acquire a low-resolution image to be reconstructed;

[0015] The model building module is configured to build a diffusion model and a conditional noise predictor based on residual prediction, the diffusion model includes a forward denoising network and a reverse denoising network. During the training process of the diffusion model and the conditional noise predictor, the residual image of the real high-resolution image and the low-resolution image is input into the forward denoising network, and Gaussian noise of t steps is iteratively added to obtain the residual noise image of the t-th step; the low-resolution image, the residual noise image of the t-th step and the number of steps t are input into the conditional noise predictor to obtain the predicted noise of the t-th step; the residual noise image of the t-th step, the predicted noise of the t-th step and the number of steps t are input into the reverse denoising network to obtain the restored residual image of the t-th step; a mean square error loss function is constructed according to the added Gaussian noise and the predicted noise, and the diffusion model and the conditional noise predictor are trained using the mean square error loss function to obtain a trained diffusion model and a trained conditional noise predictor;

[0016] The restoration module is configured to sample Gaussian noise from a standard Gaussian distribution as a residual noise image, input the low-resolution image to be reconstructed, the residual noise image of each step and the corresponding step size into the trained conditional noise predictor, predict the predicted noise of each step, and iteratively perform a denoising process of T steps through the trained diffusion model to obtain a restored residual image; the restored residual image is added to the upsampled low-resolution image to be reconstructed to obtain a restored high-resolution image.

[0017] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0018] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0019] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] (1) The facial image super-resolution method based on the residual prediction diffusion model proposed in this paper focuses on the residual information modeling between high-resolution images and low-resolution images, which significantly reduces the generation difficulty and computational complexity. By learning residual features, this method can capture image details more efficiently while reducing the number of sampling steps to achieve fast and high-quality super-resolution results. Especially when processing facial images, the residual prediction-based diffusion model can more accurately restore details based on low-resolution input, ensuring the clarity and consistency of the generated images.

[0022] (2) The face image super-resolution method based on the residual prediction diffusion model proposed in the present invention estimates the prediction noise of each step through a conditional noise predictor. The conditional noise predictor combines the conditional guidance branch with the noise prediction network. The low-resolution image is passed through the conditional guidance branch to obtain a coded feature map. The coded feature map guides the noise prediction process in the noise prediction network, which can improve the generation quality and is more efficient and superior than other diffusion models. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A schematic diagram of a flow chart of a facial image super-resolution method based on a diffusion model of residual prediction according to an embodiment of the present application;

[0025] Figure 2 A schematic diagram of a conditional noise predictor of a facial image super-resolution method based on a diffusion model of residual prediction according to an embodiment of the present application;

[0026] Figure 3 A schematic diagram of a conditional guidance branch of a facial image super-resolution method based on a diffusion model of residual prediction according to an embodiment of the present application;

[0027] Figure 4 A schematic diagram of a noise prediction network of a facial image super-resolution method based on a diffusion model of residual prediction according to an embodiment of the present application;

[0028] Figure 5 This is a comparison result diagram of the face image super-resolution method based on the residual prediction diffusion model in the embodiment of the present application before and after restoration;

[0029] Figure 6 A schematic diagram of a facial image super-resolution device based on a diffusion model of residual prediction according to an embodiment of the present application;

[0030] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] Figure 1 A facial image super-resolution method based on a diffusion model of residual prediction provided by an embodiment of the present application is shown, comprising the following steps:

[0033] S1, obtaining a low-resolution image to be reconstructed.

[0034] Specifically, a low-resolution image to be reconstructed is collected. The embodiment of the present application takes a face image as an example, so a low-resolution face image to be reconstructed can be collected.

[0035] S2, construct a diffusion model and a conditional noise predictor based on residual prediction, the diffusion model includes a forward denoising network and a reverse denoising network, during the training process of the diffusion model and the conditional noise predictor, the residual image of the real high-resolution image and the low-resolution image is input into the forward denoising network, and Gaussian noise of t steps is iteratively added to obtain the residual noise image of the t-th step; the low-resolution image, the residual noise image of the t-th step and the number of steps t are input into the conditional noise predictor to obtain the predicted noise of the t-th step, the residual noise image of the t-th step, the predicted noise of the t-th step and the number of steps t are input into the reverse denoising network to obtain the restored residual image of the t-th step; a mean square error loss function is constructed according to the added Gaussian noise and the predicted noise, and the diffusion model and the conditional noise predictor are trained using the mean square error loss function to obtain a trained diffusion model and a trained conditional noise predictor.

[0036] Specifically, the embodiment of the present application adopts a diffusion model based on residual prediction, which includes two processes, namely, a forward denoising process and a reverse denoising process, which correspond to a forward denoising network and a reverse denoising network respectively. With low-resolution images The residual image As the input of the forward noise network of the diffusion model, then in the residual image On the basis of iteratively adding T steps of Gaussian noise, for the reverse denoising process, the residual noise image is gradually iterated in the reverse denoising network to perform the denoising process of T steps, and the final restored residual image can be obtained. The loss function used in the training process of the diffusion model is the mean square error loss function constructed based on Gaussian noise and prediction noise. The specific derivation process belongs to the prior art and will not be repeated here. The prediction noise is predicted by the conditional noise predictor.

[0037] In a specific embodiment, the conditional noise predictor includes a conditional guidance branch and a noise prediction network based on Unet; a low-resolution image is input into the conditional guidance branch to obtain a coded feature map; the coded feature map, the residual noise image of the tth step and the number of steps t are input into the noise prediction network to obtain the predicted noise of the tth step.

[0038] In a specific embodiment, the conditional guidance branch includes an input layer, a feature extraction module, a global feature aggregation module and an output layer connected in sequence, the input layer includes a convolution layer with a convolution kernel size of 3×3, the low-resolution image is input to the input layer, and a shallow feature coding map is extracted; the feature extraction module includes a plurality of dense residual blocks connected in sequence, the shallow feature coding map is input to the feature extraction module, and a deep feature coding map is extracted; the global feature aggregation module includes a global average pooling layer and a fully connected layer connected in sequence, the deep feature coding map is input to the global feature aggregation module, and a global feature coding map is generated; the output layer includes a convolution layer with a convolution kernel size of 1×1, the global feature coding map is input to the output layer, and a coding feature map is generated.

[0039] In a specific embodiment, the dense residual block includes a first dense block and a second dense block, the input features of the dense residual block are input to the first dense block, and the first intermediate features are output, the input features of the dense residual block are locally residually connected to the first intermediate features to obtain the second intermediate features, the second intermediate features are input to the second dense block, and the third intermediate features are output, the second intermediate features are locally residually connected to the third intermediate features, and are globally residually connected to the input features of the dense residual block to obtain the output features of the dense residual block.

[0040] In a specific embodiment, the first dense block and the second dense block both adopt a dense structure, the dense structure includes a first convolutional layer, a second convolutional layer and a third convolutional layer connected in sequence, the convolution kernel sizes of the first convolutional layer, the second convolutional layer and the third convolutional layer are all 3×3, the first convolutional layer and the second convolutional layer both use a Leaky ReLU activation function, the input features of the dense structure are concatenated with the output features of the first convolutional layer using a dense jump connection to obtain the input features of the second convolutional layer, the input features of the dense structure and the input features of the second convolutional layer are concatenated with the output features of the second convolutional layer using a dense jump connection as the input features of the third convolutional layer.

[0041] Specifically, refer to Figure 2 The conditional noise predictor is composed of a conditional guidance branch and a noise prediction network based on Unet. The purpose of the conditional guidance branch is to transform the low-resolution image Encoding operation is performed, and this information is added to the Unet-based noise prediction network in each step of the reverse denoising process to guide the reverse denoising process to the corresponding high-resolution space. Figure 3 Specifically, the conditional guidance branch is composed of an input layer, a feature extraction module, a global feature aggregation module, and an output layer connected in series. The input layer consists of a convolutional layer with a convolution kernel size of 3×3 to extract the shallow feature encoding map. The feature extraction module is composed of multiple dense residual blocks (RRDB) stacked in sequence to gradually extract multi-level deep feature coding maps. . Each RRDB consists of two dense blocks (Dense Block) with local residual connections and global residual connections, which can effectively capture multi-scale information and improve the efficiency of gradient transfer. Each dense block contains 3 convolutional layers, and the output of each layer is spliced ​​with the input of the subsequent layer through dense jump connections. The convolution kernel size of the three convolutional layers is 3×3. The first two convolutional layers use the LeakyReLU activation function and keep the resolution of the feature map unchanged. Local residual connection: The input and output of the dense block are added through an addition operation to learn the residual representation of local features, thereby enhancing the ability to extract deep features. Global residual connection: The input and output of the entire RRDB are added through a global residual path to ensure that the features will not be significantly attenuated when stacked in deep layers. The global feature aggregation module extracts the global information of the input features through the global average pooling layer, and then maps the features to a fixed dimension through the fully connected layer to finally generate a global feature encoding map. The output layer consists of a convolution layer with a convolution kernel size of 1×1 to compress the channel dimension and generate the final encoded feature map .

[0042] In a specific embodiment, the noise prediction network includes a feature extraction path, a feature restoration path and a result generation module; the feature extraction path includes a primary feature extraction module, an intermediate feature extraction module and an advanced feature extraction module connected in sequence; the feature restoration path includes an advanced restoration module, an intermediate restoration module and a primary restoration module connected in sequence; the primary feature extraction module, the intermediate feature extraction module and the advanced feature extraction module all include a convolution layer, a cross attention layer and a downsampling layer connected in sequence; the advanced restoration module includes a convolution layer, a cross attention layer and a normalization layer connected in sequence; the primary feature extraction module, the intermediate feature extraction module and the result generation module all include a convolution layer, a cross attention layer and an upsampling layer connected in sequence; the encoded feature map and the step number t are spliced ​​and input into each cross In the attention layer, the residual noise image of the tth step is input into the primary feature extraction module to obtain a primary compressed feature map, the primary compressed feature map is input into the intermediate feature extraction module to obtain an intermediate compressed feature map, the intermediate compressed feature map is input into the advanced feature extraction module to obtain an advanced compressed feature map; the advanced compressed feature map is input into the advanced restoration module to obtain an advanced restored feature map; the advanced compressed feature map is spliced ​​with the advanced restored feature map through a jump connection and input into the intermediate restoration module to obtain an intermediate restored feature map, the intermediate compressed feature map is spliced ​​with the intermediate restored feature map through a jump connection and input into the primary restoration module to obtain a primary restored feature map; the primary compressed feature map is spliced ​​with the primary restored feature map through a jump connection and input into the result generation module to obtain the predicted noise of the tth step.

[0043] Specifically, refer to Figure 4 The noise prediction network includes a feature extraction path, a feature restoration path and a result generation module. The feature extraction path is composed of a primary feature extraction module, an intermediate feature extraction module and an advanced feature extraction module in sequence, and gradually extracts deeper feature representations. Each module realizes feature encoding through a convolution layer with a convolution kernel size of 3×3, a cross-attention layer and a downsampling layer. The downsampling layer adopts a maximum pooling layer with a step size of 2 and a pooling kernel of 2×2. In addition to the output of the convolution layer, the input of the cross-attention layer also includes a coded feature map and the number of steps t, t=1,2,…,T; the feature restoration path is composed of an advanced restoration module, an intermediate restoration module and a primary restoration module, and gradually restores the resolution and details of the features. The advanced restoration module includes a convolution layer, a cross-attention layer and a normalization layer connected in sequence, and the intermediate restoration module and the primary restoration module realize image restoration through a convolution layer, a cross-attention layer and an upsampling layer; the result generation module generates the prediction noise of the tth step through a convolution layer, a cross-attention layer and an upsampling layer, and the upsampling layer adopts bilinear interpolation. The jump connection is used to align and integrate the features of different levels in the feature extraction path and the feature restoration path, thereby improving the effect of detail reconstruction. In addition, the cross attention mechanism runs through the entire network, combining the number of steps t and the low-resolution image The encoding feature map of Embedded into the cross-attention layer of each module, it further enhances the feature modeling capability of the network.

[0044] Furthermore, the CelebA face attribute dataset was used in the training process of the diffusion model and the conditional noise predictor, and 1k images were selected from the dataset as a validation set for quantitative evaluation of the model performance. First, when preprocessing the CelebA face attribute dataset, blurry, format-incompatible, and highly similar problem images were removed. In addition, in order to improve data diversity, a variety of data enhancement methods were used, including rotation, flipping, cropping, and lighting adjustment operations; in order to simulate the requirements of super-resolution tasks, the face images in the CelebA face attribute dataset were processed by the bicubic interpolation degradation operator to generate input data suitable for super-resolution tasks; in order to comprehensively analyze the performance of the model in reconstructing super-resolution images, a variety of image quality evaluation indicators were used. These indicators cover both perceptual quality and non-perceptual quality, including: Peak Signal to Noise Ratio (PSNR↑), Structural Similarity Index Measure (SSIM↑), and Learned Perceptual Image Patch Similarity (LPIPS↓).

[0045] The experimental results are as follows:

[0046] Table 1 shows the comparison results of the facial image super-resolution method based on the diffusion model of residual prediction proposed in the embodiment of the present application with the image super-resolution methods RealESRGAN, BSRGAN, StableSR and DiffBIR in terms of PSNR, SSIM and LPIPS. The facial image super-resolution method based on the diffusion model of residual prediction proposed in the embodiment of the present application performs well in PSNR (21.4872) and SSIM (0.6391), indicating that the image quality and structural information are well preserved. At the same time, the LPIPS value is 0.3093, which is the lowest among all methods, indicating that the facial image super-resolution method based on the diffusion model of residual prediction proposed in the embodiment of the present application performs best in visual quality. Overall, the facial image super-resolution method based on the diffusion model of residual prediction proposed in the embodiment of the present application has obvious advantages in image detail restoration and visual effects.

[0047] Table 1:

[0048]

[0049] refer to Figure 5 , the image shows a low-resolution face image and a reconstruction result image. The left and right sides respectively show a low-resolution face image and a restored high-resolution face image. The left side is a face image after the original image is sampled at a low resolution, and the right side is a high-resolution face image reconstructed using the face image super-resolution method based on the diffusion model of residual prediction proposed in the embodiment of the present application. The reconstruction result of the face image super-resolution method based on the diffusion model of residual prediction proposed in the embodiment of the present application can display details and facial features more clearly. Therefore, the face image super-resolution method based on the diffusion model of residual prediction proposed in the embodiment of the present application can more accurately restore the high-frequency details of the image, reduce blur and distortion, and improve the quality of the reconstructed image. This reconstruction method has a significant improvement in image quality and clarity, especially in the details, the effect is more prominent.

[0050] S3, sampling Gaussian noise from a standard Gaussian distribution as a residual noise image, inputting the low-resolution image to be reconstructed, the residual noise image of each step and the corresponding step length into the trained conditional noise predictor, predicting the predicted noise of each step, and iteratively performing a denoising process of T steps through the trained diffusion model inverse denoising network to obtain a restored residual image; adding the restored residual image to the upsampled low-resolution image to be reconstructed to obtain a restored high-resolution image.

[0051] Specifically, the trained conditional noise predictor and the trained diffusion model are deployed by sampling the residual noise image from a standard Gaussian distribution. , and iterate step by step to get the final restored residual image Finally, the residual image is restored and the upsampled low-resolution image to be reconstructed Add together to get the final restored high-resolution image , as shown below:

[0052] ;

[0053] in, Represents an upsampling operation using bilinear interpolation.

[0054] Further references Figure 6 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a facial image super-resolution device based on a diffusion model of residual prediction, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0055] The embodiment of the present application provides a facial image super-resolution device based on a diffusion model of residual prediction, comprising:

[0056] An image acquisition module 1 is configured to acquire a low-resolution image to be reconstructed;

[0057] The model building module 2 is configured to build a diffusion model and a conditional noise predictor based on residual prediction, wherein the diffusion model includes a forward denoising network and a reverse denoising network. During the training process of the diffusion model and the conditional noise predictor, the residual image of the real high-resolution image and the low-resolution image is input into the forward denoising network, and Gaussian noise of t steps is iteratively added to obtain the residual noise image of the t-th step; the low-resolution image, the residual noise image of the t-th step and the number of steps t are input into the conditional noise predictor to obtain the predicted noise of the t-th step; the residual noise image of the t-th step, the predicted noise of the t-th step and the number of steps t are input into the reverse denoising network to obtain the restored residual image of the t-th step; a mean square error loss function is constructed according to the added Gaussian noise and the predicted noise, and the diffusion model and the conditional noise predictor are trained using the mean square error loss function to obtain a trained diffusion model and a trained conditional noise predictor;

[0058] The restoration module 3 is configured to sample Gaussian noise from a standard Gaussian distribution as a residual noise image, input the low-resolution image to be reconstructed, the residual noise image of each step and the corresponding step size into the trained conditional noise predictor, predict the predicted noise of each step, and iteratively perform a denoising process of T steps through the trained inverse denoising network in the diffusion model to obtain a restored residual image; the restored residual image is added to the upsampled low-resolution image to be reconstructed to obtain a restored high-resolution image.

[0059] Figure 7 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 7 As shown, the electronic device of this embodiment includes: a processor 701 and a memory 702; wherein the memory 702 is used to store computer-executable instructions; the processor 701 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0060] Optionally, the memory 702 may be independent or integrated with the processor 701 .

[0061] When the memory 702 is independently provided, the electronic device further includes a bus 703 for connecting the memory 702 and the processor 701 .

[0062] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 701 executes the computer execution instructions, the above method is implemented.

[0063] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 701, the above method is implemented.

[0064] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0065] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0066] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0067] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 701 to perform some steps of the methods of various embodiments of the present application.

[0068] It should be understood that the processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 701 may be any conventional processor 701, etc. The steps of the method disclosed in the invention may be directly embodied in the execution of the hardware processor 701, or may be executed by a combination of hardware and software modules in the processor 701.

[0069] The memory 702 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0070] The bus 703 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 703 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 703 in the drawings of the present application is not limited to only one bus 703 or one type of bus 703.

[0071] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0072] An exemplary storage medium is coupled to the processor 701, so that the processor 701 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 701. The processor 701 and the storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as ASIC). Of course, the processor 701 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0073] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A facial image super-resolution method based on a diffusion model of residual prediction, characterized in that: The following steps are involved: Acquire a low-resolution image to be reconstructed; A diffusion model and a conditional noise predictor based on residual prediction are constructed, wherein the diffusion model includes a forward denoising network and a reverse denoising network. During the training process of the diffusion model and the conditional noise predictor, a residual image of a real high-resolution image and a low-resolution image is input into the forward denoising network, and Gaussian noise of t steps is iteratively added to obtain a residual noise image of the t-th step; the low-resolution image, the residual noise image of the t-th step and the number of steps t are input into the conditional noise predictor to obtain the predicted noise of the t-th step; the residual noise image of the t-th step, the predicted noise of the t-th step and the number of steps t are input into the reverse denoising network to obtain a restored residual image of the t-th step; a mean square error loss function is constructed according to the added Gaussian noise and the predicted noise, and the diffusion model and the conditional noise predictor are trained using the mean square error loss function to obtain a trained diffusion model and a trained conditional noise predictor; Gaussian noise is sampled from a standard Gaussian distribution as a residual noise image, the low-resolution image to be reconstructed, the residual noise image of each step and the corresponding step length are input into a trained conditional noise predictor, the predicted noise of each step is predicted, and a denoising process of T steps is iteratively performed through a reverse denoising network in the trained diffusion model to obtain a restored residual image; the restored residual image is added to the upsampled low-resolution image to be reconstructed to obtain a restored high-resolution image.

2. The facial image super-resolution method based on the diffusion model of residual prediction according to claim 1, characterized in that: The conditional noise predictor includes a conditional guidance branch and a noise prediction network based on Unet; The low-resolution image is input into the conditional guidance branch to obtain a coding feature map; the coding feature map, the residual noise image of the t-th step and the number of steps t are input into the noise prediction network to obtain the predicted noise of the t-th step.

3. The facial image super-resolution method based on the diffusion model of residual prediction according to claim 2, characterized in that: The conditional guidance branch includes an input layer, a feature extraction module, a global feature aggregation module and an output layer connected in sequence, the input layer includes a convolution layer with a convolution kernel size of 3×3, the low-resolution image is input to the input layer, and a shallow feature coding image is extracted; The feature extraction module includes a plurality of dense residual blocks connected in sequence, the shallow feature coding map is input into the feature extraction module, and a deep feature coding map is extracted; the global feature aggregation module includes a global average pooling layer and a fully connected layer connected in sequence, the deep feature coding map is input into the global feature aggregation module, and a global feature coding map is generated; the output layer includes a convolution layer with a convolution kernel size of 1×1, and the global feature coding map is input into the output layer to generate the coding feature map.

4. The facial image super-resolution method based on the diffusion model of residual prediction according to claim 3 is characterized in that: The dense residual block includes a first dense block and a second dense block, the input features of the dense residual block are input to the first dense block, and a first intermediate feature is output, the input features of the dense residual block are locally residually connected with the first intermediate feature to obtain a second intermediate feature, the second intermediate feature is input to the second dense block, and a third intermediate feature is output, the second intermediate feature is locally residually connected with the third intermediate feature, and is globally residually connected with the input features of the dense residual block to obtain the output features of the dense residual block.

5. The facial image super-resolution method based on the diffusion model of residual prediction according to claim 4 is characterized in that: Both the first dense block and the second dense block adopt a dense structure, and the dense structure includes a first convolutional layer, a second convolutional layer, and a third convolutional layer connected in sequence. The convolution kernel sizes of the first convolutional layer, the second convolutional layer, and the third convolutional layer are all 3×3. The first convolutional layer and the second convolutional layer both use a Leaky ReLU activation function. The input features of the dense structure are concatenated with the output features of the first convolutional layer using dense jump connections to obtain the input features of the second convolutional layer. The input features of the dense structure and the input features of the second convolutional layer are concatenated with the output features of the second convolutional layer using dense jump connections as the input features of the third convolutional layer.

6. The facial image super-resolution method based on the residual prediction diffusion model according to claim 2, characterized in that: The noise prediction network includes a feature extraction path, a feature restoration path and a result generation module; the feature extraction path includes a primary feature extraction module, an intermediate feature extraction module and a high-level feature extraction module connected in sequence; The feature restoration path includes a high-level restoration module, an intermediate restoration module and a primary restoration module connected in sequence; the primary feature extraction module, the intermediate feature extraction module and the high-level feature extraction module all include a convolution layer, a cross-attention layer and a downsampling layer connected in sequence; the high-level restoration module includes a convolution layer, a cross-attention layer and a normalization layer connected in sequence; the primary feature extraction module, the intermediate feature extraction module and the result generation module all include a convolution layer, a cross-attention layer and an upsampling layer connected in sequence; the encoded feature map and the step number t are spliced ​​and input into each cross-attention layer, and the residual noise image of the t-th step is input into the primary feature extraction module to obtain a primary compressed feature map, the primary compressed feature map is input into the intermediate feature extraction module to obtain an intermediate compressed feature map, and the intermediate compressed feature map is input into the high-level feature extraction module to obtain a high-level compressed feature map; The high-level compression feature map is input into the high-level restoration module to obtain a high-level restoration feature map; The high-level compression feature map is spliced ​​with the high-level restoration feature map through a jump connection and input into the intermediate restoration module to obtain an intermediate restoration feature map; the intermediate compression feature map is spliced ​​with the intermediate restoration feature map through a jump connection and input into the primary restoration module to obtain a primary restoration feature map; the primary compression feature map is spliced ​​with the primary restoration feature map through a jump connection and input into the result generation module to obtain the prediction noise of the tth step.

7. A facial image super-resolution device based on a diffusion model of residual prediction, characterized in that: include: An image acquisition module, configured to acquire a low-resolution image to be reconstructed; A model building module is configured to build a diffusion model and a conditional noise predictor based on residual prediction, wherein the diffusion model includes a forward denoising network and a reverse denoising network. During the training process of the diffusion model and the conditional noise predictor, a residual image of a real high-resolution image and a low-resolution image is input into the forward denoising network, and Gaussian noise of t steps is iteratively added to obtain a residual noise image of the t-th step; the low-resolution image, the residual noise image of the t-th step and the number of steps t are input into the conditional noise predictor to obtain the predicted noise of the t-th step; the residual noise image of the t-th step, the predicted noise of the t-th step and the number of steps t are input into the reverse denoising network to obtain a restored residual image of the t-th step; a mean square error loss function is constructed according to the added Gaussian noise and the predicted noise, and the diffusion model and the conditional noise predictor are trained using the mean square error loss function to obtain a trained diffusion model and a trained conditional noise predictor; The restoration module is configured to sample Gaussian noise from a standard Gaussian distribution as a residual noise image, input the low-resolution image to be reconstructed, the residual noise image of each step and the corresponding step length into a trained conditional noise predictor, predict the predicted noise of each step, and iteratively perform a denoising process of T steps through a reverse denoising network in the trained diffusion model to obtain a restored residual image; and add the restored residual image to the upsampled low-resolution image to be reconstructed to obtain a restored high-resolution image.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images