Face super-resolution method and system based on dual guided diffusion model

Abstract

The present invention discloses a face super-resolution method and system based on a dual guided diffusion model. The method comprises the following steps: collecting low-resolution non-frontal face images and high-resolution frontal face images to construct training data pairs; preliminarily restoring the low-resolution non-frontal face images to obtain rough frontal face images; mapping the face images in pixel space to implicit space, enabling a diffusion model to be calculated in the implicit space, pre-training the unconditional diffusion model, using the training results as initialization parameters of the diffusion model, and freezing the encoder of a denoising network; extracting facial prior features from the rough frontal face images, and capturing the spatial and semantic correlations between the facial prior features and denoising features through a hybrid cross-attention mechanism; extracting facial identity coding information and embedding it into a denoising network; initializing a Gaussian noise map, iteratively denoising using the trained diffusion model, mapping the denoising results in the implicit space to the pixel space, and finally reconstructing a high-resolution frontal face image.

Description

Technical Field

[0001] The present invention belongs to the technical field of face super-resolution, and in particular relates to a face super-resolution method based on a dual guided diffusion model. Background Art

[0002] In recent years, with the rapid increase in the number of public transportation hubs, facial recognition technology has been widely used in public security, financial payments, identity authentication and other fields. However, in the process of facial image acquisition, due to the uncontrollable imaging conditions of cameras in public scenes and the influence of external interference factors such as the distance and relative position between the camera and the target, the collected facial images often have image degradation problems such as low resolution and facial posture changes, resulting in low-quality non-frontal facial images. Such facial images not only have blurred details, but also have key facial features distorted or even blocked. Such degradation problems not only affect human visual observation, but also significantly reduce the performance of downstream intelligent algorithms. Therefore, restoring high-quality frontal facial images from low-resolution non-frontal facial images is crucial for many facial analysis tasks.

[0003] To address this challenge, many researchers have adopted face super-resolution techniques to enhance the detail and clarity of facial images, thereby converting low-resolution facial images into high-resolution ones. Improving image quality through super-resolution can effectively improve the accuracy and reliability of facial recognition. Consequently, face super-resolution has been widely used in security surveillance, video communications, digital entertainment, medical imaging, and other fields. In recent years, with the rise of deep learning, neural network-based face super-resolution methods have made significant progress. By learning the mapping relationship between large numbers of low-resolution and high-resolution image pairs, they achieve more accurate face super-resolution.

[0004] Early face super-resolution methods were primarily based on basic convolutional neural networks and generative adversarial networks. Diffusion models have recently attracted widespread attention in low-level vision tasks due to their significant advantages in generating detailed and highly realistic images. Face super-resolution methods based on diffusion models have achieved remarkable performance when processing low-resolution face images that are close to the front. These methods typically consist of two stages: a diffusion stage and a denoising stage. The diffusion stage continuously adds Gaussian noise to the high-resolution image to disrupt the image's data distribution, ultimately degrading the high-resolution image into a pure Gaussian noise image. The denoising stage utilizes a denoising network to iteratively denoise the Gaussian noise image at various noise levels, adjusting it based on the input low-resolution face, ultimately reconstructing a clear high-resolution face image.

[0005] However, mainstream diffusion models still face significant challenges. When processing low-resolution, non-frontal face images, traditional face super-resolution models struggle to effectively recover high-fidelity, high-resolution, frontal faces from these low-resolution, non-frontal faces. Because the goal of face super-resolution reconstruction is highly deterministic and requires generating high-resolution faces of a specific identity, the denoising process is often conditional. Traditional diffusion-based face super-resolution methods simply use low-resolution images as conditional input to constrain the learning of the denoising network to generate realistic, high-resolution faces of a specific identity. However, real-world facial poses are complex and varied, including various horizontal and vertical angles. Low-resolution, non-frontal faces exhibit both rigid and non-rigid degradation, further exacerbating the data distribution discrepancy between the conditional and target face images. Traditional diffusion models struggle to establish an effective connection between the conditional and target face images, resulting in poor controllability of the generated face images. Ultimately, while diffusion models produce realistic face images, they exhibit poor fidelity of facial features and identity. Summary of the Invention

[0006] The present invention is aimed at the problems existing in the prior art and provides a face super-resolution method and system based on a dual guided diffusion model. First, face images with different facial postures and different image resolutions are collected and divided into low-resolution non-frontal face images and high-resolution frontal face images to construct training data pairs. The low-resolution non-frontal face images are then fed into a face preview module to obtain a rough frontal face image. The face images in the pixel space are mapped to the implicit space using a latent space encoder so that the diffusion model is calculated in the implicit space. The unconditional diffusion model is pre-trained on the face dataset and the training results are used as the diffusion model. The initialization parameters of the denoising network are set, and the encoder of the denoising network is frozen. A facial prior guidance module is used to extract facial prior features from the rough frontal face image, and the spatial and semantic correlation between the facial prior features and the denoising features is captured through a hybrid cross-attention mechanism. An identity consistency guidance module is used to extract facial identity encoding information and embed it into the denoising network to constrain the change of facial identity. A Gaussian noise map is initialized, and the noise map is iteratively denoised using a trained diffusion model to obtain a clean face image. The denoising result in the latent space is mapped to the pixel space using a latent space decoder, and finally a high-resolution frontal face image is reconstructed. The method of the present invention can simultaneously achieve face super-resolution reconstruction and face pose correction, restoring a high-fidelity, high-resolution frontal face image from a low-resolution non-frontal face image.

[0007] To achieve the above object, the present invention adopts a technical solution: a face super-resolution method based on a dual guided diffusion model, comprising at least the following steps:

[0008] S1: Collecting facial images with different facial postures and different image resolutions to construct training data pairs of low-resolution non-frontal facial images and high-resolution frontal facial images; the facial images with offset angles are non-frontal facial images;

[0009] S2: Input the low-resolution non-frontal face image obtained in step S1 into the face preview module to capture facial posture changes and obtain a rough frontal face image;

[0010] S3: Use the latent space encoder to map the face image in pixel space to the implicit space, so that the diffusion model is calculated in the implicit space. Pre-train the unconditional diffusion model on the face dataset, use the training results as the initialization parameters of the diffusion model, and freeze the encoder of the denoising network;

[0011] S4: A facial prior guidance module is used to extract facial prior features from rough frontal face images. Facial prior features are used to guide the learning of the denoising network at different levels of feature space, and a hybrid cross-attention mechanism is used to capture the spatial and semantic correlations between facial prior features and denoising features.

[0012] S5: Use the identity encoder in the identity consistency guidance module to extract consistent facial identity encoding features and embed them into the middle layer of the denoising network;

[0013] S6: Initialize the Gaussian noise map and use the trained diffusion model to iteratively denoise the noise map to obtain a clean face image. Use the latent space decoder to map the latent space denoising result to the pixel space, and finally reconstruct a high-resolution frontal face image.

[0014] As an improvement of the present invention, in step S1, the collected facial image is cropped and aligned, and the non-frontal facial image is downsampled to obtain a low-resolution non-frontal facial image.

[0015] As an improvement of the present invention, in the step S2 of obtaining a rough frontal face image, the face preview module is pre-trained using a regression loss function, which consists of two parts: a global regression loss function and a key area regression loss function, wherein the global regression loss is:

[0016] L gr =||I CR -I HF || 2

[0017] Where, I CR =ξ(I LN ),ξ is the face preview module, I LN For low-resolution non-frontal faces, I CRis the predicted rough frontal face, I HF is the label face image;

[0018] The key area regression loss is:

[0019]

[0020] Where R represents the set of key feature regions; r represents a region in R; n is the number of key feature regions; and They represent the pixels of the returned face and the high-quality frontal face in region r respectively.

[0021] As another improvement of the present invention, the hybrid cross attention mechanism in step S4 is composed of a channel cross attention module and a spatial cross attention module;

[0022] The channel cross attention module aggregates the potential vector of the global spatial dimension through global average pooling and global maximum pooling, processes the potential vector using a multi-layer perceptron, and normalizes it using an activation function;

[0023] The spatial cross attention module uses one-dimensional convolution to compress the facial prior feature channel into one dimension and normalizes it using an activation function;

[0024] The output results of the two branches of the channel cross attention module and the spatial cross attention module are multiplied with the denoising network features channel by channel and element by element and fused.

[0025] As another improvement of the present invention, the hybrid cross attention mechanism is specifically:

[0026]

[0027] Where W s represents the two-dimensional activation attention map, f cb represents a stacked structure consisting of a 1×1 convolutional layer and a batch normalization layer, represents the element-wise multiplication operation, Represents the input facial prior branch feature map, Represents the input denoising network feature map, Represents the attention feature map in the spatial crisscross attention branch.

[0028] As another improvement of the present invention, the identity encoder in step S5 is a residual network with the last fully connected classification layer removed, and the identity encoder is trained using a triplet loss function, wherein the triplet loss function is specifically:

[0029]

[0030] Among them, I HF , I CR and I CK denote high-resolution frontal faces, rough high-resolution frontal faces, and false-identity faces, respectively; δ denotes the identity encoder and α denotes a hyperparameter.

[0031] In order to achieve the above-mentioned purpose, the present invention also adopts a technical solution: a face super-resolution system based on a dual guided diffusion model, comprising at least: a pre-trained face preview module, a denoising network, a face prior guidance module, and an identity consistency guidance module.

[0032] The pre-trained face preview module is used to convert low-resolution non-frontal face images into rough frontal face images;

[0033] The denoising network comprises an encoder, an intermediate layer, and a decoder, and is composed of a cascade of non-linear activation residual blocks (NFB) layers, upsampling layers, downsampling layers, and skip connections. A pre-trained facial diffusion prior is constructed by training an unconditional diffusion model on a large facial dataset. The encoder of the denoising network is frozen during training, and the parameters of the intermediate layers and decoder are allowed to be updated during subsequent training.

[0034] The facial prior guidance module is used to extract facial prior features from a rough frontal face image and capture the spatial and semantic correlations between the facial prior features and the denoised features through a hybrid cross-attention mechanism;

[0035] The identity consistency guidance module is used to extract consistent identity coding features and embed the identity coding features into the middle layer of the denoising network. The identity consistency guidance module includes an identity encoder, and the identity encoder is trained using a triplet loss function. The triplet loss minimizes the distance between the identity coding features of the anchor sample and the positive sample in the feature space, and maximizes the distance between the identity coding features of the anchor sample and the negative sample.

[0036] As an improvement of the present invention, the pre-trained facial preview module includes an encoder, an intermediate layer and a decoder, each part is composed of stacked NFBs, and a spatial transformation module (STN) module is embedded after each residual block to capture facial posture changes; the encoder and decoder are connected by a jump connection.

[0037] As another improvement of the present invention, the facial prior guidance module is parallel to the denoising network, and includes an encoder with the same structure as the denoising network. The encoder of the facial prior guidance module is initialized using the pre-trained initial parameters of the denoising network, and the decoder part uses convolutional layers and jump connections to process feature maps of different feature levels.

[0038] Compared with the prior art, the present invention has the following technical advantages and effects:

[0039] (1) The present invention proposes an innovative diffusion model to restore a true high-resolution face image while correcting facial posture. Specifically, a face super-resolution method based on a dual-guided diffusion model is proposed. A coarse-to-fine processing flow is adopted. First, a rough high-resolution frontal face image is obtained using a face preview module. Then, the powerful generation capability of the implicit diffusion model is used to refine the facial details, and finally a high-quality frontal face image is generated.

[0040] (2) The dual guidance structure composed of a facial prior guidance module and an identity consistency guidance module proposed in this paper controls the generation process of the implicit diffusion model, thereby enhancing the fidelity of the face generated by the diffusion model in complex facial degradation scenarios.

[0041] (3) This paper proposes a hybrid cross-attention mechanism to deeply explore the association of spatial and semantic information between facial prior features and denoising network features, thereby enhancing the accuracy of facial details. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is an example of a low-resolution non-frontal face image that is prone to appearing;

[0043] Figure 2 This is a schematic diagram of the structure of the face super-resolution system based on the dual guided diffusion model;

[0044] Figure 3 It is a structural diagram of the face preview module in the system of the present invention;

[0045] Figure 4 It is a schematic diagram of the structure of the implicit diffusion model in the system of the present invention;

[0046] Figure 5 2. It is a schematic diagram of the structure of the hybrid cross attention mechanism in the system of the present invention;

[0047] Figure 6 Schematic diagram of the identity encoder training process of the present invention;

[0048] Figure 7 This is a comparison chart of the reconstruction results of different models;

[0049] Figure 8 This is a comparison of low-resolution face reconstruction results in different poses;

[0050] Figure 9 This is a flowchart of the steps of the face super-resolution method based on the dual guided diffusion model of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0052] Example 1

[0053] In the existing technical field, the quality of face images in the dataset is often poor and contains facial postures at different angles, such as Figure 1 As shown in FIG, the problem of blurred facial details caused by low resolution and missing key facial features caused by posture changes makes it difficult to associate a real face from existing low-resolution non-frontal face images, which will have a great negative impact on downstream face analysis tasks. Therefore, the present invention discloses a face super-resolution method based on a dual guided diffusion model, which includes the following steps:

[0054] Step S1: Collect face images with different facial poses and different image resolutions, and construct training data pairs of low-resolution non-frontal face images and high-resolution frontal face images.

[0055] Facial images with different facial postures are collected, and facial images with no offset angles are defined as frontal facial images, while facial images with offset angles are defined as non-frontal facial images. The collected faces are cropped and aligned, and non-frontal faces are downsampled to obtain low-resolution non-frontal facial images, and the high-resolution frontal facial images of the corresponding identities are used as label images.

[0056] Step S2: Pre-train the face preview module, feed the low-resolution non-frontal face image into the face preview module to obtain a rough frontal face image.

[0057] The facial preview module adopts an encoder-decoder architecture, which consists of multiple cascaded residual blocks without non-linear activation. A spatial transformation module is used after each residual block to capture facial posture changes. The encoder uses downsampling operations to reduce the size of the feature map, and the decoder uses upsampling operations to expand the size of the feature map, and uses jump connections to fuse feature maps at different levels.

[0058] The face preview module is pre-trained using a regression loss function to initially eliminate the degradation problem in non-frontal low-resolution face images. The regression loss function consists of two parts: global regression and key area regression loss function, thereby minimizing the difference between the predicted face image and the high-resolution frontal face image, and strengthening the learning of key facial feature areas.

[0059] Step S3: Use the latent space encoder to map the face image in the pixel space to the implicit space, so that the diffusion model is calculated in the implicit space. Pre-train the unconditional diffusion model on the face dataset, use the training results as the initialization parameters of the diffusion model, and freeze the encoder of the denoising network.

[0060] The denoising network consists of an encoder, an intermediate layer and a decoder. These modules are composed of cascaded nonlinear activation residual blocks, upsampling layers, downsampling layers and jump connections. An unconditional diffusion model is trained on a face dataset to generate a facial diffusion prior. The diffusion prior is used to initialize the denoising network, and the encoder of the denoising network is frozen. The encoder parameters remain unchanged in the subsequent training process.

[0061] Step S4: A facial prior guidance module is used to extract facial prior features from the rough frontal face image, and the facial prior features are used to guide the learning of the denoising network at different levels of feature space. The spatial and semantic correlations between the facial prior features and the denoising features are captured through a hybrid cross-attention mechanism.

[0062] The facial prior guidance module extracts facial prior features from the rough frontal face in the implicit space. This module is a branch parallel to the denoising network and contains an encoder with the same structure as the denoising network. The encoder of this module is initialized using the pre-trained denoising network initial parameters to transfer facial prior knowledge. The decoder part uses convolutional layers and jump connections to process feature maps at different feature levels to align them with the feature dimensions of the denoising network.

[0063] The hybrid cross-attention mechanism consists of a channel cross-attention module and a spatial cross-attention module; the channel cross-attention module automatically establishes connections between facial prior features and denoising feature channels to explore valuable feature channels, aggregates the potential vector of the global spatial dimension through global average pooling and global maximum pooling, processes the potential vector using a multi-layer perceptron, and standardizes it using an activation function; the spatial cross-attention module adaptively activates effective facial feature areas, captures the positional dependency between facial prior features and denoising features, compresses the facial prior feature channels to one dimension using one-dimensional convolution, and standardizes them using an activation function; finally, the output results of the two branches are multiplied and fused with the denoising network features channel by channel and element by element respectively.

[0064] Step S5: The identity consistency guidance module is used to extract facial identity encoding information and embed it into the denoising network to constrain the changes in facial identity.

[0065] The identity consistency guidance module uses an identity encoder to extract consistent identity encoding features and embeds these identity encoding features into the middle layer of the denoising network. Specifically, the identity encoder is a residual network with the last fully connected classification layer removed. It is trained using a triplet loss function, using high-resolution frontal faces, rough frontal faces, and faces of other identities as anchor samples, positive samples, and negative samples, respectively. The triplet loss minimizes the distance between the identity encoding features of anchor samples and positive samples in feature space, and maximizes the distance between the identity encoding features of anchor samples and negative samples.

[0066] Step S6: Initialize the Gaussian noise map, use the trained diffusion model to iteratively denoise the noise map to obtain a clean face image, use the latent space decoder to map the latent space denoising result to the pixel space, and finally reconstruct a high-resolution frontal face image.

[0067] Initialize the Gaussian noise map, use the trained denoising network to gradually predict the noise added in the current step, use the implicit sampling algorithm to iteratively denoise the noise map, use the latent space decoder to map the implicit space denoising result to the pixel space, and finally reconstruct a high-resolution frontal face image.

[0068] Example 2

[0069] This embodiment is similar to Figure 1 The image distribution shown in , a total of 32,000 face images were trained on 250 subjects, of which the first 200 subjects were used for training and the remaining 50 subjects were used for testing. The structure of the system of the present invention is as follows Figure 2 As shown in the figure, the target face is gradually recovered from the low-resolution non-frontal face in a coarse-to-fine manner. First, the face preview module is used to recover the target face from the low-resolution non-frontal face (I LN ) to obtain a rough high-resolution frontal face (I CR ), and then use the powerful generation ability of the implicit diffusion model to refine facial details, and finally generate a realistic high-resolution frontal face image (I HF ).

[0070] A face super-resolution method based on a dual guided diffusion model, such as Figure 9 As shown, the specific method includes the following steps:

[0071] Step S1: Collect and process data

[0072] Facial images (128×128 pixels) in different facial postures are collected, including 13 different angles (-90°, -75°, -60°, -45°, -30°, -15°, 0°, +15°, +30°, +45°, +60°, +75°, +90°). Facial images with offset angle postures (non-0°) are defined as non-frontal facial images, and facial images with 0° posture of the corresponding identity are defined as frontal facial images. The collected faces are cropped and aligned, and the non-frontal facial images are downsampled 8 times to obtain non-frontal low-resolution facial images of 16×16 pixels. The high-resolution frontal facial images of the corresponding identities are used as label images.

[0073] Step S2: Pre-train the face preview module, feed the low-resolution non-frontal face image into the face preview module to obtain a rough frontal face image.

[0074] The overall structure of the face preview module is as follows Figure 3 As shown in Figure 1, the face preview module consists of three parts: encoder, intermediate layer, and decoder. Each part consists of stacked non-linear activation residual blocks (NFBs), and each residual block is embedded in a spatial transformer (STN) module to capture facial pose changes. The NFB layer introduces layer normalization to enhance the training stability of the model and removes non-linear units by replacing the classic activation function with simplified channel attention and simple gate switches. The number of stacked NFB layers in the encoder, intermediate layer, and decoder are [2, 2, 4, 8], [8], and [2, 2, 2, 2], respectively. The encoder performs four 2× downsampling operations and the decoder performs four 2× upsampling operations to ensure that the input and output scales are the same. The encoder and decoder are connected by skip connections to achieve feature fusion.

[0075] The face preview module uses a regression loss function for pre-training. By minimizing the difference between the output face image and the high-resolution frontal face image, it initially eliminates the degradation problem in the non-frontal low-resolution face image. This regression loss function consists of two parts: global regression and key area regression loss function. The global regression loss is as follows:

[0076] L gr =||I CR -I HF || 2 (1)

[0077] Where, I CR =ξ(I LN ),ξ is the face preview module, I LN For low-resolution non-frontal faces, I CR is the predicted rough frontal face, I HFis the label face image. The key area regression loss is as follows:

[0078]

[0079] Where R represents the set of key feature regions, including the locations of eyes, nose, and mouth, whose coordinates are estimated based on five predefined facial landmarks; r represents a region in R; n is the number of key feature regions; and They represent the pixels of the returned face and the high-quality frontal face in region r respectively.

[0080] The overall loss function of the face preview module is L cr =L kr +L gr , the loss function is used for training for 24 cycles with a batch size of 8, the adaptive momentum estimation optimizer is used for optimization, and the learning rate is set to 5e-4.

[0081] Step S3: Use the latent space encoder to map the face image in the pixel space to the latent space. The structure of the denoising network is as follows: Figure 4 As shown in the "Denoising Network" in the figure, it includes an encoder, intermediate layers, and a decoder. These modules are composed of cascaded NFB layers, upsampling layers, downsampling layers, and skip connections. The pre-trained facial diffusion prior is constructed by training an unconditional diffusion model on a large facial dataset. The encoder of the denoising network is frozen during training to ensure that the feature encoder retains the diffusion prior knowledge of the true facial contour and texture details. The intermediate layer and decoder parameters are allowed to update during subsequent training to cope with complex facial condition changes.

[0082] Step S4: Use the facial prior guidance module to extract facial prior features from the rough frontal face image, and use the facial prior features to guide the denoising network learning at different feature space levels. The structure of the facial prior guidance module is as follows: Figure 4 As shown in the "Facial Prior Guidance Module".

[0083] The facial prior guidance module extracts facial guidance features at different levels from the rough frontal face in the latent space. It is a branch running parallel to the denoising network and includes an encoder with the same structure as the denoising network. The encoder of this module is initialized using the initial parameters of the pretrained denoising network to transfer facial prior knowledge. The decoder uses 1×1 convolutional layers and skip connections to align the encoder feature maps at different feature levels with the feature dimensions of the denoising network.

[0084] Then, the spatial and semantic correlations between the facial prior features and the denoised features are captured through a hybrid cross-attention mechanism; the overall structure of the hybrid cross-attention mechanism is as follows: Figure 5As shown, the module is inserted between the middle layer and multiple feature layers of the decoder. The calculation process is as follows:

[0085]

[0086] Where, Represents the input denoising network feature map, and Represent the attention feature maps in the channel cross attention branch and the spatial cross attention branch, W fuse Represents the 3×3 convolution operation on the fused feature map, and i represents the level of the feature map.

[0087] Among them, the channel cross attention aggregates the global average pooling (GAP) (i.e., f in Formula 4 gap ) and global maximum pooling (GMP) (i.e., f in Formula 4 gmp ) to form a potential vector in the global spatial dimension, and then, a multi-layer perceptron containing two fully connected layers (FC) is used (i.e., f in Formula 4 mlp ) processes the latent vector and uses the Sigmoid function to map the value to the range of [0,1]. The channel-level attention information extraction formula is as follows:

[0088]

[0089] Where W c Represents a one-dimensional channel attention map; and Represent element-wise addition and element-wise multiplication operations, respectively. Represents the input facial prior branch feature map.

[0090] The spatial cross attention branch is designed to adaptively activate the effective facial feature area and capture the position dependency between the facial prior features and the denoising features. First, the convolutional layer is used to transform The feature channel is compressed into one dimension, and then the Sigmoid activation function is used to map it to the range of [0,1]. Finally, this value is combined with Multiply them together to suppress the response of irrelevant areas and guide the model to focus on key facial areas. The formula is as follows:

[0091]

[0092] Where W s represents the two-dimensional activation attention map, f cb represents a stacked structure consisting of a 1×1 convolutional layer and a batch normalization layer, Represents an element-wise multiplication operation.

[0093] Step S5: Use the identity consistency guidance module to extract facial identity encoding information, and embed the encoding information into the denoising network to constrain facial identity changes. The process is as follows: Figure 4 The specific process is as follows:

[0094] The identity consistency guidance module uses the identity encoder to extract the identity code of the identity consistency and embeds the identity code into the middle layer of the denoising network. The identity encoder is a residual network with the last fully connected classification layer removed. The identity encoder is trained using a triplet loss function. The training process of the identity encoder is as follows: Figure 6 As shown, the triplet loss function formula is as follows:

[0095]

[0096] In the formula, I HF , I CR and I CR They represent high-resolution frontal faces, rough high-resolution frontal faces, and false identity faces, which are used as anchor samples, positive samples, and negative samples, respectively. δ represents the identity encoder, and α represents the hyperparameter. The ternary loss is achieved by minimizing the I HF and I CR The distance between them, and maximizing the I HF and I CK The distance between them is trained to ensure that the embedding of faces with the same identity is as close to the real face as possible, while the embedding of faces with different identities is as different as possible. CR The encoding features representing the identity are extracted from the network, and the identity embedding is processed by 1×1 convolution to reshape the dimension so that it is aligned with the middle layer dimension of the denoising network. Finally, it is input to the middle layer of the denoising network by element-by-element addition to constrain the identity changes in the generation process.

[0097] Step S6: Initialize the Gaussian noise map z~N(0,1), use the trained denoising network to gradually predict the noise added in the current step, use the implicit sampling algorithm to calculate the denoising result of the previous step based on the noise of the current step, continuously iterate the denoising of the noise map to obtain the final denoising result, use the latent space decoder to map the denoising result to the pixel space, and finally reconstruct a high-resolution frontal face image.

[0098] In the overall network training and inference process of this embodiment, first, during the training process, the face preview module is pre-trained using formulas (1) and (2), and a rough frontal face image is obtained; the facial pixel space is mapped to the latent space using a latent space encoder, so that the diffusion and denoising processes are performed in the latent space, and then the denoising network and identity encoder of the diffusion model are pre-trained and the pre-trained parameters are loaded, and the diffusion model is trained using the L2 distance as the loss function, as shown in the following formula:

[0099]

[0100] Where t is the time step, ∈ is sampled from a Gaussian distribution, ∈ θ represents the diffusion model, θ represents the model parameters, α i is a hyperparameter that changes with t, I CR is a rough frontal face, ε represents the latent space encoder, and δ represents the identity encoder. The training parameters are as follows: Adam optimizer is used for 24 epochs, batch size is 8, and learning rate is set to 1e-4.

[0101] During the inference process, the trained denoising network is used to extract the noise image z t Estimate the noise-free face image z0 in

[15] . First, load the pre-trained face preview module to obtain a rough frontal face image. Then, use the rough frontal face image to construct facial prior features and identity constraint features. Next, load the trained diffusion model parameters to perform iterative denoising in the latent space. Calculate the noise added in the previous step based on the current step noise. Continuously iterate to finally obtain the clean image z0. The sampling formula is as follows:

[0102]

[0103] Among them, z t is noise sampled from a standard normal distribution, is a hyperparameter that changes over time and is calculated in the same way as formula (9), where t is the time step, ∈ θ represents the diffusion model, I CR is a rough frontal face, ε represents the latent space encoder, δ represents the identity encoder, and σ is the variance, which defaults to 0. 50 iterations are performed using the sampling formula (Formula 10) to obtain z0, and finally z0 is mapped to the pixel space through the latent space decoder to obtain a high-resolution frontal face.

[0104] Test Case

[0105] The proposed method is compared with low-resolution face reconstruction using different models and different poses, as shown in the following example. Figure 7 and Figure 8 shown. Figure 7is a comparison chart of reconstruction results of different models, where Figure 7 (a) is the input low-resolution non-frontal face image, Figure 7 (b) is a schematic diagram of the results of Real-ESRGAN+DRGAN. Figure 7 (c) is a schematic diagram of the results of DRGAN+Real-ESRGAN. Figure 7 (d) is a schematic diagram of the results of SR3+DAGAN. Figure 7 (e) is a schematic diagram of the results of DAGAN+SR3. Figure 7 (f) is a schematic diagram of the TANN results. Figure 7 (g) is a schematic diagram of the Refusion result. Figure 7 (h) is a schematic diagram of the DiffBIR results. Figure 7 (i) is a schematic diagram of the results of the method of the present invention, Figure 7 (j) is the label image. , Figure 7 It can be clearly seen that the reconstructed image of the method in this case is clearer, more accurate, and the effect is more obvious. The present invention can accurately restore a high-resolution frontal face image from a low-resolution non-frontal face image.

[0106] Figure 8 is a comparison of low-resolution face reconstruction results in different poses, where Figure 8 (a) The input is a low-resolution non-frontal face image. Figure 8 (b) is a schematic diagram of the DiffBIR results. Figure 8 (c) is a schematic diagram of the result of the present invention, and (d) is a label image. Figure 8 From the visualization of the restoration results in different postures, it can be seen that the method of the present invention has good fidelity and identity consistency for faces in different postures.

[0107] In summary, the method and system of the present invention can simultaneously achieve face super-resolution reconstruction and face posture correction, and restore high-fidelity high-resolution frontal face images from low-resolution non-frontal face images.

[0108] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. Face super-resolution method based on dual guided diffusion model, characterized by , including at least the following steps: S1: Collect facial images with different facial poses and image resolutions, and construct training data pairs of low-resolution non-frontal facial images and high-resolution frontal facial images; facial images with offset angles are non-frontal facial images; S2: Input the low-resolution non-frontal face image obtained in step S1 into the face preview module to capture facial posture changes and obtain a rough frontal face image; S3: Use the latent space encoder to map the face image in pixel space to the implicit space, so that the diffusion model is calculated in the implicit space. Pre-train the unconditional diffusion model on the face dataset, use the training results as the initialization parameters of the diffusion model, and freeze the encoder of the denoising network. S4: A facial prior guidance module is used to extract facial prior features from rough frontal face images. Facial prior features are used to guide the learning of the denoising network at different levels of feature space, and a hybrid cross-attention mechanism is used to capture the spatial and semantic correlations between facial prior features and denoising features. S5: Use the identity encoder in the identity consistency guidance module to extract consistent facial identity encoding features and embed them into the middle layer of the denoising network; S6: Initialize the Gaussian noise map and use the trained diffusion model to iteratively denoise the noise map to obtain a clean face image. Use the latent space decoder to map the latent space denoising result to the pixel space, and finally reconstruct a high-resolution frontal face image.

2. The face super-resolution method based on the dual guided diffusion model according to claim 1, wherein: In step S1, the collected facial image is cropped and aligned, and the non-frontal facial image is down-sampled to obtain a low-resolution non-frontal facial image.

3. The face super-resolution method based on the dual guided diffusion model according to claim 1, wherein: In the step S2 of obtaining a rough frontal face image, the face preview module is pre-trained using a regression loss function. The regression loss function consists of two parts: a global regression loss function and a key area regression loss function. The global regression loss is: L gr =||I CR -I HF || 2 Where, I CR =ξ(I LN ),ξ is the face preview module, I LN For low-resolution non-frontal faces, I CR is the predicted rough frontal face, I HF is the label face image; The key area regression loss is: Where R represents the set of key feature regions; r represents a region in R; n is the number of key feature regions; and They represent the pixels of the returned face and the high-quality frontal face in region r respectively.

4. The face super-resolution method based on the dual guided diffusion model according to claim 1, wherein: The hybrid cross attention mechanism in step S4 consists of a channel cross attention module and a spatial cross attention module; The channel cross attention module aggregates the potential vector of the global spatial dimension through global average pooling and global maximum pooling, processes the potential vector using a multi-layer perceptron, and normalizes it using an activation function; The spatial cross-attention module uses one-dimensional convolution to compress the facial prior feature channel to one dimension and standardizes it using an activation function; the output results of the two branches of the channel cross-attention module and the spatial cross-attention module are multiplied with the denoising network features channel by channel and element by element and fused.

5. The face super-resolution method based on the dual guided diffusion model according to claim 4, characterized in that: The hybrid cross attention mechanism is specifically: Where W s represents the two-dimensional activation attention map, f cb represents a stacked structure consisting of a 1×1 convolutional layer and a batch normalization layer, represents the element-wise multiplication operation, Represents the input facial prior branch feature map, Represents the input denoising network feature map, Represents the attention feature map in the spatial crisscross attention branch.

6. The face super-resolution method based on the dual guided diffusion model according to claim 5, characterized in that: In step S5, the identity encoder is a residual network with the last fully connected classification layer removed. The identity encoder is trained using a triplet loss function. Specifically, the triplet loss function is: Among them, I HF , I CR and I CK denote high-resolution frontal faces, rough high-resolution frontal faces, and false-identity faces, respectively; δ denotes the identity encoder and α denotes a hyperparameter.

7. Face super-resolution system based on dual guided diffusion model, characterized by: At least: Pre-trained face preview module, denoising network, face prior guidance module, identity consistency guidance module, The pre-trained face preview module is used to convert low-resolution non-frontal face images into rough frontal face images; The denoising network comprises an encoder, an intermediate layer, and a decoder, and is composed of a cascade of non-linear activation residual blocks (NFB) layers, upsampling layers, downsampling layers, and skip connections. A pre-trained facial diffusion prior is constructed by training an unconditional diffusion model on a large facial dataset. The encoder of the denoising network is frozen during training, and the parameters of the intermediate layers and decoder are allowed to be updated during subsequent training. The facial prior guidance module is used to extract facial prior features from a rough frontal face image and capture the spatial and semantic correlations between the facial prior features and the denoised features through a hybrid cross-attention mechanism; The identity consistency guidance module is used to extract consistent identity coding features and embed the identity coding features into the middle layer of the denoising network. The identity consistency guidance module includes an identity encoder, and the identity encoder is trained using a triplet loss function. The triplet loss minimizes the distance between the identity coding features of the anchor sample and the positive sample in the feature space, and maximizes the distance between the identity coding features of the anchor sample and the negative sample.

8. The face super-resolution system based on the dual guided diffusion model according to claim 7, characterized in that: The pre-trained face preview module includes an encoder, an intermediate layer and a decoder, each of which is composed of stacked NFBs. A spatial transformer (STN) module is embedded after each residual block to capture facial posture changes; the encoder and decoder are connected by a skip connection.

9. The face super-resolution system based on the dual guided diffusion model according to claim 8, characterized in that: The facial prior guidance module is parallel to the denoising network and includes an encoder with the same structure as the denoising network. The encoder of the facial prior guidance module is initialized using the pre-trained initial parameters of the denoising network. The decoder part uses convolutional layers and jump connections to process feature maps at different feature levels.

Images