Face super-resolution method and system based on dual guide diffusion model

Abstract

The invention discloses a face super-resolution method and system based on a dual guide diffusion model, and the method comprises the steps: collecting low-resolution non-front face images and high-resolution front face images, and constructing a training data pair; after the low-resolution non-front face image is preliminarily restored, a rough front face image is obtained; mapping a face image of a pixel space to an implicit space, enabling a diffusion model to calculate in the implicit space, pre-training an unconditional diffusion model, taking a training result as an initialization parameter of the diffusion model, and freezing an encoder of a denoising network; extracting face priori features from the rough front face image, and capturing spatial and semantic correlation between the face priori features and de-noising features through a hybrid cross attention mechanism; face identity coding information is extracted, and a denoising network is embedded; and initializing a Gaussian noise map, carrying out continuous iterative denoising by using the trained diffusion model, mapping a denoising result in an implicit space to a pixel space, and finally reconstructing a high-resolution front face image.

Description

Technical Field

[0001] The present invention belongs to the technical field of human face super-resolution, and in particular relates to a human face super-resolution method based on a dual guided diffusion model. Background Art

[0002] In recent years, with the rapid increase in the number of public transportation hubs, face recognition technology has been widely used in the fields of public security, financial payment, identity authentication, etc. However, in the process of acquiring face images, due to the uncontrollable imaging conditions of cameras in public scenes and the influence of external interference factors such as the distance and relative position between the camera and the target, the collected face images often have image degradation problems such as low resolution and facial posture changes, resulting in low-quality non-frontal face images. Such face images not only have blurred details, but also have distorted or even blocked key facial features. Such degradation problems not only affect human visual observation, but also greatly reduce the performance of downstream intelligent algorithms. Therefore, restoring high-quality frontal face images from low-resolution non-frontal face images is crucial for many face analysis tasks.

[0003] In response to this challenge, many researchers use face super-resolution technology to improve the level of detail and clarity of facial images, thereby converting low-resolution face images into high-resolution face images. Improving image quality through super-resolution technology can effectively improve the accuracy and reliability of face recognition. Therefore, face super-resolution technology is widely used in security monitoring, video communication, digital entertainment, medical imaging and other fields. In recent years, with the rise of deep learning, face super-resolution methods based on neural networks have made significant progress. By learning the mapping relationship between a large number of low-resolution and high-resolution image pairs, more accurate face super-resolution can be achieved.

[0004] Early face super-resolution methods were mainly based on basic convolutional neural networks and generative adversarial networks. Due to the significant advantages of diffusion models in generating details and highly realistic images, diffusion models have recently attracted widespread attention in low-level visual tasks. Face super-resolution methods based on diffusion models have achieved remarkable performance in processing low-resolution face images that are close to the front. This type of method usually consists of two stages: diffusion stage and denoising stage. The diffusion stage is to continuously add Gaussian noise to the high-resolution image to destroy the data distribution of the image, and finally degenerate the high-resolution image into a pure Gaussian noise image; the denoising stage is to use the denoising network to iteratively denoise the Gaussian noise image at various noise levels, and adjust it according to the input low-resolution face, and finally reconstruct a clear high-resolution face image.

[0005] However, the mainstream diffusion model still faces severe challenges. When processing low-resolution non-frontal face images, traditional face super-resolution models are difficult to effectively restore high-fidelity high-resolution frontal faces from low-resolution non-frontal faces. Since the goal of face super-resolution reconstruction is highly deterministic and requires the generation of high-resolution faces of specific identities, the denoising process is often controlled by conditions. Traditional face super-resolution methods based on diffusion models simply use low-resolution images as conditional inputs to constrain the learning of denoising networks to generate real high-resolution faces of specific identities. However, facial postures in real scenes are complex and changeable, including various horizontal angles and pitch angles. Low-resolution non-frontal faces contain both rigid degradation and non-rigid degradation, which further exacerbates the data distribution difference between conditional faces and target face images. Traditional diffusion models find it difficult to establish an effective connection between conditional face images and corresponding target face images, resulting in poor controllability of generated face image results. In the end, although the face images generated by the diffusion model have a better sense of reality, the fidelity of facial features and identities is poor. Summary of the invention

[0006] The present invention aims at the problems existing in the prior art and provides a face super-resolution method and system based on a dual guided diffusion model. First, face images with different facial postures and different image resolutions are collected, and they are divided into low-resolution non-frontal face images and high-resolution frontal face images to construct training data pairs. Then, the low-resolution non-frontal face images are sent to a face preview module to obtain a rough frontal face image. The face image in the pixel space is mapped to the implicit space using a latent space encoder, so that the diffusion model is calculated in the implicit space. The unconditional diffusion model is pre-trained on the face data set, and the training results are used as the diffusion model. The initialization parameters of the denoising network are set, and the encoder of the denoising network is frozen; the facial prior guidance module is used to extract facial prior features from the rough frontal face image, and the spatial and semantic correlation between the facial prior features and the denoising features is captured through a hybrid cross-attention mechanism; the identity consistency guidance module is used to extract the facial identity encoding information, and it is embedded into the denoising network to constrain the change of facial identity; the Gaussian noise map is initialized, and the noise map is iteratively denoised using the trained diffusion model to obtain a clean face image, and the denoising result in the implicit space is mapped to the pixel space using the latent space decoder, and finally a high-resolution frontal face image is reconstructed. The method of the present invention can simultaneously realize super-resolution reconstruction of the face and correction of the face posture, and restore a high-fidelity high-resolution frontal face image from a low-resolution non-frontal face image.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is: a face super-resolution method based on a dual guided diffusion model, comprising at least the following steps:

[0008] S1: Collecting facial images with different facial postures and different image resolutions, and constructing training data pairs of low-resolution non-frontal facial images and high-resolution frontal facial images; the facial images with offset angles are non-frontal facial images;

[0009] S2: inputting the low-resolution non-frontal face image obtained in step S1 into the face preview module to capture the facial posture changes and obtain a rough frontal face image;

[0010] S3: Use the latent space encoder to map the face image in the pixel space to the implicit space, so that the diffusion model is calculated in the implicit space, pre-train the unconditional diffusion model on the face dataset, use the training results as the initialization parameters of the diffusion model, and freeze the encoder of the denoising network;

[0011] S4: A facial prior guidance module is used to extract facial prior features from rough frontal face images, and the facial prior features are used to guide the learning of the denoising network at different levels of feature space. The spatial and semantic correlations between facial prior features and denoising features are captured through a hybrid cross-attention mechanism.

[0012] S5: Use the identity encoder in the identity consistency guidance module to extract consistent facial identity encoding features and embed them into the middle layer of the denoising network;

[0013] S6: Initialize the Gaussian noise map, use the trained diffusion model to iteratively denoise the noise map to obtain a clean face image, use the latent space decoder to map the latent space denoising result to the pixel space, and finally reconstruct a high-resolution frontal face image.

[0014] As an improvement of the present invention, in step S1, the collected facial image is cropped and aligned, and the non-frontal facial image is downsampled to obtain a low-resolution non-frontal facial image.

[0015] As an improvement of the present invention, in the step S2 of obtaining a rough frontal face image, the face preview module is pre-trained using a regression loss function, and the regression loss function consists of two parts: a global regression loss function and a key area regression loss function, wherein the global regression loss is:

[0016] L gr =||I CR -I HF || 2

[0017] In the formula, I CR =ξ(I LN ),ξ is the face preview module, I LN For low-resolution non-frontal faces, I CRis the predicted rough frontal face, I HF is a label face image;

[0018] The key area regression loss is:

[0019]

[0020] In the formula, R represents the set of key feature regions; r represents a region in R; n is the number of key feature regions; and They represent the pixels of the returned face and the high-quality frontal face in region r respectively.

[0021] As another improvement of the present invention, the hybrid cross attention mechanism in step S4 is composed of a channel cross attention module and a spatial cross attention module;

[0022] The channel cross attention module aggregates the potential vector of the global spatial dimension through global average pooling and global maximum pooling, processes the potential vector using a multi-layer perceptron, and normalizes it using an activation function;

[0023] The spatial cross attention module uses one-dimensional convolution to compress the facial prior feature channel into one dimension and standardizes it using an activation function;

[0024] The output results of the two branches of the channel cross-attention module and the spatial cross-attention module are multiplied and fused with the denoising network features channel by channel and element by element.

[0025] As another improvement of the present invention, the hybrid cross attention mechanism is specifically:

[0026]

[0027] Where W s represents the two-dimensional activation attention map, f cb represents a stacked structure consisting of a 1×1 convolutional layer and a batch normalization layer. represents an element-wise multiplication operation, represents the input facial prior branch feature map, Represents the input denoising network feature map, Represents the attention feature map in the spatial criss-cross attention branch.

[0028] As another improvement of the present invention, the identity encoder in step S5 is a residual network with the last fully connected classification layer removed, and the identity encoder is trained using a triplet loss function, and the triplet loss function is specifically:

[0029]

[0030] Among them, I HF ,I CR and I CK denote high-resolution frontal faces, rough high-resolution frontal faces, and false-identity faces, respectively, δ denotes the identity encoder, and α denotes a hyperparameter.

[0031] In order to achieve the above object, the present invention also adopts a technical solution: a face super-resolution system based on a dual guided diffusion model at least includes: a pre-trained face preview module, a denoising network, a face prior guidance module, and an identity consistency guidance module.

[0032] The pre-trained face preview module is used to convert a low-resolution non-frontal face image into a rough frontal face image;

[0033] The denoising network comprises an encoder, an intermediate layer and a decoder, and is composed of a cascaded non-linear activation residual block (NFB) layer, an upsampling layer, a downsampling layer and a jump connection. The pre-trained facial diffusion prior is constructed by training an unconditional diffusion model on a large facial dataset. The encoder of the denoising network is frozen during training, and the parameters of the intermediate layer and the decoder are allowed to be updated in the subsequent training process.

[0034] The facial prior guidance module is used to extract facial prior features from a rough frontal face image and capture the spatial and semantic correlations between the facial prior features and the denoising features through a hybrid cross-attention mechanism;

[0035] The identity consistency guidance module is used to extract consistent identity coding features and embed the identity coding features into the middle layer of the denoising network. The identity consistency guidance module includes an identity encoder, and the identity encoder is trained using a triplet loss function. The triplet loss minimizes the distance between the anchor point sample identity coding features and the positive sample identity coding features in the feature space, and maximizes the distance between the anchor point sample identity coding features and the negative sample identity coding features.

[0036] As an improvement of the present invention, the pre-trained facial preview module includes an encoder, an intermediate layer and a decoder, each of which is composed of stacked NFBs, and a spatial transformation module (STN) module is embedded after each residual block to capture facial posture changes; the encoder and the decoder are connected by a jump connection.

[0037] As another improvement of the present invention, the facial prior guidance module is parallel to the denoising network, and includes an encoder with the same structure as the denoising network. The encoder of the facial prior guidance module is initialized using pre-trained denoising network initial parameters, and the decoder part uses convolutional layers and jump connections to process feature maps of different feature levels.

[0038] Compared with the prior art, the present invention has the following technical advantages and effects:

[0039] (1) The present invention proposes an innovative diffusion model to restore a true high-resolution face image while correcting facial posture. Specifically, a face super-resolution method based on a dual guided diffusion model is proposed. A coarse-to-fine processing flow is adopted. First, a rough high-resolution frontal face is obtained by using a face preview module. Then, the powerful generation capability of the implicit diffusion model is used to refine the facial details, and finally a high-quality frontal face image is generated.

[0040] (2) The dual guidance structure composed of a facial prior guidance module and an identity consistency guidance module proposed in the present invention controls the generation process of the implicit diffusion model, thereby enhancing the fidelity of the face generated by the diffusion model in complex facial degradation scenarios.

[0041] (3) The present invention proposes a hybrid cross-attention mechanism to deeply explore the association between the spatial and semantic information between facial prior features and denoising network features, thereby enhancing the accuracy of facial details. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is an example of a low-resolution non-frontal face image that is prone to appearing;

[0043] Figure 2 It is a schematic diagram of the structure of the face super-resolution system based on the dual guided diffusion model;

[0044] Figure 3 It is a structural schematic diagram of a face preview module in the system of the present invention;

[0045] Figure 4 It is a schematic diagram of the structure of the implicit diffusion model in the system of the present invention;

[0046] Figure 5 It is a schematic diagram of the structure of the hybrid cross attention mechanism in the system of the present invention;

[0047] Figure 6 It is a schematic diagram of the identity encoder training process of the present invention;

[0048] Figure 7 This is a comparison chart of the reconstruction results of different models;

[0049] Figure 8 This is a comparison of low-resolution face reconstruction results in different poses;

[0050] Fig. 9 It is a flowchart of the steps of the face super-resolution method based on the dual guided diffusion model of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0052] Example 1

[0053] In the existing technical field, the quality of face images in the dataset is often poor and contains facial postures at different angles, such as Figure 1 As shown in the figure, the problem of blurred facial details caused by low resolution and missing facial key features caused by posture changes makes it difficult to associate a real face from the existing low-resolution non-frontal face image, which will have a great negative impact on the downstream face analysis task. Therefore, the present invention discloses a face super-resolution method based on a dual guided diffusion model, which includes the following steps:

[0054] Step S1: collect face images with different facial postures and different image resolutions, and construct training data pairs of low-resolution non-frontal face images and high-resolution frontal face images.

[0055] Facial images with different facial postures are collected, and facial images with no offset angles are defined as frontal facial images, and facial images with offset angles are defined as non-frontal facial images. The collected faces are cropped and aligned, and non-frontal faces are downsampled to obtain low-resolution non-frontal facial images, and the high-resolution frontal facial images of the corresponding identities are used as label images.

[0056] Step S2: Pre-train the face preview module, send the low-resolution non-frontal face image to the face preview module to obtain a rough frontal face image.

[0057] The face preview module adopts an encoder-decoder architecture, which consists of multiple cascaded residual blocks without non-linear activation. A spatial transformation module is used after each residual block to capture facial posture changes. The encoder uses downsampling operations to reduce the size of the feature map, and the decoder uses upsampling operations to expand the size of the feature map, and uses jump connections to fuse feature maps of different levels.

[0058] The face preview module is pre-trained with a regression loss function to preliminarily eliminate the degradation problem in non-frontal low-resolution face images. The regression loss function consists of two parts: global regression and key area regression loss function, thereby minimizing the difference between the predicted face image and the high-resolution frontal face image, and strengthening the learning of key facial feature areas.

[0059] Step S3: Use the latent space encoder to map the face image in the pixel space to the implicit space, so that the diffusion model is calculated in the implicit space, pre-train the unconditional diffusion model on the face dataset, use the training results as the initialization parameters of the diffusion model, and freeze the encoder of the denoising network.

[0060] The denoising network contains an encoder, an intermediate layer and a decoder. These modules are composed of cascaded nonlinear activation residual blocks, upsampling layers, downsampling layers and jump connections. An unconditional diffusion model is trained on a face dataset to generate a facial diffusion prior, which is used to initialize the denoising network and freeze the encoder of the denoising network. The encoder parameters remain unchanged in the subsequent training process.

[0061] Step S4: The facial prior guidance module is used to extract facial prior features from the rough frontal face image, and the facial prior features are used to guide the learning of the denoising network at different levels of feature space, and the spatial and semantic correlations between facial prior features and denoising features are captured through a hybrid cross-attention mechanism.

[0062] The facial prior guided module extracts facial prior features from the rough frontal face in the implicit space. This module is a branch parallel to the denoising network and contains an encoder with the same structure as the denoising network. The encoder of this module is initialized using the pre-trained denoising network initial parameters to transfer facial prior knowledge. The decoder part uses convolutional layers and jump connections to process feature maps of different feature levels to align them with the feature dimensions of the denoising network.

[0063] The hybrid cross-attention mechanism consists of a channel cross-attention module and a spatial cross-attention module; the channel cross-attention module automatically establishes connections between facial prior features and denoising feature channels to explore valuable feature channels, aggregates the potential vector of the global spatial dimension through global average pooling and global maximum pooling, processes the potential vector using a multi-layer perceptron, and standardizes it using an activation function; the spatial cross-attention module adaptively activates effective facial feature areas, captures the positional dependency between facial prior features and denoising features, compresses facial prior feature channels to one dimension using one-dimensional convolution, and standardizes them using an activation function; finally, the output results of the two branches are multiplied and fused with the denoising network features channel by channel and element by element.

[0064] Step S5: Use the identity consistency guidance module to extract facial identity encoding information and embed it into the denoising network to constrain the change of facial identity.

[0065] The identity consistency guidance module uses the identity encoder to extract consistent identity encoding features and embeds the identity encoding features into the middle layer of the denoising network. Specifically, the identity encoder is a residual network with the last fully connected classification layer removed. The identity encoder is trained using a triplet loss function. The frontal high-resolution face, the rough frontal face, and the face of other identities are used as anchor samples, positive samples, and negative samples, respectively. The triplet loss minimizes the distance between the identity encoding features of the anchor samples and the identity encoding features of the positive samples in the feature space, and maximizes the distance between the identity encoding features of the anchor samples and the identity encoding features of the negative samples.

[0066] Step S6: Initialize the Gaussian noise map, use the trained diffusion model to iteratively denoise the noise map to obtain a clean face image, use the latent space decoder to map the latent space denoising result to the pixel space, and finally reconstruct a high-resolution frontal face image.

[0067] Initialize the Gaussian noise map, use the trained denoising network to gradually predict the noise added in the current step, use the implicit sampling algorithm to iteratively denoise the noise map, use the latent space decoder to map the implicit space denoising result to the pixel space, and finally reconstruct a high-resolution frontal face image.

[0068] Example 2

[0069] This embodiment is similar to Figure 1 The image distribution shown in , 250 subjects were trained with a total of 32,000 face images, of which the first 200 subjects were used for training and the remaining 50 subjects were used for testing. The structure of the system of the present invention is as follows Figure 2 As shown in the figure, the target face is gradually restored from the low-resolution non-frontal face in a coarse-to-fine manner. First, the face preview module is used to recover the target face from the low-resolution non-frontal face (I LN ) to obtain a rough high-resolution frontal face (I CR ), and then use the powerful generation ability of the implicit diffusion model to refine the facial details, and finally generate a realistic high-resolution frontal face image (I HF ).

[0070] A face super-resolution method based on a dual guided diffusion model, such as Fig. 9 As shown, the specific method includes the following steps:

[0071] Step S1: Collect and process data

[0072] Face images (128×128 pixels) in different facial postures are collected, including 13 different angles (-90°, -75°, -60°, -45°, -30°, -15°, 0°, +15°, +30°, +45°, +60°, +75°, +90°). Face images with offset angle postures (non-0°) are defined as non-frontal face images, and face images with 0° posture of the corresponding identity are defined as frontal face images. The collected faces are cropped and aligned, and the non-frontal face images are downsampled 8 times to obtain non-frontal low-resolution face images of 16×16 pixels. The high-resolution frontal face images of the corresponding identities are used as label images.

[0073] Step S2: Pre-train the face preview module, send the low-resolution non-frontal face image to the face preview module, and obtain a rough frontal face image.

[0074] The overall structure of the face preview module is as follows Figure 3 As shown in Figure 1, the face preview module consists of three parts: encoder, intermediate layer and decoder. Each part consists of stacked non-linear activation residual blocks (NFBs). Each residual block is embedded with a spatial transformer (STN) module to capture facial pose changes. The NFB layer introduces layer normalization to enhance the training stability of the model, and removes non-linear units by replacing the classic activation function with simplified channel attention and simple gate switches. The number of stacked NFB layers in the encoder, intermediate layer and decoder are [2,2,4,8], [8] and [2,2,2,2], respectively. The encoder performs four 2× downsampling operations and the decoder performs four 2× upsampling operations to ensure that the input and output scales are the same. The encoder and decoder are connected by skip connections to achieve feature fusion.

[0075] The face preview module is pre-trained using a regression loss function. By minimizing the difference between the output face image and the frontal high-resolution face image, the degradation problem in the non-frontal low-resolution face image is initially eliminated. The regression loss function of this part consists of two parts: the global regression loss function and the key area regression loss function. The global regression loss is as follows:

[0076] L gr =||I CR -I HF || 2 (1)

[0077] In the formula, I CR =ξ(I LN ),ξ is the face preview module, I LN For low-resolution non-frontal faces, I CR is the predicted rough frontal face, I HFis a label face image. The key area regression loss is as follows:

[0078]

[0079] Where R represents the set of key feature regions, including the locations of eyes, nose and mouth, whose coordinates are estimated based on five predefined facial key points; r represents a region in R; n is the number of key feature regions; and They represent the pixels of the returned face and the high-quality frontal face in region r respectively.

[0080] The overall loss function of the face preview module is L cr =L kr +L gr ,The loss function is used for training for 24 epochs, the batch size is 8, the adaptive momentum estimation optimizer is used for optimization, and the learning rate is set to 5e-4.

[0081] Step S3: Use the latent space encoder to map the face image in the pixel space to the latent space. The structure of the denoising network is as follows: Figure 4 As shown in the "Denoising Network" in the figure, it contains encoders, intermediate layers and decoders. These modules are composed of cascaded NFB layers, upsampling layers, downsampling layers and skip connections. The pre-trained facial diffusion prior is constructed by training an unconditional diffusion model on a large facial dataset. The encoder of the denoising network is frozen during training to ensure that the feature encoder retains the diffusion prior knowledge of the true facial contour and texture details. The intermediate layer and decoder parameters are allowed to be updated in the subsequent training process to cope with complex facial condition changes.

[0082] Step S4: Use the facial prior guidance module to extract facial prior features from the rough frontal face image, and use the facial prior features to guide the denoising network learning at different feature space levels. The structure of the facial prior guidance module is as follows: Figure 4 As shown in the “Facial Prior Guidance Module”.

[0083] The facial prior guidance module extracts different levels of facial guidance features from the rough frontal face in the latent space. It is a branch parallel to the denoising network and contains an encoder with the same structure as the denoising network. The encoder of this module is initialized using the initial parameters of the pre-trained denoising network to transfer facial prior knowledge. The decoder uses 1×1 convolutional layers and jump connections to align the encoder feature maps of different feature levels to align them with the feature dimensions of the denoising network.

[0084] Then, the spatial and semantic correlations between the facial prior features and the denoised features are captured through a hybrid cross-attention mechanism; the overall structure of the hybrid cross-attention mechanism is as follows: Figure 5As shown, the module is inserted between the middle layer and multiple feature layers of the decoder. The calculation process is as follows:

[0085]

[0086] In the formula, Represents the input denoising network feature map, and Respectively represent the attention feature maps in the channel cross-attention branch and the spatial cross-attention branch, W fuse It represents the 3×3 convolution operation on the fused feature map, and i represents the level of the feature map.

[0087] Among them, the channel cross attention aggregates the global average pooling (GAP) (i.e., f in Formula 4) gap ) and global maximum pooling (GMP) (i.e., f in Formula 4 gmp ) to form a latent vector in the global spatial dimension, and then use a multilayer perceptron containing two fully connected layers (FC) (i.e., f in Formula 4) mlp ) processes the latent vector and uses the Sigmoid function to map the value to the range of [0,1]. The channel-level attention information extraction formula is as follows:

[0088]

[0089] Where W c represents a one-dimensional channel attention map; and Represent element-by-element addition and element-by-element multiplication operations, respectively. Represents the input facial prior branch feature map.

[0090] The spatial cross attention branch aims to adaptively activate the effective facial feature regions and capture the positional dependencies between facial prior features and denoising features. First, a convolutional layer is used to convert The feature channel is compressed into one dimension, and then the Sigmoid activation function is used to map it to the range of [0,1]. Finally, this value is combined with Multiply them to suppress the response of irrelevant areas and guide the model to focus on key facial areas. The formula is as follows:

[0091]

[0092] Where W s represents the two-dimensional activation attention map, f cb represents a stacked structure consisting of a 1×1 convolutional layer and a batch normalization layer. Represents an element-wise multiplication operation.

[0093] Step S5: Use the identity consistency guidance module to extract the facial identity encoding information, and embed the encoding information into the denoising network to constrain the facial identity change. The process is as follows: Figure 4 The specific process is as follows:

[0094] The identity consistency guidance module uses the identity encoder to extract the identity code with identity consistency and embeds the identity code into the middle layer of the denoising network. The identity encoder is a residual network with the last fully connected classification layer removed. The identity encoder is trained using a triplet loss function. The training process of the identity encoder is as follows: Figure 6 As shown, the triple loss function formula is as follows:

[0095]

[0096] In the formula, I HF ,I CR and I CR denote high-resolution frontal faces, rough high-resolution frontal faces, and faces of false identities, which are used as anchor samples, positive samples, and negative samples, respectively. δ denotes the identity encoder, and α denotes a hyperparameter. The ternary loss is calculated by minimizing the I HF and I CR The distance between them, and the maximum value of I in the feature space HF and I CK The distance between them is used for training, so as to ensure that the embedding of faces with the same identity is as close to the real face as possible, while the embedding of faces with different identities is as different as possible. Then, the trained identity encoder is used to extract the face from I CR The encoding features representing the identity are extracted from the network, and the identity embedding is processed by 1×1 convolution to reshape the dimension so that it is aligned with the middle layer dimension of the denoising network. Finally, it is input to the middle layer of the denoising network by element-by-element addition to constrain the identity changes in the generation process.

[0097] Step S6: Initialize the Gaussian noise map z~N(0,1), use the trained denoising network to gradually predict the noise added in the current step, use the implicit sampling algorithm to calculate the denoising result of the previous step according to the current step noise, continuously iterate the denoising of the noise map to obtain the final denoising result, use the latent space decoder to map the denoising result to the pixel space, and finally reconstruct a high-resolution frontal face image.

[0098] In the overall network training and reasoning process of this embodiment, first, during the training process, the face preview module is pre-trained using formulas (1) and (2), and a rough frontal face image is obtained; the facial pixel space is mapped to the latent space using a latent space encoder, so that the diffusion and denoising processes are performed in the latent space, and then the denoising network and identity encoder of the diffusion model are pre-trained and the pre-trained parameters are loaded, and the diffusion model is trained using the L2 distance as the loss function, and the formula is as follows:

[0099]

[0100] Where t is the time step, ∈ is sampled from a Gaussian distribution, ∈ θ represents the diffusion model, θ represents the model parameters, α i is a hyperparameter that changes with t, I CR is a rough frontal face, ε represents the latent space encoder, and δ represents the identity encoder. The training parameters are as follows: the Adam optimizer is used for 24 rounds, the batch size is 8, and the learning rate is set to 1e-4.

[0101] During the inference process, the trained denoising network is used to extract the noise image z t Estimate the noise-free face image z0 in . First, load the pre-trained face preview module to obtain a rough frontal face image; then use the rough frontal face image to construct facial prior features and identity constraint features; then load the trained diffusion model parameters to perform iterative denoising in the latent space, calculate the noise added in the previous step based on the current step noise, and iterate continuously to finally obtain a clean image z0. The sampling formula is as follows:

[0102]

[0103] Among them, z t is the noise sampled from a standard normal distribution, is a hyperparameter that changes over time, and its calculation is the same as formula (9), t is the time step, ∈ θ represents the diffusion model, I CR is a rough frontal face, ε represents the latent space encoder, δ represents the identity encoder, and σ is the variance, which defaults to 0. The sampling formula (Formula 10) is used for 50 iterations to obtain z0, and finally z0 is mapped to the pixel space through the latent space decoder to obtain a high-resolution frontal face.

[0104] Test Case

[0105] The proposed method is compared with low-resolution face reconstruction using different models and different poses. Figure 7 and Figure 8 shown. Figure 7is a comparison chart of the reconstruction results of different models, where Figure 7 (a) is the input low-resolution non-frontal face image, Figure 7 (b) is a schematic diagram of the results of Real-ESRGAN+DRGAN. Figure 7 (c) is a schematic diagram of the results of DRGAN+Real-ESRGAN. Figure 7 (d) is a schematic diagram of the result of SR3+DAGAN. Figure 7 (e) is a schematic diagram of the results of DAGAN+SR3. Figure 7 (f) is a schematic diagram of the TANN results. Figure 7 (g) is a schematic diagram of the Refusion result. Figure 7 (h) is a schematic diagram of the DiffBIR results. Figure 7 (i) is a schematic diagram of the results of the method of the present invention, Figure 7 (j) is the label image. , Figure 7 It can be clearly seen that the reconstructed image of the method in this case is clearer, more accurate, and more effective. The present invention can accurately restore a high-resolution frontal face image from a low-resolution non-frontal face image.

[0106] Figure 8 is a comparison of low-resolution face reconstruction results in different poses, where: Figure 8 (a) The input is a low-resolution non-frontal face image. Figure 8 (b) is a schematic diagram of the DiffBIR results. Figure 8 (c) is a schematic diagram of the result of the present invention, and (d) is a label image. Figure 8 It can be seen from the visualization of the restoration results in different postures that the method of the present invention has good fidelity and identity consistency for faces in different postures.

[0107] In summary, the method and system of the present invention can simultaneously realize face super-resolution reconstruction and face posture correction, and restore a high-fidelity high-resolution frontal face image from a low-resolution non-frontal face image.

[0108] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. Face super-resolution method based on dual guided diffusion model, characterized by , including at least the following steps: S1: Collecting facial images with different facial postures and different image resolutions, and constructing training data pairs of low-resolution non-frontal facial images and high-resolution frontal facial images; the facial images with offset angles are non-frontal facial images; S2: inputting the low-resolution non-frontal face image obtained in step S1 into the face preview module to capture the facial posture changes and obtain a rough frontal face image; S3: Use the latent space encoder to map the face image in the pixel space to the implicit space, so that the diffusion model is calculated in the implicit space, pre-train the unconditional diffusion model on the face dataset, use the training results as the initialization parameters of the diffusion model, and freeze the encoder of the denoising network; S4: A facial prior guidance module is used to extract facial prior features from rough frontal face images, and the facial prior features are used to guide the learning of the denoising network at different levels of feature space. The spatial and semantic correlations between facial prior features and denoising features are captured through a hybrid cross-attention mechanism. S5: Use the identity encoder in the identity consistency guidance module to extract consistent facial identity encoding features and embed them into the middle layer of the denoising network; S6: Initialize the Gaussian noise map, use the trained diffusion model to iteratively denoise the noise map to obtain a clean face image, use the latent space decoder to map the latent space denoising result to the pixel space, and finally reconstruct a high-resolution frontal face image.

2. The face super-resolution method based on the dual guided diffusion model as claimed in claim 1, characterized in that: In the step S1, the collected face image is cropped and aligned, and the non-frontal face image is down-sampled to obtain a low-resolution non-frontal face image.

3. The face super-resolution method based on the dual guided diffusion model as claimed in claim 1, characterized in that: In the step S2 of obtaining a rough frontal face image, the face preview module is pre-trained using a regression loss function, which consists of two parts: a global regression loss function and a key area regression loss function, wherein the global regression loss is: L gr =||I CR -I HF || 2 In the formula, I CR =ξ(I LN ),ξ is the face preview module, I LN For low-resolution non-frontal faces, I CR is the predicted rough frontal face, I HF is a label face image; The key area regression loss is: In the formula, R represents the set of key feature regions; r represents a region in R; n is the number of key feature regions; and They represent the pixels of the returned face and the high-quality frontal face in region r respectively.

4. The face super-resolution method based on the dual guided diffusion model as claimed in claim 1, characterized in that: The hybrid cross-attention mechanism in step S4 is composed of a channel cross-attention module and a spatial cross-attention module; The channel cross attention module aggregates the potential vector of the global spatial dimension through global average pooling and global maximum pooling, processes the potential vector using a multi-layer perceptron, and normalizes it using an activation function; The spatial cross-attention module uses one-dimensional convolution to compress the facial prior feature channel to one dimension and standardizes it using an activation function; the output results of the two branches of the channel cross-attention module and the spatial cross-attention module are multiplied and fused with the denoising network features channel by channel and element by element.

5. The face super-resolution method based on the dual guided diffusion model as claimed in claim 4, characterized in that: The hybrid cross attention mechanism is specifically: Where W s represents the two-dimensional activation attention map, f cb represents a stacked structure consisting of a 1×1 convolutional layer and a batch normalization layer. represents an element-wise multiplication operation, represents the input facial prior branch feature map, Represents the input denoising network feature map, Represents the attention feature map in the spatial criss-cross attention branch.

6. The face super-resolution method based on the dual guided diffusion model as claimed in claim 5, characterized in that: The identity encoder in step S5 is a residual network with the last fully connected classification layer removed, and the identity encoder is trained using a triplet loss function, which is specifically: Among them, I HF ,I CR and I CK denote high-resolution frontal faces, rough high-resolution frontal faces, and false-identity faces, respectively, δ denotes the identity encoder, and α denotes a hyperparameter.

7. A face super-resolution system based on a dual guided diffusion model, characterized in that: At least: Pre-trained face preview module, denoising network, face prior guidance module, identity consistency guidance module, The pre-trained face preview module is used to convert a low-resolution non-frontal face image into a rough frontal face image; The denoising network comprises an encoder, an intermediate layer and a decoder, and is composed of a cascaded non-linear activation residual block (NFB) layer, an upsampling layer, a downsampling layer and a jump connection. The pre-trained facial diffusion prior is constructed by training an unconditional diffusion model on a large facial dataset. The encoder of the denoising network is frozen during training, and the parameters of the intermediate layer and the decoder are allowed to be updated in the subsequent training process. The facial prior guidance module is used to extract facial prior features from a rough frontal face image and capture the spatial and semantic correlations between the facial prior features and the denoising features through a hybrid cross-attention mechanism; The identity consistency guidance module is used to extract consistent identity coding features and embed the identity coding features into the middle layer of the denoising network. The identity consistency guidance module includes an identity encoder, and the identity encoder is trained using a triplet loss function. The triplet loss minimizes the distance between the anchor point sample identity coding features and the positive sample identity coding features in the feature space, and maximizes the distance between the anchor point sample identity coding features and the negative sample identity coding features.

8. The face super-resolution system based on dual guided diffusion model as claimed in claim 7, characterized in that: The pre-trained facial preview module includes an encoder, an intermediate layer and a decoder, each of which is composed of stacked NFBs, and a spatial transformation (STN) module is embedded after each residual block to capture facial posture changes; the encoder and the decoder are connected by a jump connection.

9. The face super-resolution system based on dual guided diffusion model as claimed in claim 8, characterized in that: The facial prior guidance module is parallel to the denoising network, and includes an encoder with the same structure as the denoising network. The encoder of the facial prior guidance module is initialized using the pre-trained initial parameters of the denoising network, and the decoder part uses convolutional layers and jump connections to process feature maps of different feature levels.

Images