Face super-resolution generative adversarial network based on attitude condition constraint and generation method

Abstract

The invention discloses a face super-resolution generative adversarial network based on attitude condition constraint and a generation method, and the network at least comprises a side face attention guiding module, an attention fusion transformation module, a guiding transformation sub-network, an upper sampling sub-network, and a discrimination network. A unified generative adversarial network is established to realize the combination of face super-resolution and posture correction tasks, and a real high-resolution face image is restored while face posture correction is performed. According to the method, super-resolution reconstruction and posture correction of the face can be realized at the same time, and a real high-resolution front face image is restored from a low-resolution side face image.

Description

Technical Field

[0001] The present invention belongs to the technical field of face restoration, and mainly relates to a face super-resolution generative adversarial network based on posture condition constraints and a generation method. Background Art

[0002] In real-life scenarios, including traffic supervision, public security law enforcement, and public surveillance, face recognition technology plays an increasingly important role. On the one hand, the cameras installed at traffic intersections can not only identify license plates, but also collect driver facial image data for identification and comparison, which facilitates the traffic department to confirm the identity of traffic violators; on the other hand, the country's emphasis on urban safety has also prompted the popularization of surveillance cameras in the streets and alleys of the city. These cameras can not only capture various illegal and criminal behaviors, but also use advanced face recognition technology to quickly track suspects, providing powerful law enforcement assistance for the public security department. However, face recognition technology often relies on high-quality face image input. Face images obtained in non-controlled scenes such as traffic intersections usually have low resolution, which directly leads to a decrease in the accuracy and reliability of face recognition algorithms. In order to solve this problem, face super-resolution technology came into being. Face super-resolution is to reconstruct high-resolution face images with rich details and high clarity by finely processing and enhancing low-resolution face images. Improving image quality through super-resolution technology can effectively improve the accuracy and reliability of face recognition.

[0003] In recent years, with the rise of deep learning, face super-resolution methods based on neural networks have made significant progress. Among them, the generative adversarial network (GAN) can generate high-quality face images without making explicit parameter distribution assumptions on face images. Therefore, more and more researchers have proposed face super-resolution methods based on generative adversarial networks. Face super-resolution methods based on generative adversarial networks usually include two stages, namely the adversarial training stage and the generation stage. The adversarial training stage is to conduct adversarial training on the generative network and the discriminative network. The generative network deceives the discriminative network as much as possible, and the discriminative network distinguishes the true and false images reconstructed by the generative network as much as possible, and finally makes the data distribution generated by the generative network close to the real distribution. The generation stage is to use the trained generative network, input a low-resolution face image, and finally reconstruct a high-resolution face image.

[0004] Since the goal of face super-resolution reconstruction is highly deterministic, it is necessary to generate high-resolution faces of specific identities. Therefore, the generative network is often limited by conditions, and the existing generative adversarial network has only achieved good results in the face super-resolution task under the frontal face posture. However, in practical applications, due to factors such as shooting angle and face movement, the face images captured often have different postures, and the face feature information contained in such images is very limited. Most of the existing face super-resolution methods based on generative adversarial networks assume that the face posture is fixed, ignoring the impact of posture changes on the restoration effect. Therefore, when faced with low-resolution face images with large posture changes, these methods often find it difficult to achieve satisfactory restoration results, and the performance of the model will be significantly reduced. Summary of the invention

[0005] The present invention is aimed at the problem that the generation of face super-resolution is limited and the generation effect is poor in the prior art, and proposes a face super-resolution generation adversarial network and generation method based on posture condition constraints, which at least includes a side face attention guidance module, an attention fusion transformation module, a guidance transformation sub-network, an upsampling sub-network and a discriminant network, and establishes a unified generation adversarial network to realize the combination of face super-resolution and posture correction tasks, and restores a real high-resolution face image while correcting the face posture. The present invention can simultaneously realize face super-resolution reconstruction and posture correction, and restore a real high-resolution frontal face image from a low-resolution side face image.

[0006] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a face super-resolution generative adversarial network based on posture condition constraints at least includes a side face attention guidance module, an attention fusion transformation module, a guidance transformation sub-network, an upsampling sub-network and a discriminant network,

[0007] The side face attention guiding module generates a Gaussian distribution feature heat map based on the prior facial feature points of the input low-resolution side face image, so as to guide the generative adversarial network to focus on key facial organs;

[0008] The attention fusion transformation module: consists of channel attention, spatial transformation network and spatial attention sequence, which is used to embed in the generative adversarial network to assist the generative adversarial network to gradually and finely process the face image, thereby improving the fidelity and structural similarity of the generated image;

[0009] The guided transformation subnetwork: adopts a U-Net structure, and is composed of a downsampling encoder and an upsampling decoder, and is used to process low-resolution side face images and facial feature heat maps thereof, and output a front face feature map; each layer of the downsampling encoder and the upsampling decoder is jump-connected via a connection layer;

[0010] The upsampling subnetwork is composed of a deconvolution layer and an attention fusion transformation module. Finally, a convolution layer is used to transform the feature map into an RGB face image, which is used to upsample the frontal face feature map, amplify the image, and enhance key information such as subtle facial expressions, textures or contours;

[0011] The discriminant network is used to input the generated face images and real face images, so as to perform model adversarial training with the generative adversarial network and supervise the generation performance of the generative adversarial network.

[0012] As an improvement of the present invention, the channel attention: uses global average pooling, obtains a weight map of each channel through Sigmoid, multiplies the weight map with the original feature to obtain a feature map after channel attention, and realizes adaptive calibration of features in the channel dimension;

[0013] The spatial transformation network: compensates for the in-plane rotation, translation and scale transformation of the face by predicting and applying affine transformation parameters;

[0014] The spatial attention: uses global maximum pooling and global average pooling, performs splicing and convolution according to channels, and then obtains the spatial attention weight matrix through the Sigmoid activation function. The weight matrix is ​​multiplied by the original feature map to obtain the spatial attention feature map.

[0015] As another improvement of the present invention, in the downsampling encoder in the guided transformation subnetwork, the pooling layer, the convolution layer, and the attention fusion transformation module are stacked in sequence, and the upsampling decoder is stacked in sequence with the connection layer, the deconvolution layer, and the attention fusion transformation module.

[0016] In order to achieve the above object, the present invention also adopts a technical solution: a face super-resolution generation method based on posture condition constraints, comprising the following steps:

[0017] S1. Collecting and preprocessing data: collecting and preprocessing face images under different posture conditions, wherein the face images are divided into front face images and side face images, and constructing training data pairs of low-resolution side face images and high-resolution front face images;

[0018] S2, generate facial feature heat map: send the low-resolution side face image constructed in step S1 to the side face attention guidance module for processing, generate a Gaussian distribution feature heat map based on the prior facial feature points, and connect it with the original image in the channel direction and then input it into the generative adversarial network;

[0019] S3, correcting face posture: the facial feature heat map generated in step S2 is connected with the corresponding original image in the channel dimension as input, and then input into the guided transformation subnetwork in the generative adversarial network, combined with the attention fusion transformation module and the jump connection for multi-scale feature fusion, to obtain the face feature map after posture correction;

[0020] S4, face image reconstruction: the face feature map output by the guided transformation subnetwork in step S3 is sent to the upsampling subnetwork in the generative adversarial network, and the face feature map after posture correction is refined and reconstructed;

[0021] S5, model training: sending the refined and reconstructed face image obtained in step S4 and the corresponding real face image to the discriminant network, training the generative adversarial network model, and using the loss function to supervise the model parameters of the generative adversarial network to improve the network performance;

[0022] S6. Result output: The low-resolution side face image constructed in step S1 is input into the adversarial network trained in step S5 to reconstruct a high-resolution front face image.

[0023] As an improvement of the present invention, the preprocessing in step S1 at least includes performing face extraction and alignment on the collected face images, and downsampling the side face images to obtain low-resolution side face images.

[0024] As another improvement of the present invention, the prior facial feature points in step S2 include at least five pairs, whose positions correspond to the center of the left eye, the center of the right eye, the tip of the nose, the left corner of the mouth and the right corner of the mouth, respectively. A two-dimensional Gaussian distribution is used to fill the values ​​of the pixel points corresponding to each pair of feature point coordinates and their nearby pixel points, and the standard deviation is set to 0.8.

[0025] As another improvement of the present invention, in the upsampling network of step S4, the upsampling rate of the deconvolution layer is 2, the convolution kernel size is 5×5, the step size is 2, and a convolution layer is used to transform the feature map into a 128×3×3 RGB face image, the convolution kernel size is 5×5, and the step size is 2.

[0026] As a further improvement of the present invention, step S5 specifically includes the following steps:

[0027] S51: Use the generated face image as the input of the discriminant network, perform forward propagation to obtain the discriminant result, use the label image to do the same operation, and construct the two discriminant results as adversarial loss;

[0028] S52: Use the generated face image as the input of the VGG19 network, perform forward propagation to obtain the feature map, and then use the label image to do the same operation, and construct the two feature maps as the identity preservation loss;

[0029] S53: Use l2 loss function to calculate the pixel loss of generated image and label image;

[0030] S54: The three losses in steps S51, S52 and S53 are constructed into a total loss function, and the parameters of the generative adversarial network model are improved by minimizing the losses.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] (1) The present invention proposes an innovative generative adversarial network that can restore a true high-resolution face image while correcting the face posture. Specifically, a generative adversarial network (GT-GAN) based on posture condition constraints is proposed. In the side face attention guidance module, a Gaussian distributed feature heat map is generated according to the prior facial feature points, and is connected with the original image in the channel dimension and then sent to the network to guide the network to focus on the key facial organs of the side face, which is more efficient and accurate.

[0033] (2) The present invention embeds the attention fusion transformation module into the generative adversarial network, utilizes the attention mechanism and the spatial transformation network, and combines it with a sequential feature enhancement strategy to enhance the details of facial reconstruction and improve the authenticity and network performance of the network-generated images.

[0034] (3) The generative adversarial network of the present invention can well restore high-quality frontal face images under low resolution and different face posture conditions, and has good robustness to facial expressions and slight lighting changes, thereby improving the quality of mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is an example diagram of a low-resolution side face image in Embodiment 2 of the present invention;

[0036] Figure 2 is an example diagram of the side face attention guiding module of the present invention;

[0037] Figure 3 It is a schematic diagram of the structure of the generative adversarial network of the present invention;

[0038] Figure 4 is a structural diagram of the attention fusion transformation module of the present invention;

[0039] Figure 5 is a schematic diagram of the face restoration result of Example 2, wherein:

[0040] Figure (a) is a schematic diagram of an input low-resolution side face image;

[0041] Figure (b) is a schematic diagram of the result after the method of the present invention;

[0042] Figure (c) is the label image;

[0043] Figure 6 It is a flow chart of the steps of the generation method of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0045] Example 1

[0046] A face super-resolution generative adversarial network based on posture condition constraints at least includes a side face attention guidance module, an attention fusion transformation module, a guidance transformation sub-network, an upsampling sub-network and a discriminant network.

[0047] The side face attention guidance module first locates the coordinates of the eyes, nose, and mouth in the side face image based on the prior facial feature points, and fills them with Gaussian distribution, where the pixel points where the mean is located are the feature points corresponding to the organs, and the standard deviation is set to 0.8. After obtaining the feature heat map, it is connected with the original image in the channel direction and sent to the generation network together.

[0048] The attention fusion transformation module consists of channel attention, spatial transformation network, and spatial attention sequence; channel attention uses global maximum pooling and global average pooling, and then uses Sigmoid to obtain the weight map of each channel, which can be multiplied with the original feature to obtain the feature map after channel attention, realizing the adaptive calibration of features in the channel dimension; the spatial transformation network compensates for the face's in-plane rotation, translation, and scale transformation by predicting and applying affine transformation parameters; spatial attention uses global maximum pooling and global average pooling, and performs splicing and convolution according to channels, and then uses Sigmoid activation function to obtain the spatial attention weight matrix, which can be multiplied with the original feature map to obtain the spatial attention feature map. The guided transformation subnetwork adopts the U-Net structure, which consists of a downsampling encoder and an upsampling decoder. The downsampling encoder stacks the pooling layer, convolution layer, and attention fusion transformation module in sequence, and the upsampling decoder stacks the connection layer, deconvolution layer, and attention fusion transformation module in sequence. Each layer of the downsampling encoder and the upsampling decoder is skipped through the connection layer.

[0049] The main body of the upsampling subnetwork consists of a deconvolution layer and an attention fusion transformation module, and finally a convolution layer is used to transform the feature map into an RGB face image.

[0050] Use the generated face image as the input of the discriminant network, perform forward propagation to obtain the discriminant result, then use the label image to do the same operation, construct the two discriminant results as the discriminator loss, and then obtain the adversarial loss; use the generated face image as the input of the VGG19 network, perform forward propagation to obtain the feature map, then use the label image to do the same operation, and construct the two feature maps as the identity preservation loss; use the l2 loss function to calculate the pixel loss of the generated image and the label image. The above three losses are constructed as a total loss function to minimize the loss so that the generated network can learn model parameters that are more similar to high-quality input conditions under low-quality input conditions.

[0051] Finally, the trained generative network is used to process the low-resolution side face image, and finally a high-resolution front face image is reconstructed. The present invention can simultaneously realize super-resolution face reconstruction and posture correction, and restore a true high-resolution front face image from a low-resolution side face image.

[0052] Example 2

[0053] The original input image of this embodiment is a profile face image (128×128 pixels) that has been downsampled 8 times to a degraded image (16×16 pixels). Figure 1 As shown in the figure, it can be seen that the face images are severely degraded at low resolution and in different postures, and it is difficult to associate the real face from the existing images, especially the texture and features of local details, which will have a great impact on downstream recognition tasks. In addition, due to the lack of feature information of side faces, the restoration task is extremely challenging.

[0054] This embodiment is similar to Figure 1 The image distribution shown in the training data was 38,987 images of 250 subjects, of which the first 200 were used for training and the rest were used for testing. Figure 6 As shown, the specific steps include:

[0055] Step S1: collect face images under different posture conditions, divide them into front face images and side face images, and construct training data pairs of low-resolution side face images and high-resolution front face images.

[0056] Face images with different postures are collected, and the faces are extracted and aligned and scaled to 128×128 pixels. They are divided into frontal face images and side face images according to the posture. The side faces are then downsampled 8 times to obtain low-resolution side faces, and the corresponding high-resolution frontal face images are used as label images.

[0057] Step S2: The low-resolution side face image is sent to the side face attention guidance module for processing, and a Gaussian distribution feature heat map is generated according to the prior facial feature points. The heat map is connected with the original image in the channel direction and then input into the generation network, such as Figure 2 shown.

[0058] First, read the coordinates of the prior side face facial feature points. Each side face image corresponds to a set of feature point coordinates. Each set of feature point coordinates has five pairs, and their positions correspond to the center of the left eye, the center of the right eye, the tip of the nose, the left corner of the mouth, and the right corner of the mouth.

[0059] The values ​​of the pixels corresponding to each pair of coordinates and their nearby pixels are filled with a two-dimensional Gaussian distribution, where the pixel at the mean is the location of the corresponding feature point of the organ, and the standard deviation is set to 0.8. After obtaining the feature heat map, it is connected with the original image in the channel direction and sent to the generation network together.

[0060] Step S3: Use the guided transformation subnetwork in the generative network and combine it with the attention fusion transformation module to process the image, and use the skip connection for multi-scale feature fusion, such as Figure 3 and Figure 4 As shown in Figure 3 This is a schematic diagram of the structure of the generative adversarial network of this embodiment, where PAG (Profile Attention Guidance) is the profile attention guidance module, and AFTM (Attention Fusion Transformation Module) is the attention fusion transformation module; Figure 4 It is a structural diagram of the attention fusion transformation module.

[0061] First, the proposed attention fusion transformation module is composed of channel attention (CA), spatial transformer network (STN), and spatial attention (SA) in sequence; CA uses global maximum pooling and global average pooling, and then uses 1×1 convolution to perform channel compression to obtain two feature maps, which are added and then passed through Sigmoid to obtain the weight map of each channel. Multiplying them with the original features can obtain the feature map after channel attention, realizing the adaptive calibration of features in the channel dimension; STN consists of a positioning network and a grid generator. The positioning network is a small convolutional neural network. According to the input image, this network will output a set of parameters that define A spatial transformation can be a translation, rotation, scaling, or more complex affine transformation or nonlinear transformation. The grid generator receives the transformation parameters predicted by the positioning network and generates a coordinate grid that represents the new position of each pixel in the input image mapped to the output image. STN compensates for the in-plane rotation, translation, and scale transformation of the face by predicting and applying affine transformation parameters. SA uses global maximum pooling and global average pooling, and performs splicing and convolution according to channels. The convolution kernel size is 3×3, and then the spatial attention weight matrix is ​​obtained through the Sigmoid activation function. The weight matrix is ​​multiplied by the original feature map to obtain the spatial attention feature map.

[0062] After that, the original image is connected with the feature heat map and sent to the guided transformation subnetwork, which adopts the U-Net structure and consists of a downsampling encoder and an upsampling decoder. The downsampling encoder stacks the pooling layer, convolution layer, and attention fusion transformation module in sequence, where the pooling layer kernel size is 2×2 and the convolution layer kernel size is 3×3, and finally obtains a feature map of size 512×2×2. After unfolding, it passes through a fully connected layer and reconstructs it into a feature map of 512×2×2, and then inputs the upsampling decoder. The upsampling decoder stacks the connection layer, deconvolution layer, and attention fusion transformation module in sequence, where the deconvolution layer upsampling rate is 2 and the convolution kernel size is 3×3, and finally obtains a feature map of 512×32×32. Each layer of the downsampling encoder and the upsampling decoder is skipped through the connection layer.

[0063] Step S4: The facial feature map output by the guided transformation subnetwork is sent to the upsampling subnetwork in the generation network to gradually refine and reconstruct the facial feature map after posture correction. The upsampling subnetwork is mainly composed of a deconvolution layer and an attention fusion transformation module. The upsampling rate of the deconvolution layer is 2, the convolution kernel size is 5×5, and the step size is 2. Finally, a convolution layer is used to transform the feature map into a 128×3×3 RGB face image, with a convolution kernel size of 5×5 and a step size of 2.

[0064] Step S5: Send the reconstructed face and the corresponding real face to the discriminant network for adversarial training with the generative network, and use the loss function to supervise the generation performance of the generative network. The specific steps are as follows:

[0065] (1) Use the generated face image as the input of the discriminant network, perform forward propagation to obtain the discriminant result, and then use the label image to do the same operation. The two discriminant results are constructed as the discriminator loss L D , the formula is as follows:

[0066]

[0067] In the formula, Represents the generated image With label image h i The joint distribution between them, D and d represent the discriminant network and its parameters. During the training process, L is minimized D And update the parameters of the discriminator.

[0068] The goal of the generative network is to deceive the discriminative network as much as possible so that the generated images can pass the detection of the discriminative network, so the adversarial loss L adv It can be expressed as:

[0069]

[0070] During the training process, minimize L D With L adv , and update the discriminative network and the generative network alternately.

[0071] (2) Use the generated face image as the input of the VGG19 network, perform forward propagation to obtain the feature map output by the ReLu32 layer, and then use the label image to do the same operation, and construct the two feature maps as the identity preservation loss L id , the formula is as follows:

[0072]

[0073] Where Φ(·) represents the feature map extracted by the ReLU32 layer in VGG19.

[0074] (3) Use the l2 loss function to calculate the pixel loss L between the generated image and the label image mse , the formula is as follows:

[0075]

[0076] (4) The above three losses are constructed to generate the network loss function L G , by minimizing L GThe generated network learns model parameters that are more similar to those under high-quality input conditions under low-quality input conditions. The formula is as follows:

[0077] L G =L mse +αL id +βL adv

[0078] Where α and β are constants, both set to 0.01.

[0079] Step S6: Use the trained generative network to process the low-resolution side face image, and finally reconstruct a high-resolution front face image.

[0080] Figure 5 The following is a schematic diagram showing the restoration results of some face images after the method of the present invention, where: Figure 5 (a) is the input low-resolution side face image. It can be seen from the figure that the original low-resolution side face image is blurry and has a very low resolution. Figure 5 (b) is the result output diagram after the method of the present invention, Figure 5 (c) is a label image. It can be seen from the figure that after the method of the present invention, a high-resolution frontal face image can be well restored from a low-resolution side face image, and the effect is obvious.

[0081] In summary, the method of the present invention first collects face images with different postures and divides them into two categories: frontal face images and side face images, downsamples the side faces to make them degenerate into low-resolution images, and then constructs low-resolution side face image and high-resolution frontal face image training data pairs; the low-resolution side face images are sent to the side face attention guidance module for processing, and a Gaussian distributed feature heat map is generated according to the prior facial feature points, and then connected with the original image in the channel direction and input into the generative adversarial network to guide the network to focus on the key facial organs of the side face; the guided transformation subnet in the generative adversarial network is used The invention relates to a method for super-resolution face reconstruction and posture correction. The invention relates to a method for super-resolution face reconstruction and posture correction. The method comprises the following steps: a first step is to construct a super-resolution face reconstruction network, and a second step is to construct a super-resolution face reconstruction network. The first step is to construct a super-resolution face reconstruction network, and the ...

[0082] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A face super-resolution generative adversarial network based on pose constraints, characterized by: At least it includes a side face attention guidance module, an attention fusion transformation module, a guidance transformation sub-network, an upsampling sub-network and a discriminant network. The side face attention guiding module generates a Gaussian distributed feature heat map based on the prior facial feature points of the input low-resolution side face image, so as to guide the generative adversarial network to focus on key facial organs; The attention fusion transformation module: consists of channel attention, spatial transformation network and spatial attention sequence, embedded in the generative adversarial network, and assists the generative adversarial network in processing face images; The guided transformation subnetwork: adopts a U-Net structure, and is composed of a downsampling encoder and an upsampling decoder, and is used to process low-resolution side face images and facial feature heat maps thereof, and output a front face feature map; each layer of the downsampling encoder and the upsampling decoder is jump-connected via a connection layer; The upsampling subnetwork is composed of a deconvolution layer and an attention fusion transformation module. Finally, a convolution layer is used to transform the feature map into an RGB face image, which is used to upsample the frontal face feature map, amplify the image and enhance the key facial information; The discriminant network is used to input the generated face image and the real face image, so as to perform model adversarial training with the generative adversarial network and supervise the generation performance of the generative adversarial network.

2. The face super-resolution generative adversarial network based on posture condition constraints as claimed in claim 1, characterized in that: The channel attention: uses global average pooling, obtains the weight map of each channel through Sigmoid, and multiplies it with the original feature to obtain the feature map after channel attention, so as to realize adaptive calibration of features in the channel dimension; The spatial transformation network: compensates for the in-plane rotation, translation and scale transformation of the face by predicting and applying affine transformation parameters; The spatial attention: uses global maximum pooling and global average pooling, performs splicing and convolution according to channels, and then obtains the spatial attention weight matrix through the Sigmoid activation function. The weight matrix is ​​multiplied by the original feature map to obtain the spatial attention feature map.

3. The face super-resolution generative adversarial network based on posture condition constraints as claimed in claim 2, characterized in that: In the downsampling encoder in the guided transformation subnetwork, pooling layers, convolution layers, and attention fusion transformation modules are stacked in sequence, and the upsampling decoder stacks connection layers, deconvolution layers, and attention fusion transformation modules in sequence.

4. A face super-resolution generation method based on posture condition constraints using the adversarial network as claimed in claim 1, characterized in that: The steps include: S1. Collecting and preprocessing data: collecting and preprocessing face images under different posture conditions, wherein the face images are divided into front face images and side face images, and constructing training data pairs of low-resolution side face images and high-resolution front face images; S2, generating a facial feature heat map: sending the low-resolution side face image constructed in step S1 to the side face attention guidance module for processing, and generating a Gaussian distribution feature heat map based on the prior facial feature points; S3, correcting face posture: the facial feature heat map generated in step S2 is connected with the corresponding original image in the channel dimension as input, and then input into the guided transformation subnetwork in the generative adversarial network, combined with the attention fusion transformation module and the jump connection for multi-scale feature fusion, to obtain the face feature map after posture correction; S4, face image reconstruction: the face feature map output by the guided transformation subnetwork in step S3 is sent to the upsampling subnetwork in the generative adversarial network, and the face feature map after posture correction is refined and reconstructed; S5, model training: sending the refined and reconstructed face image obtained in step S4 and the corresponding real face image to the discriminant network, training the generative adversarial network model, and using the loss function to supervise the model parameters of the generative adversarial network; S6. Result output: The low-resolution side face image constructed in step S1 is input into the generative adversarial network trained in step S5 to reconstruct a high-resolution front face image.

5. The face super-resolution generation method based on posture condition constraints as claimed in claim 4, characterized in that: The preprocessing in step S1 at least includes performing face extraction and alignment on the collected face images, and downsampling the side face images to obtain low-resolution side face images.

6. The method for generating super-resolution face based on posture condition constraints as claimed in claim 6, characterized in that: The prior facial feature points in step S2 include at least five pairs, whose positions correspond to the center of the left eye, the center of the right eye, the tip of the nose, the left corner of the mouth and the right corner of the mouth respectively. The values ​​of the pixel points corresponding to each pair of feature point coordinates and the nearby pixel points are filled using a two-dimensional Gaussian distribution, and the standard deviation is set to 0.

8.

7. The method for generating super-resolution face based on posture condition constraints as claimed in claim 4, characterized in that: In the upsampling network of step S4, the upsampling rate of the deconvolution layer is 2, the convolution kernel size is 5×5, and the step size is 2. A convolution layer is used to transform the feature map into a 128×3×3 RGB face image, the convolution kernel size is 5×5, and the step size is 2.

8. The method for generating super-resolution face based on posture condition constraints as claimed in claim 4, characterized in that: The step S5 specifically includes the following steps: S51: Use the generated face image as the input of the discriminant network, perform forward propagation to obtain the discriminant result, use the label image to do the same operation, and construct the two discriminant results as adversarial loss; S52: Use the generated face image as the input of the VGG19 network, perform forward propagation to obtain the feature map, and then use the label image to do the same operation, and construct the two feature maps as the identity preservation loss; S53: Use l2 loss function to calculate the pixel loss of generated image and label image; S54: The three losses in steps S51, S52 and S53 are constructed into a total loss function, and the parameters of the generative adversarial network model are improved by minimizing the losses.

Images