Image enhancement method and device based on average face fusion analysis
By using an average face fusion analysis method and facial difference analysis and feature extraction techniques, the problem of unclear enhancement results for low-resolution face images is solved, and high-resolution face image restoration and feature enhancement are achieved.
Patent Information
- Application Number
- CN202511132676.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-13
- Publication Date
- 2025-11-21
AI Technical Summary
Existing low-resolution face image enhancement methods produce results that lack clear facial structure and details.
By acquiring the input face image and the average face image with a frontal pose, face difference analysis and feature extraction are performed. Then, a feature enhancement network and a super-resolution enhancement network are used to recover a high-resolution face image.
It recovers high-resolution facial images, improving image clarity and accuracy, and enhancing the representation of facial features.
Smart Images
Figure CN120997106A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to an image enhancement method and apparatus based on average face fusion analysis. Background Technology
[0002] With the advancement of deep learning technology in various computer vision fields, deep learning methods for face super-resolution have made significant progress in recent years and have become mainstream algorithms. However, low-resolution face images often face unknown degradation problems, and these degradation phenomena cannot provide enough information for robust feature representation. In order to solve this problem in real-world low-resolution scenes, in 2021, Chen proposed a progressive semantic-aware style transformation framework called PSFRGAN[1], which formalizes the restoration of low-quality face images into a multi-scale progressive restoration process through semantic-aware style transformation. In 2023, in order to capture facial structure information, Wang et al. proposed a spatiotemporal joint network (SFMNet) for face super-resolution. This method aims to explore the correlation between the spatial domain and the frequency domain to enhance the representation of extended face information. However, the results generated by PSFRGAN and SFMNet lack clear facial structure and details, and are therefore not satisfactory.
[0003] Therefore, this invention develops an image enhancement method and apparatus based on average face fusion analysis to solve the above problems. Summary of the Invention
[0004] This invention proposes an image enhancement method and apparatus based on average face fusion analysis to solve the problem that existing low-resolution face image enhancement results lack clear facial structure and details.
[0005] The present invention achieves the above objectives through the following technical solutions: This invention provides an image super-resolution enhancement method based on average face fusion analysis, comprising: The information obtained includes the input face image and the average face image in a frontal pose; Perform face difference analysis on the input face image and the average face image, and parse the average face image into a parsely registered face image with the same pose as the input face image; Facial features are extracted from the input face image to obtain a face feature map; The face feature map, the parsed and registered face image, and the average face image are input together into the feature enhancement network to output an intermediate super-resolution image. The intermediate super-resolution image and the input face image are then input together into the super-resolution enhancement network to output the final super-resolution image.
[0006] Specifically, facial difference analysis is performed on the input face image and the average face image, including: The average face image and the input face image are sequentially processed through three trained first convolutional neural networks to obtain intermediate result images; The intermediate result image is input into the trained differential variational autoencoder, which outputs a differential parsing image.
[0007] Specifically, the intermediate result image is input into the trained differential variational autoencoder, and the output differential resolution image includes: Construct a differential variational autoencoder, which includes an encoding network and a decoding network; The intermediate result image is input into the coding network to obtain the latent variable representation; The latent variable representation is input into the decoding network, and the difference analysis image is output.
[0008] Specifically, facial feature extraction is performed on the input face image, including: The input face image is sequentially fed into four trained first convolutional networks and four trained second convolutional networks, outputting a first feature set. The trained first convolutional network includes two convolutional layers and a max pooling layer connected in sequence. The trained second convolutional network includes a deconvolutional layer and two convolutional layers connected in sequence. According to the order in which the input face image is input, the output of the fourth max pooling layer is used as the input of the first deconvolutional layer, the output of the third max pooling layer is used as the input of the second deconvolutional layer, the output of the second max pooling layer is used as the input of the third deconvolutional layer, and the output of the first max pooling layer is used as the input of the fourth deconvolutional layer. The first feature set is concatenated to obtain the concatenated features; The spliced features are input into the trained third convolutional network to output a face feature map. The trained third convolutional network includes three convolutional layers connected in sequence.
[0009] Specifically, the facial feature map, the parsed and registered facial image, and the average facial image are input together into a feature enhancement network to output an intermediate super-resolution image. The intermediate super-resolution image and the input facial image are then input together into a super-resolution enhancement network to output a final super-resolution image, including: Construct and train a first U-net network as a feature enhancement network, and construct and train a second U-net network as a super-resolution enhancement network; The face feature map, the parsed and registered face image, and the average face image are input together into the first U-net network to output the intermediate super-resolution image; The intermediate super-resolution image and the input face image are input together into the second U-net network to output the final super-resolution image. The loss function of the second U-net network is:
[0010] in, I 1 represents the intermediate super-resolution image. I t This indicates the input face image, which is the label data from the training phase of the second U-net network.
[0011] Another aspect of the present invention provides an image super-resolution enhancement device based on average face fusion analysis, comprising: The acquisition module is used to acquire information, including an input face image and an average face image with a frontal pose. A face difference analysis module is used to perform face difference analysis on the input face image and the average face image, and to analyze the average face image into a registered face image with the same pose as the input face image. A face feature extraction module is used to extract face features from the input face image to obtain a face feature map; The face super-resolution enhancement module inputs the face feature map, the parsed and registered face image, and the average face image into the feature enhancement network, outputs an intermediate super-resolution image, and inputs the intermediate super-resolution image and the input face image into the super-resolution enhancement network to output the final super-resolution image.
[0012] Specifically, the face difference analysis module includes: The processing module is used to process the average face image and the input face image sequentially through three trained first convolutional neural networks to obtain an intermediate result image; The parsing module is used to input the intermediate result image into the trained differential variational autoencoder and output a differential parsing image.
[0013] Specifically, the parsing module includes: The construction module is used to construct a differential variational autoencoder, which includes an encoding network and a decoding network; An encoding module is used to input the intermediate result image into the encoding network to obtain a latent variable representation; A decoding module is used to input the latent variable representation into the decoding network and output a difference analysis image.
[0014] Specifically, the facial feature extraction module includes: The input module is used to sequentially input the input face image into four trained first convolutional networks and four trained second convolutional networks, and output a first feature set. The trained first convolutional network includes two convolutional layers and a max pooling layer connected in sequence. The trained second convolutional network includes a deconvolutional layer and two convolutional layers connected in sequence. According to the order in which the input face image is input, the output of the fourth max pooling layer is used as the input of the first deconvolutional layer, the output of the third max pooling layer is used as the input of the second deconvolutional layer, the output of the second max pooling layer is used as the input of the third deconvolutional layer, and the output of the first max pooling layer is used as the input of the fourth deconvolutional layer. A splicing module is used to splice the first feature set to obtain spliced features; The output module is used to input the spliced features into the trained third convolutional network and output a face feature map. The trained third convolutional network includes three convolutional layers connected in sequence.
[0015] Specifically, the face super-resolution enhancement module includes: The construction and training module is used to construct and train a first U-net network as a feature enhancement network and to construct and train a second U-net network as a super-resolution enhancement network. An enhancement module is used to input the face feature map, the parsed and registered face image, and the average face image into a first U-net network and output an intermediate super-resolution image. The super-resolution enhancement module is used to input the intermediate super-resolution image and the input face image together into a second U-net network, and output the final super-resolution image. The loss function of the second U-net network is:
[0016] in, I 1 represents the intermediate super-resolution image. I t This indicates the input face image, which is the label data from the training phase of the second U-net network.
[0017] The beneficial effects of this invention are as follows: This invention recovers a high-resolution face image from an input low-resolution face image and uses the average face of the high-resolution image as a reference. It extracts face features through decoupling and then uses these features to achieve face super-resolution. Attached Figure Description
[0018] Figure 1 This is the main flowchart of an embodiment of the present invention; Figure 2 This is a structural diagram of the difference analysis module according to an embodiment of the present invention; Figure 3 This is a structural diagram of the face super-resolution enhancement module according to an embodiment of the present invention; Figure 4 This is a diagram of the differential variational autoencoder network structure according to an embodiment of the present invention; Figure 5 This is a structural diagram of the face feature extraction module according to an embodiment of the present invention. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0020] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0021] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0022] In the description of this invention, it should be understood that the terms "upper," "lower," "inner," "outer," "left," "right," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are only used to facilitate the description of this invention and to simplify the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.
[0023] Furthermore, the terms "first," "second," etc., are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.
[0024] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, terms such as "set" and "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0025] In this article, "low resolution" refers to a resolution range of 16*16-32*32; "high resolution" refers to a resolution range of 128*128-256*256.
[0026] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0027] This embodiment uses one NVIDIA GTX3060ti GPU to run the experiment. The dataset includes more than 30,000 low-resolution face images, covering face image data with various poses and background information. In this embodiment, 30,000 faces are randomly selected for face super-resolution enhancement, and 25,000, 1,000, and 4,000 images are used as training set, validation set, and test sample data, respectively.
[0028] like Figure 1 As shown, an image super-resolution enhancement method based on average face fusion analysis includes: The information obtained includes the input face image and the average face image in a frontal pose; Perform face difference analysis on the input face image and the average face image, and parse the average face image into a parsely registered face image with the same pose as the input face image; Facial features are extracted from the input face image to obtain a face feature map; The face feature map, the parsed and registered face image, and the average face image are input together into the feature enhancement network to output an intermediate super-resolution image. The intermediate super-resolution image and the input face image are then input together into the super-resolution enhancement network to output the final super-resolution image.
[0029] The input face image is a low-resolution face image, while the average face image is a high-resolution face image. Since the average face is always in a frontal position, pose resolution registration is performed on the average face image to obtain a reconstructed face corresponding to the input face image. This process adjusts the frontal average face to the same pose as the actual input face image, thereby enabling more accurate subsequent processing and comparison.
[0030] The role of feature enhancement networks is to further process and optimize images to improve the quality and detail of the final result. They can better extract and enhance facial features, thereby improving the clarity and accuracy of the image.
[0031] By combining the input face image, the overall state and details of the intermediate super-resolution image are constrained and expressed, and the final super-resolution image is output.
[0032] like Figure 2 As shown, in some embodiments, facial difference analysis is performed on the input face image and the average face image, including: The average face image and the input face image are sequentially processed through three trained first convolutional neural networks to obtain intermediate result images; These trained first convolutional neural networks extract and process features from the input image, generating an intermediate result. This intermediate result represents the average facial features with the same pose as the low-resolution face. The key to this stage is to extract and preserve important feature information from the image through the convolutional operations of the deep learning network, laying the foundation for subsequent processing steps.
[0033] The intermediate result image is input into the trained differential variational autoencoder, which outputs a differential parsing image.
[0034] The average face image and the input face image not only differ in pose, but also in many other aspects such as hair, face shape, eyes, and nose. Specifically, the pose of the input face image may differ from the average face image, leading to variations in facial expression and angle. Furthermore, features such as the background, hair style, facial contours, eye shape, and nose structure of the input face image may also differ significantly from the average face image. These differences affect the overall appearance and feature representation of the image.
[0035] Due to these differences, a deep analysis of the distinct features between the input image and the average face image is performed. The goal of the differential variational encoder is to identify and understand these differences so that the features of the target face can be accurately compared with standard features in the reference image. Through this comparison, the unique features of the target face and its similarities and differences with the average face can be revealed, thus providing valuable information for subsequent image processing and matching.
[0036] like Figure 4 As shown, in some embodiments, the intermediate result image is input into a trained differential variational autoencoder to output a differential resolution image, including: Construct a differential variational autoencoder, which includes an encoding network and a decoding network; The intermediate result image is input into the coding network to obtain the latent variable representation; The latent variable representation is input into the decoding network, and the difference analysis image is output.
[0037] like Figure 5 As shown, in some embodiments, facial feature extraction of the input face image includes: The input face image is sequentially fed into four trained first convolutional networks and four trained second convolutional networks, outputting a first feature set. The trained first convolutional network includes two convolutional layers and a max pooling layer connected in sequence. The trained second convolutional network includes a deconvolutional layer and two convolutional layers connected in sequence. According to the order in which the input face image is input, the output of the fourth max pooling layer is used as the input of the first deconvolutional layer, the output of the third max pooling layer is used as the input of the second deconvolutional layer, the output of the second max pooling layer is used as the input of the third deconvolutional layer, and the output of the first max pooling layer is used as the input of the fourth deconvolutional layer. The first feature set is concatenated to obtain the concatenated features; The spliced features are input into the trained third convolutional network to output a face feature map. The trained third convolutional network includes three convolutional layers connected in sequence.
[0038] like Figure 3 As shown, in some embodiments, the face feature map, the parsed and registered face image, and the average face image are input together into a feature enhancement network to output an intermediate super-resolution image. The intermediate super-resolution image and the input face image are then input together into a super-resolution enhancement network to output a final super-resolution image, including: Construct and train a first U-net network as a feature enhancement network, and construct and train a second U-net network as a super-resolution enhancement network; The face feature map, the parsed and registered face image, and the average face image are input together into the first U-net network to output the intermediate super-resolution image; The intermediate super-resolution image and the input face image are input together into the second U-net network to output the final super-resolution image. The loss function of the second U-net network is:
[0039] in, I 1 represents the intermediate super-resolution image. I t This indicates the input face image, which is the label data from the training phase of the second U-net network.
[0040] like Figure 1As shown, the present invention also provides an image super-resolution enhancement device based on average face fusion analysis, comprising: The acquisition module is used to acquire information, including an input face image and an average face image with a frontal pose. A face difference analysis module is used to perform face difference analysis on the input face image and the average face image, and to analyze the average face image into a registered face image with the same pose as the input face image. A face feature extraction module is used to extract face features from the input face image to obtain a face feature map; The face super-resolution enhancement module inputs the face feature map, the parsed and registered face image, and the average face image into the feature enhancement network, outputs an intermediate super-resolution image, and inputs the intermediate super-resolution image and the input face image into the super-resolution enhancement network to output the final super-resolution image.
[0041] The face difference analysis module includes: The processing module is used to process the average face image and the input face image sequentially through three trained first convolutional neural networks to obtain an intermediate result image; The parsing module is used to input the intermediate result image into the trained differential variational autoencoder and output a differential parsing image.
[0042] The parsing module includes: The construction module is used to construct a differential variational autoencoder, which includes an encoding network and a decoding network; An encoding module is used to input the intermediate result image into the encoding network to obtain a latent variable representation; A decoding module is used to input the latent variable representation into the decoding network and output a difference analysis image.
[0043] The facial feature extraction module includes: The input module is used to sequentially input the input face image into four trained first convolutional networks and four trained second convolutional networks, and output a first feature set. The trained first convolutional network includes two convolutional layers and a max pooling layer connected in sequence. The trained second convolutional network includes a deconvolutional layer and two convolutional layers connected in sequence. According to the order in which the input face image is input, the output of the fourth max pooling layer is used as the input of the first deconvolutional layer, the output of the third max pooling layer is used as the input of the second deconvolutional layer, the output of the second max pooling layer is used as the input of the third deconvolutional layer, and the output of the first max pooling layer is used as the input of the fourth deconvolutional layer. A splicing module is used to splice the first feature set to obtain spliced features; The output module is used to input the spliced features into the trained third convolutional network and output a face feature map. The trained third convolutional network includes three convolutional layers connected in sequence.
[0044] The face super-resolution enhancement module includes: The construction and training module is used to construct and train a first U-net network as a feature enhancement network and to construct and train a second U-net network as a super-resolution enhancement network. An enhancement module is used to input the face feature map, the parsed and registered face image, and the average face image into a first U-net network and output an intermediate super-resolution image. The super-resolution enhancement module is used to input the intermediate super-resolution image and the input face image together into a second U-net network, and output the final super-resolution image. The loss function of the second U-net network is:
[0045] in, I 1 represents the intermediate super-resolution image. I t This indicates the input face image, which is the label data from the training phase of the second U-net network.
[0046] The present invention will be further described below through a performance comparison and evaluation of the present invention and existing methods: The performance of face super-resolution was tested using a test set. The test data mainly included low-resolution face images. This technology uses image quality evaluation metrics PSNR, SSIM, and LPIPS as objective evaluation standards for the enhancement effect.
[0047] The comparison algorithms employed include non-uniform brightness enhancement algorithms ASFFNet, CodeFormer, WaveletSRNet, SFMNet, and WFEN. These algorithms were all trained on simulated datasets and tested on corresponding test sets under fully trained conditions, allowing for objective metric comparisons. Tests show that the experimental results of this invention result in clearer facial details and less interference.
[0048] Table 1 shows the performance comparison and evaluation results of the present invention and the comparative algorithm.
[0049] Table 1
[0050] As can be seen from Table 1, the present invention outperforms other comparative algorithms in terms of objective metrics such as PSNR, SSIM, and LPIPS.
[0051] As can be seen, the present invention can significantly represent low-quality facial images and can reconstruct low-quality facial images.
[0052] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. An image super-resolution enhancement method based on average face fusion analysis, characterized in that, include: The information obtained includes the input face image and the average face image in a frontal pose; Perform face difference analysis on the input face image and the average face image, and parse the average face image into a parsely registered face image with the same pose as the input face image; Facial features are extracted from the input face image to obtain a face feature map; The face feature map, the parsed and registered face image, and the average face image are input together into the feature enhancement network to output an intermediate super-resolution image. The intermediate super-resolution image and the input face image are then input together into the super-resolution enhancement network to output the final super-resolution image.
2. The image super-resolution enhancement method based on average face fusion analysis according to claim 1, characterized in that, Perform face difference analysis on the input face image and the average face image, including: The average face image and the input face image are sequentially processed through three trained first convolutional neural networks to obtain intermediate result images; The intermediate result image is input into the trained differential variational autoencoder, which outputs a differential parsing image.
3. The image super-resolution enhancement method based on average face fusion analysis according to claim 2, characterized in that, The intermediate result image is input into the trained differential variational autoencoder, which outputs a differential resolution image, including: Construct a differential variational autoencoder, which includes an encoding network and a decoding network; The intermediate result image is input into the coding network to obtain the latent variable representation; The latent variable representation is input into the decoding network, and the difference analysis image is output.
4. The image super-resolution enhancement method based on average face fusion analysis according to claim 1, characterized in that, Extracting facial features from the input face image includes: The input face image is sequentially fed into four trained first convolutional networks and four trained second convolutional networks, outputting a first feature set. The trained first convolutional network includes two convolutional layers and a max pooling layer connected in sequence. The trained second convolutional network includes a deconvolutional layer and two convolutional layers connected in sequence. According to the order in which the input face image is input, the output of the fourth max pooling layer is used as the input of the first deconvolutional layer, the output of the third max pooling layer is used as the input of the second deconvolutional layer, the output of the second max pooling layer is used as the input of the third deconvolutional layer, and the output of the first max pooling layer is used as the input of the fourth deconvolutional layer. The first feature set is concatenated to obtain the concatenated features; The spliced features are input into the trained third convolutional network to output a face feature map. The trained third convolutional network includes three convolutional layers connected in sequence.
5. The image super-resolution enhancement method based on average face fusion analysis according to claim 1, characterized in that, The facial feature map, the analytically registered facial image, and the average facial image are input together into a feature enhancement network to output an intermediate super-resolution image. The intermediate super-resolution image and the input facial image are then input together into a super-resolution enhancement network to output a final super-resolution image, including: Construct and train a first U-net network as a feature enhancement network, and construct and train a second U-net network as a super-resolution enhancement network; The face feature map, the parsed and registered face image, and the average face image are input together into the first U-net network to output the intermediate super-resolution image; The intermediate super-resolution image and the input face image are input together into the second U-net network to output the final super-resolution image. The loss function of the second U-net network is: ， in, I 1 represents the intermediate super-resolution image. I t This indicates the input face image, which is the label data from the training phase of the second U-net network.
6. An image super-resolution enhancement device based on average face fusion analysis, characterized in that, include: The acquisition module is used to acquire information, including an input face image and an average face image with a frontal pose. A face difference analysis module is used to perform face difference analysis on the input face image and the average face image, and to analyze the average face image into a registered face image with the same pose as the input face image. A face feature extraction module is used to extract face features from the input face image to obtain a face feature map; The face super-resolution enhancement module inputs the face feature map, the parsed and registered face image, and the average face image into the feature enhancement network, outputs an intermediate super-resolution image, and inputs the intermediate super-resolution image and the input face image into the super-resolution enhancement network to output the final super-resolution image.
7. The image super-resolution enhancement device based on average face fusion analysis according to claim 6, characterized in that, The face difference analysis module includes: The processing module is used to process the average face image and the input face image sequentially through three trained first convolutional neural networks to obtain an intermediate result image; The parsing module is used to input the intermediate result image into the trained differential variational autoencoder and output a differential parsing image.
8. The image super-resolution enhancement device based on average face fusion analysis according to claim 7, characterized in that, The parsing module includes: The construction module is used to construct a differential variational autoencoder, which includes an encoding network and a decoding network; An encoding module is used to input the intermediate result image into the encoding network to obtain a latent variable representation; A decoding module is used to input the latent variable representation into the decoding network and output a difference analysis image.
9. The image super-resolution enhancement device based on average face fusion analysis according to claim 6, characterized in that, The facial feature extraction module includes: The input module is used to sequentially input the input face image into four trained first convolutional networks and four trained second convolutional networks, and output a first feature set. The trained first convolutional network includes two convolutional layers and a max pooling layer connected in sequence. The trained second convolutional network includes a deconvolutional layer and two convolutional layers connected in sequence. According to the order in which the input face image is input, the output of the fourth max pooling layer is used as the input of the first deconvolutional layer, the output of the third max pooling layer is used as the input of the second deconvolutional layer, the output of the second max pooling layer is used as the input of the third deconvolutional layer, and the output of the first max pooling layer is used as the input of the fourth deconvolutional layer. A splicing module is used to splice the first feature set to obtain spliced features; The output module is used to input the spliced features into the trained third convolutional network and output a face feature map. The trained third convolutional network includes three convolutional layers connected in sequence.
10. An image super-resolution enhancement device based on average face fusion analysis according to claim 6, characterized in that, The face super-resolution enhancement module includes: The construction and training module is used to construct and train a first U-net network as a feature enhancement network and to construct and train a second U-net network as a super-resolution enhancement network. An enhancement module is used to input the face feature map, the parsed and registered face image, and the average face image into a first U-net network and output an intermediate super-resolution image. The super-resolution enhancement module is used to input the intermediate super-resolution image and the input face image together into a second U-net network, and output the final super-resolution image. The loss function of the second U-net network is: ， in, I 1 represents the intermediate super-resolution image. I t This indicates the input face image, which is the label data from the training phase of the second U-net network.