Face restoration model training method, face restoration method, face restoration system and face restoration equipment
By combining technical means of generating adversarial networks and convolutional neural networks in the face repair model, the problem of high computing resource consumption of existing blind face recovery models is solved, and more efficient resource utilization and better universality are achieved.
Patent Information
- Application Number
- CN202411860635.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-17
- Publication Date
- 2025-04-15
AI Technical Summary
The existing blind face recovery model has a large amount of calculation and parameters, which leads to high consumption of computing resources and is difficult to deploy on edge devices with resource-constrained resources, and lacks universality.
The face repair module is obtained by distillation based on pre-trained generative adversarial networks, and the noise reduction removal module is obtained by combining preset convolutional neural network training. The channel segmentation feature transformation layer is used to connect the potential features of the two modules to obtain the target face repair model.
It significantly reduces the calculation amount and parameters of the model, reduces the consumption of computing resources, improves the operation efficiency, makes the model more suitable for edge devices, and improves universality.
Smart Images
Figure CN119831882A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing technology, and in particular to a training method for a face restoration model, a face restoration method, a system and a device. Background Art
[0002] Face images in the real world may be subject to various unknown and complex degradations, resulting in low-quality face images with different conditions such as low resolution, noise, blur and artifacts. The task of blind face restoration is to restore these low-quality face images to high-quality faces, which is a challenging task in the field of computer vision.
[0003] In existing research, such as GFPGAN (Generative Face Prior GAN, a generative adversarial network based on generative face prior), GPEN (Generative Prior Embedded Network), and CodeFormer (an artificial intelligence photo editing software), high-quality blind face generation effects have been achieved. Although the generation quality of blind face restoration has improved, these models have large computational complexity and parameter counts and require a lot of computing resources. The high computing resource requirements make these blind face restoration models difficult to deploy on resource-constrained edge devices and are not universal. Summary of the invention
[0004] The technical problem to be solved by the present disclosure is to overcome the defects of the blind face restoration model in the prior art, such as large amount of calculation and parameters, high consumption of computing resources, and lack of universality, and to provide a training method for a face restoration model, a face restoration method, a system and a device.
[0005] The present invention solves the above technical problems through the following technical solutions:
[0006] The present disclosure provides a training method for a face restoration model, the training method comprising:
[0007] The face restoration module is obtained by distillation based on the pre-trained generative adversarial network;
[0008] Wherein, the face restoration module includes a depth-separable modulated convolutional layer, and the face restoration module is used to obtain prior information of face generation;
[0009] A noise reduction and removal module is obtained based on the preset convolutional neural network training;
[0010] Wherein, the preset convolutional neural network includes a depth separable layer, and the noise reduction and removal module is used to perform noise reduction processing on the face image;
[0011] The potential features of the face restoration module and the noise reduction removal module are connected based on the channel segmentation feature transformation layer to obtain a target face restoration model.
[0012] Preferably, the step of obtaining a face restoration module by distilling a pre-trained generative adversarial network comprises:
[0013] The generative adversarial network is used as a teacher model, and the face restoration module is used as a student model to train and obtain an initial restoration module;
[0014] Based on the image scale requirement, the building blocks and weights of the initial restoration module are adjusted to obtain the face restoration module.
[0015] Preferably, the step of obtaining the noise reduction and removal module based on the preset convolutional neural network training includes:
[0016] The encoding module, bottleneck module and decoding module in the preset convolutional neural network are set to a group normalization mode to obtain the noise reduction and removal module.
[0017] Preferably, the step of connecting the potential features of the face restoration module and the noise reduction removal module based on the channel segmentation feature transformation layer to obtain the target face restoration model includes:
[0018] Connecting the potential features of the face restoration module and the noise reduction removal module based on the channel segmentation feature transformation layer to obtain an initial face restoration model;
[0019] Using at least one of a reconstruction loss function, an adversarial loss function, a facial component loss function, and an identity preservation loss function to train the initial face restoration model to obtain the target face restoration model;
[0020] and / or,
[0021] The generative adversarial network includes StyleGAN V2 (Style-Based Generative Adversarial Network, an improved version of a variant of the generative adversarial network GAN);
[0022] and / or,
[0023] The preset convolutional neural network includes U-Net (a convolutional neural network model).
[0024] The present disclosure also provides a face restoration method, the face restoration method comprising:
[0025] Input the initial face image into the face restoration model to obtain the target face image;
[0026] Wherein, the face restoration model is obtained based on the training method of the face restoration model described above.
[0027] The present disclosure also provides a training system for a face restoration model, the training system comprising:
[0028] A distillation module is used to obtain a face restoration module based on the pre-trained generative adversarial network distillation;
[0029] Wherein, the face restoration module includes a depth-separable modulated convolutional layer, and the face restoration module is used to obtain prior information of face generation;
[0030] A training module, used for obtaining a noise reduction and removal module based on a preset convolutional neural network training;
[0031] Wherein, the preset convolutional neural network includes a depth separable layer, and the noise reduction and removal module is used to perform noise reduction processing on the face image;
[0032] A connection module is used to connect the potential features of the face restoration module and the noise reduction removal module based on a channel segmentation feature transformation layer to obtain a target face restoration model.
[0033] Preferably, the distillation module comprises:
[0034] A training unit, used for using the generative adversarial network as a teacher model and the face restoration module as a student model to train and obtain an initial restoration module;
[0035] The adjusting unit is used to adjust the building blocks and weights of the initial restoration module based on the image scale requirement to obtain the face restoration module.
[0036] Preferably, the training module is also used to set the encoding module, bottleneck module and decoding module in the preset convolutional neural network to a group normalization mode to obtain the noise reduction and removal module.
[0037] Preferably, the connection module comprises:
[0038] A connection unit, used for connecting the potential features of the face restoration module and the noise reduction removal module based on the channel segmentation feature transformation layer to obtain an initial face restoration model;
[0039] A loss training unit, configured to train the initial face restoration model using at least one of a reconstruction loss function, an adversarial loss function, a facial component loss function, and an identity preservation loss function, so as to obtain the target face restoration model;
[0040] and / or,
[0041] The generative adversarial network includes StyleGAN V2;
[0042] and / or,
[0043] The preset convolutional neural network includes U-Net.
[0044] The present disclosure also provides a face restoration system, the face restoration system comprising:
[0045] A face restoration module is used to input the initial face image into the face restoration model to obtain a target face image;
[0046] Wherein, the face restoration model is obtained based on the training system of the face restoration model described above.
[0047] The present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and used to run on the processor, and when the processor executes the computer program, the training method of the face restoration model or the face restoration method described above is implemented.
[0048] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the training method of the face restoration model or the face restoration method described above is implemented.
[0049] The present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements the training method for a face restoration model or the face restoration method as described above.
[0050] On the basis of being in accordance with the common sense in the art, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present disclosure.
[0051] The positive and progressive effects of this disclosure are:
[0052] The present invention connects a face restoration module including a depthwise separable modulation convolution layer and a noise reduction and removal module including a depthwise separable layer to obtain a target face restoration model. The model adopts a depthwise separable convolution that is more suitable for edge devices, and the number of parameters is only one-sixth of that of GFPGAN, which significantly reduces the amount of calculation and parameters of the model, reduces the consumption of computing resources, improves operating efficiency, is more suitable for edge devices, and improves universality. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 This is a flowchart of the training method of the face restoration model of Embodiment 1 of the present disclosure;
[0054] Figure 2 This is a flowchart of the training method of the face restoration model of Embodiment 2 of the present disclosure;
[0055] Figure 3 This is a structural diagram of a face restoration module according to Embodiment 2 of the present disclosure;
[0056] Figure 4 This is a structural schematic diagram of a noise reduction and removal module according to Embodiment 2 of the present disclosure;
[0057] Figure 5 This is a schematic diagram of the structure of the target face restoration model of Embodiment 2 of the present disclosure;
[0058] Figure 6 This is a module diagram of a training system for a face restoration model according to Embodiment 4 of the present disclosure;
[0059] Figure 7 This is a module diagram of a training system for a face restoration model according to Embodiment 5 of the present disclosure;
[0060] Figure 8 This is a schematic diagram of the structure of an electronic device according to Embodiment 7 of the present disclosure. DETAILED DESCRIPTION
[0061] The present disclosure is further described below by way of examples, but the present disclosure is not limited to the scope of the examples.
[0062] Prefixes such as "first" and "second" are used in the embodiments of the present disclosure only to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. The use of prefixes such as ordinal numbers to distinguish description objects in the embodiments of the present disclosure does not constitute a limitation on the described objects. For the statement of the described objects, please refer to the description in the context of the claims or embodiments, and no unnecessary limitation should be constituted due to the use of such prefixes. In addition, in the description of the present embodiment, unless otherwise specified, the meaning of "plurality" is two or more.
[0063] In the embodiments of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0064] Example 1
[0065] This embodiment provides a training method for a face restoration model. Figure 1 As shown, the training method includes:
[0066] S101, distilling a face restoration module based on a pre-trained generative adversarial network;
[0067] Among them, the face restoration module includes a depth-separable modulated convolutional layer, and the face restoration module is used to obtain prior information of face generation;
[0068] S102, obtaining a noise reduction and removal module based on a preset convolutional neural network training;
[0069] The preset convolutional neural network includes a depth separable layer, and the noise reduction and removal module is used to perform noise reduction on the face image;
[0070] S103, connecting the potential features of the face restoration module and the noise reduction removal module based on the channel segmentation feature transformation layer to obtain a target face restoration model.
[0071] Specifically, the principle of depthwise separable convolution is to decompose the standard convolution into depthwise convolution and pointwise convolution, which can achieve effective dimensionality reduction of standard convolution.
[0072] The generative adversarial network can be pre-trained in FFHQ (Flickr-Faces-HQ Dataset, a high-quality face dataset), and the face restoration module MobileStyleGAN is obtained by distillation based on the pre-trained generative adversarial network, so that the face restoration module implicitly contains the prior information of face generation, and the face restoration module is a lightweight generative model. The generative adversarial network includes StyleGAN V2, and the preset convolutional neural network includes U-Net.
[0073] A lightweight U-Net is designed using the axial depth-separable layer as the denoising module MobileU-Net to obtain a preliminary denoised image. For example, the axial depth-separable layer proposed by U-Lite (a lightweight U-Net-based model) is used to design a MobileU-Net with an input image size of 256×256 and an output size of 256×256.
[0074] In order to enable the face restoration module MobileStyleGAN to fully utilize the potential features of the noise reduction and removal module MobleU-Net, the CS-SFT (Channel Segmentation Spatial Feature Transform) layer used in the GFPGAN model is adopted to connect the multi-scale potential features of the two modules from coarse to fine to obtain the final target face restoration model MobileGFPGAN.
[0075] In this embodiment, a face restoration module including a depthwise separable modulation convolution layer and a noise reduction and removal module including a depthwise separable layer are connected to obtain a target face restoration model. A depthwise separable convolution that is more suitable for edge devices is adopted, and the number of parameters is only one-sixth of that of GFPGAN, which significantly reduces the amount of computation and parameters of the model, reduces the consumption of computing resources, improves operating efficiency, makes it more suitable for edge devices, and improves universality.
[0076] Example 2
[0077] This embodiment provides a training method for a face restoration model, which is a further improvement on Embodiment 1.
[0078] In one feasible solution, Figure 2 As shown, step S101 includes:
[0079] S1011, using the generative adversarial network as a teacher model and the face restoration module as a student model to train and obtain an initial restoration module;
[0080] S1012: Based on the image scale requirement, the building blocks and weights of the initial restoration module are adjusted to obtain a face restoration module.
[0081] Specifically, the image scale requirement, for example, a resolution of 512x512, is used. According to the image scale requirement, the building block of the initial restoration module is removed. The building block is the last layer of the initial restoration module. Only the generated result with an output size resolution of 512x512 is retained, and the weight information of the last layer in the training weight is removed.
[0082] like Figure 3 As shown, the face restoration module MobileStyleGAN includes several structures including latent feature z, feature mapping network MappingNetwork, modulated convolution ModulatedConv2d, depth-separable modulated convolution ModulatedDWConv2d, inverse discrete wavelet transform upsampling IDWT Upsample and toFreq.
[0083] The potential feature z is normalized and then sent to the feature mapping network MappingNetwork.
[0084] The feature mapping network Mapping Network consists of seven layers of fully connected layers FC, and the output features w are sent to the modulated convolution ModulatedConv2d and the depth-separable modulated convolution ModulatedDWConv2d respectively.
[0085] ModulatedConv2d adopts the modulated convolution in StyleGAN V2.
[0086] The depth-separable modulated convolution ModulatedDWConv2d adopts the depth-separable modulated convolution ModulatedDWConv2d in MobileStyleGAN.
[0087] Inverse discrete wavelet transform upsampling IDWT Upsample uses the inverse discrete wavelet transform upsampling in MobileStyleGAN.
[0088] toFreq adopts the modulated convolution in StyleGAN.
[0089] In this solution, the initial restoration module is adjusted according to the image scale requirement to obtain a face restoration module, which meets the scale requirement of image generation and improves the user experience.
[0090] In one feasible solution, step S102 includes:
[0091] S1021. Setting the encoding module, bottleneck module and decoding module in the preset convolutional neural network to a group normalization mode to obtain a noise reduction and removal module.
[0092] Specifically, Figure 4 As shown, the noise reduction module includes several structures including an encoding module Encode Block, a bottleneck module BottleNeck Block and a decoding module Decode Block. In order to overcome the influence of small Batch Size (batch processing) on model prediction, the normalization method of Group Normalization is adopted.
[0093] The encoding module Encode Block uses the modified Encode Block in U-Lite. Since BatchNormalization does not work well under small Batch Size, the Batch Normalization of EncodeBlock in U-Lite is changed to Group Normalization to obtain a more stable output result. Among them, DW Conv, PW Conv and GN represent depth convolution, point-wise convolution and group processing normalization respectively, and Axial DW Conv is the axial depth convolution used in U-Lite. GLUE represents the activation function.
[0094] The bottleneck module BottleNeck Block adopts the modified BottleNeck Block in U-Lite. The Batch Normalization of the BottleNeck Block in U-Lite is changed to Group Normalization.
[0095] The decoding module Decode Block uses the modified Decode Block in U-Lite. The Batch Normalization of the Decode Block in U-Lite is changed to Group Normalization. Upsample uses the upsampling method of PyTorch (an open source machine learning library and a deep learning framework).
[0096] In this solution, the encoding module, bottleneck module and decoding module in the preset convolutional neural network are set to group normalization mode to obtain a denoising module, which overcomes the problem of poor training effect under small batch size and can effectively run in small batch training data. It is very suitable for model training under resource-constrained conditions, and can obtain more stable output results, thereby improving the stability, reliability and universality of the denoising module.
[0097] In one feasible solution, step S103 includes:
[0098] S1031, connecting the potential features of the face restoration module and the noise reduction removal module based on the channel segmentation feature transformation layer to obtain an initial face restoration model;
[0099] S1032. Use at least one of a reconstruction loss function, an adversarial loss function, a facial component loss function, and an identity preservation loss function to train the initial face restoration model to obtain a target face restoration model.
[0100] Specifically, the reconstruction loss function (Restoration Loss) uses the L1 loss function and the perceptual loss function L per , the formula is as follows:
[0101]
[0102] Among them, L rec represents the reconstruction loss function, Φ represents the pre-trained VGG-19 (a deep convolutional neural network) using the feature maps of {conv1, ..., conv5} before activation; λ l1 and λ per Represent the loss weights of L1 loss function and perceptual loss function respectively. represents the face image predicted by the face restoration model, y represents the real face image. ||·|| represents the L1 norm.
[0103] Adversarial Loss encourages the model to generate realistic images. The formula is as follows:
[0104]
[0105] Among them, L adv represents the adversarial loss function, E represents the expected value, D represents the discriminator, and λ adv represents the loss weight of the adversarial function. Softplus(·) represents the activation function.
[0106] The facial component loss function introduces component discriminators for the left eye, right eye, and mouth, and uses the Gram matrix to calculate the feature similarity of the facial components of the generated image and the real image. The formula is as follows:
[0107]
[0108] Among them, L comp represents the facial component loss function, ROI represents the region of interest from the component set {left eye, right eye, mouth}, and D ROI represents the local discriminator for each region of interest, ψ represents the multi-resolution features from the learned discriminator, and λ local and λ fs Represent the loss weights of the local discriminant loss function and the feature style loss function respectively.
[0109] The identity preserving loss function uses the pre-trained ArcFace (a face recognition model) to maintain the consistency of the facial features of the generated image and the real image. The formula is as follows:
[0110]
[0111] Among them, L id represents the identity preservation loss function, η represents ArcFace, λ id Represents the loss weight of the identity preserving loss function.
[0112] The overall loss function is the sum of the above loss functions, and the formula is as follows:
[0113]
[0114] Among them, L total Represents the overall loss function.
[0115] According to actual needs, at least one of the reconstruction loss function, adversarial loss function, facial component loss function and identity preservation loss function can be adopted, or the sum of the reconstruction loss function, adversarial loss function, facial component loss function and identity preservation loss function can be used as the overall loss function, and the initial face restoration model can be trained in the face dataset to obtain the trained target face restoration model MobileGFPGAN.
[0116] like Figure 5 As shown in the figure, the target face restoration model MobileGFPGAN includes a denoising removal module DegradationRemoval, a pretrained face restoration module MobileStyleGAN, a channel segmentation spatial feature transform (CS-SFT) layer, an adaptive average pooling (AVG) layer, and a multi-layer perceptron MLP, an inverse discrete wavelet transform upsampling IDWTUpsample, a face discriminator, a mouth discriminator, an eye discriminator, and a face recognition extractor Face RecognitionFeature Extractor.
[0117] The channel segmentation spatial feature transform (CS-SFT) layer adopts the channel segmentation spatial feature transform (CS-SFT) layer in GFPGAN.
[0118] Adaptive average pooling (AVG) layers and multi-layer perceptrons (MLPs) are used to adjust the dimensions of latent features to further reduce the number of parameters.
[0119] Inverse discrete wavelet transform upsampling IDWT Upsample uses the inverse discrete wavelet transform upsampling in MobileStyleGAN.
[0120] D_global is the discriminator for the entire face, using the discriminator of StyleGAN v2.
[0121] D_mouth is the mouth discriminator, which adopts the mouth discriminator in GFPGAN.
[0122] D_eye is the eye discriminator, which adopts the eye discriminator in GFPGAN.
[0123] The Face Recognition Feature Extractor uses the pre-trained face recognition model ArcFace.
[0124] In the training process of the target face restoration model, the low-quality (LQ) face image is used to extract latent features (F_latent) and spatial features (F_spatial) through the Degradation Removal module, and then the dimension of the latent features is adjusted through the adaptive average pooling (AVG) layer and the multi-layer perceptron MLP, and the latent features are mapped to the latent code (Latent code), and the feature F_GAN is generated according to the latent code. The F_GAN is spatially modulated by the extracted spatial features, and a high-quality (HQ) face image is obtained based on the channel segmentation spatial feature transform (CS-SFT) layer and IDWT (Inverse Discrete Wavelet Transform). For the high-quality (HQ) face image, the ROI (Region of Interest) align is used to extract the features of the region of interest and perform alignment operations.
[0125] In this scheme, the target face restoration model is trained through various types of loss functions to ensure the accuracy and reliability of the target face restoration model.
[0126] The working principle of the training method of the face restoration model of this embodiment is explained below with reference to specific examples:
[0127] The pre-trained generative adversarial network StyleGAN V2 is used as the teacher model, and the face restoration module is used as the student model to train the initial restoration module; based on the image scale requirement, the building blocks and weights of the initial restoration module are adjusted to obtain the face restoration module; the encoding module, the bottleneck module and the decoding module in the preset convolutional neural network U-Net are set to the group normalization mode to obtain the denoising and removal module; the potential features of the face restoration module and the denoising and removal module are connected based on the channel segmentation feature transformation layer to obtain the initial face restoration model; the initial face restoration model is trained using at least one of the reconstruction loss function, the adversarial loss function, the facial component loss function and the identity preservation loss function to obtain the target face restoration model.
[0128] In this embodiment, a face restoration module including a depthwise separable modulation convolution layer and a noise reduction and removal module including a depthwise separable layer are connected to obtain a target face restoration model. A depthwise separable convolution that is more suitable for edge devices is adopted, and the number of parameters is only one-sixth of that of GFPGAN, which significantly reduces the amount of computation and parameters of the model, reduces the consumption of computing resources, improves operating efficiency, makes it more suitable for edge devices, and improves universality.
[0129] Example 3
[0130] This embodiment provides a face restoration method, which includes:
[0131] Input the initial face image into the face restoration model to obtain the target face image;
[0132] The face restoration model is obtained based on the training method of the face restoration model in Example 1 or Example 2.
[0133] Specifically, the trained face restoration model MobileGFPGAN is used for face restoration. The FaceRestoreHelper (a utility class provided in the FaceXLib library) in the FaceXLib library (an open source library for face-related tasks) is used to identify the faces in the initial face image and crop them separately; the interpolate (a function in the torch.nn.functional library) method in the torch.nn.functional library (an important submodule in the PyTorch framework) is used to scale the cropped faces to 256×256 size; the scaled cropped faces are input into the face restoration model MobileGFPGAN to obtain the restored cropped face images, and then put back to the position of the original initial face image; Real-ESRGAN (an advanced image super-resolution technology) is used to clarify the background of the overall image to obtain the target face image.
[0134] In this embodiment, the initial face image is input into the face restoration model to obtain the target face image, thereby improving the quality of the target face image and enhancing the user experience.
[0135] Example 4
[0136] This embodiment provides a training system for a face restoration model. Figure 6 As shown, the training system includes:
[0137] Distillation module 1, used to obtain a face restoration module based on the pre-trained generative adversarial network distillation;
[0138] Among them, the face restoration module includes a depth-separable modulated convolutional layer, and the face restoration module is used to obtain prior information of face generation;
[0139] Training module 2, used for obtaining a noise reduction and removal module based on a preset convolutional neural network training;
[0140] The preset convolutional neural network includes a depth separable layer, and the noise reduction and removal module is used to perform noise reduction on the face image;
[0141] The connection module 3 is used to connect the potential features of the face restoration module and the noise reduction removal module based on the channel segmentation feature transformation layer to obtain a target face restoration model.
[0142] In this implementation, a face restoration module including a depthwise separable modulation convolution layer and a noise reduction and removal module including a depthwise separable layer are connected to obtain a target face restoration model. A depthwise separable convolution that is more suitable for edge devices is adopted, and the number of parameters is only one-sixth of that of GFPGAN, which significantly reduces the amount of computation and parameters of the model, reduces the consumption of computing resources, improves operating efficiency, makes it more suitable for edge devices, and improves universality.
[0143] Example 5
[0144] This embodiment provides a training system for a face restoration model, which is a further improvement on Embodiment 4.
[0145] In one feasible solution, Figure 7 As shown, the distillation module 1 comprises:
[0146] A training unit 11 is used to use the generative adversarial network as a teacher model and the face restoration module as a student model to train and obtain an initial restoration module;
[0147] The adjustment unit 12 is used to adjust the building blocks and weights of the initial restoration module based on the image scale requirement to obtain a face restoration module.
[0148] In an implementable solution, the training module 2 is also used to set the encoding module, bottleneck module and decoding module in the preset convolutional neural network to a group normalization mode to obtain a noise reduction and removal module.
[0149] In an implementable solution, the connection module 3 includes:
[0150] A connection unit 31, used for connecting the potential features of the face restoration module and the noise reduction removal module based on the channel segmentation feature transformation layer to obtain an initial face restoration model;
[0151] The loss training unit 32 is used to train the initial face restoration model using at least one of a reconstruction loss function, an adversarial loss function, a facial component loss function and an identity preservation loss function to obtain a target face restoration model.
[0152] In one feasible solution, the generative adversarial network includes StyleGAN V2.
[0153] In one feasible solution, the preset convolutional neural network includes U-Net.
[0154] In this implementation, a face restoration module including a depthwise separable modulation convolution layer and a noise reduction and removal module including a depthwise separable layer are connected to obtain a target face restoration model. A depthwise separable convolution that is more suitable for edge devices is adopted, and the number of parameters is only one-sixth of that of GFPGAN, which significantly reduces the amount of computation and parameters of the model, reduces the consumption of computing resources, improves operating efficiency, makes it more suitable for edge devices, and improves universality.
[0155] Example 6
[0156] This embodiment provides a face restoration system, which includes:
[0157] A face restoration module is used to input the initial face image into the face restoration model to obtain a target face image;
[0158] The face restoration model is obtained based on the training system of the face restoration model in Example 5.
[0159] In this embodiment, the initial face image is input into the face restoration model to obtain the target face image, thereby improving the quality of the target face image and enhancing the user experience.
[0160] As for the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The system embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution.
[0161] Example 7
[0162] Figure 8 This is a schematic diagram of the structure of an electronic device shown in an example embodiment of the present disclosure, the electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor, and when the processor executes the computer program, it implements the training method of the face restoration model or the face restoration method described in any of the above embodiments. Figure 8 The electronic device 90 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0163] like Figure 8As shown, the electronic device 90 may be in the form of a general-purpose computing device, for example, it may be a server device. The components of the electronic device 90 may include, but are not limited to: at least one processor 91, at least one memory 92, and a bus 93 connecting different system components (including the memory 92 and the processor 91).
[0164] The bus 93 includes a data bus, an address bus, and a control bus.
[0165] The memory 92 may include a volatile memory, such as a random access memory (RAM) 921 and / or a cache memory 922 , and may further include a read-only memory (ROM) 923 .
[0166] The memory 92 may also include a program tool 925 (or utility) having a set (at least one) of program modules 924, such program modules 924 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.
[0167] The processor 91 executes various functional applications and data processing by running the computer program stored in the memory 92, such as the training method of the face restoration model or the face restoration method provided in any of the above embodiments.
[0168] The electronic device 90 may also communicate with one or more external devices 94 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 95. Furthermore, the electronic device 90 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 96. As shown, the network adapter 96 communicates with other modules of the electronic device 90 via a bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 90, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.
[0169] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.
[0170] Example 8
[0171] The embodiments of the present disclosure also provide a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the training method for a face restoration model or the face restoration method provided in any of the above embodiments is implemented.
[0172] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.
[0173] Example 9
[0174] The present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for training a face restoration model or methods for restoring a face.
[0175] Among them, the program code for executing the computer program product of the present disclosure can be written in any combination of one or more programming languages, and the program code can be executed completely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or completely on the remote device.
[0176] Although the specific embodiments of the present disclosure are described above, those skilled in the art should understand that this is only an example, and the protection scope of the present disclosure is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present disclosure, but these changes and modifications all fall within the protection scope of the present disclosure.
Claims
1. A training method for a face restoration model, characterized in that: The training method comprises: The face restoration module is obtained by distillation based on the pre-trained generative adversarial network; Wherein, the face restoration module includes a depth-separable modulated convolutional layer, and the face restoration module is used to obtain prior information of face generation; A noise reduction and removal module is obtained based on the preset convolutional neural network training; Wherein, the preset convolutional neural network includes a depth separable layer, and the noise reduction and removal module is used to perform noise reduction processing on the face image; The potential features of the face restoration module and the noise reduction removal module are connected based on the channel segmentation feature transformation layer to obtain a target face restoration model.
2. The training method for a face restoration model according to claim 1, characterized in that: The step of obtaining a face restoration module based on distillation of a pre-trained generative adversarial network comprises: The generative adversarial network is used as a teacher model, and the face restoration module is used as a student model to train and obtain an initial restoration module; Based on the image scale requirement, the building blocks and weights of the initial restoration module are adjusted to obtain the face restoration module.
3. The training method for a face restoration model according to claim 1, wherein: The step of obtaining a noise reduction and removal module based on a preset convolutional neural network training comprises: The encoding module, bottleneck module and decoding module in the preset convolutional neural network are set to a group normalization mode to obtain the noise reduction and removal module.
4. The training method for a face restoration model according to any one of claims 1 to 3, characterized in that: The step of connecting the potential features of the face restoration module and the noise reduction removal module based on the channel segmentation feature transformation layer to obtain a target face restoration model includes: Connecting the potential features of the face restoration module and the noise reduction removal module based on the channel segmentation feature transformation layer to obtain an initial face restoration model; Using at least one of a reconstruction loss function, an adversarial loss function, a facial component loss function, and an identity preservation loss function to train the initial face restoration model to obtain the target face restoration model; and / or, The generative adversarial network includes StyleGAN V2; and / or, The preset convolutional neural network includes U-Net.
5. A face restoration method, characterized in that: The face restoration method comprises: Input the initial face image into the face restoration model to obtain the target face image; The face restoration model is obtained based on the training method of the face restoration model described in any one of claims 1 to 4.
6. A training system for a face restoration model, characterized in that: The training system comprises: A distillation module is used to obtain a face restoration module based on the pre-trained generative adversarial network distillation; Wherein, the face restoration module includes a depth-separable modulated convolutional layer, and the face restoration module is used to obtain prior information of face generation; A training module, used for obtaining a noise reduction and removal module based on a preset convolutional neural network training; Wherein, the preset convolutional neural network includes a depth separable layer, and the noise reduction and removal module is used to perform noise reduction processing on the face image; A connection module is used to connect the potential features of the face restoration module and the noise reduction removal module based on the channel segmentation feature transformation layer to obtain a target face restoration model.
7. A face restoration system, characterized in that: The face restoration system comprises: A face restoration module is used to input the initial face image into the face restoration model to obtain a target face image; Wherein, the face restoration model is obtained based on the training system of the face restoration model described in claim 6.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and used to run on the processor, characterized in that: When the processor executes the computer program, it implements the training method of the face restoration model described in any one of claims 1 to 4, or implements the face restoration method described in claim 5.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the training method of the face restoration model described in any one of claims 1 to 4, or implements the face restoration method described in claim 5.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the training method for a face restoration model as described in any one of claims 1 to 5, or implements the face restoration method as described in claim 5.