Face image restoration model training method

Abstract

The invention is suitable for the technical field of image processing, and provides a training method of a face image restoration model. The training method comprises the following steps: acquiring a training sample and a face key point sample image corresponding to the training sample; inputting the face sample image and the face key point sample image into an initial face image restoration model to trigger a generation network to extract face identity features of the face sample image constrained by a first feature extraction network and face key point features of the face key point sample image constrained by a second feature extraction network, predicting a restored sample image corresponding to the face sample image; and training the initial face image restoration model based on the restoration sample image and the restoration reference image to obtain a target face image restoration model, the target face image restoration model being used for performing face image restoration on the to-be-restored face image. According to the invention, high-quality face image restoration can be realized.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a training method for a facial image restoration model. Background Art

[0002] Facial image restoration aims to restore high-quality facial images from low-quality facial images, and facial image restoration models are usually used for facial image restoration. However, existing facial image restoration models still have some limitations. For example, existing facial image restoration models usually only restore facial images based on one prior information, which makes it difficult to achieve high-quality facial image restoration. Summary of the invention

[0003] The embodiment of the present application provides a training method for a facial image restoration model, which can achieve high-quality facial image restoration.

[0004] In a first aspect, an embodiment of the present application provides a training method for a facial image restoration model, wherein the initial facial image restoration model includes: a first feature extraction network, a second feature extraction network connected to the first feature extraction network, and a generation network, wherein the training method includes: Acquire a training sample and a facial key point sample image corresponding to the training sample, wherein the training sample includes a facial sample image and a restored reference image corresponding to the facial sample image, and the image quality of the restored reference image is higher than that of the facial sample image; Inputting the facial sample image and the facial key point sample image into the initial facial image restoration model to trigger the generation network to predict a restored sample image corresponding to the facial sample image based on the facial identity features of the facial sample image constrained by the first feature extraction network and the facial key point features of the facial key point sample image constrained by the second feature extraction network; Based on the restored sample image and the restored reference image, the initial facial image restoration model is trained to obtain a target facial image restoration model, and the target facial image restoration model is used to perform facial image restoration on the facial image to be restored.

[0005] In an embodiment of the present application, the initial facial image restoration model includes a first feature extraction network, a second feature extraction network connected to the first feature extraction network, and a generation network. By obtaining training samples and facial key point sample images corresponding to the training samples, and inputting the facial sample images and facial key point sample images in the training samples into the initial facial image restoration model, the generation network can be triggered to predict the restored sample image corresponding to the facial sample image based on the facial identity features of the facial sample images constrained by the first feature extraction network and the facial key point features of the facial key point sample images constrained by the second feature extraction network. Based on the restored sample images and the restored reference images in the training samples, the initial facial image restoration model can be trained to obtain a target facial image restoration model. This solution uses the facial identity features of the facial sample image constrained by the first feature extraction network to enable the generation network to provide high-quality restoration effects while keeping the facial identity features unchanged. The facial key point features constrained by the second feature extraction network can assist the generation network in capturing the positions of facial key points, focusing on the restoration effect of facial key points. Therefore, the generation network trains the initial facial image restoration model based on the facial identity features constrained by the extraction network and the facial key point features constrained by the second feature extraction network, which can enable the target facial image restoration model to provide high-quality restoration effects when restoring the facial image to be restored. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0007] Figure 1 It is a flowchart of a training method for a facial image restoration model provided in an embodiment of the present application; Figure 2 is a structural example diagram of an initial facial image restoration model provided in an embodiment of the present application; Figure 3 is a structural example diagram of a down-sampling residual block provided in an embodiment of the present application; Figure 4 is a structural example diagram of an upsampling residual block provided in an embodiment of the present application; Figure 5 is a structural example diagram of a style convolutional network provided in an embodiment of the present application; Figure 6 is a diagram showing an example structure of a conditional control network provided in an embodiment of the present application; Figure 7is a structural example diagram of a fusion module provided in an embodiment of the present application; Figure 8 is another structural example diagram of the style convolutional network provided in an embodiment of the present application; Fig. 9 It is a flowchart of a method for generating training samples provided in an embodiment of the present application; Figure 10-1 is an example of a face image; Figure 10-2 This is an example diagram of a standard template; Figure 10-3 This is an example of a facial image after alignment; Figure 10-4 is an example image of a face sample image; Figure 10-5 is an example image of the restored reference image; Fig.11 It is a flowchart of a facial image restoration method provided in an embodiment of the present application; Fig.12 It is a structural schematic diagram of a training device for a facial image restoration model provided in an embodiment of the present application; Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0008] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0009] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0010] It should be noted that the information collection process (such as facial image collection process) / feature extraction process involved in this application is performed with the user's knowledge and permission, that is, the information collection process / feature extraction process complies with the requirements of laws and regulations and does not constitute an act that harms the public interest.

[0011] Existing facial image restoration models usually include a priori networks for various facial parts. Given a low-quality facial image, three part sub-images are obtained, which are mapped to the high-quality feature z space corresponding to each part through their respective encoders. Subsequently, the restored part results are obtained through the part prior network, and each restored part is restored to the overall facial image to obtain the final restored result. The problem with this method is that it is not an end-to-end method. At the same time, because it needs to combine the priori results of different parts and relies on the accuracy of part detection, when accuracy problems arise, abnormalities are prone to occur.

[0012] The facial image restoration model provided by the embodiment of the present application includes a first feature extraction network, a second feature extraction network connected to the first feature extraction network, and a generation network. The facial identity features of the facial sample image constrained by the first feature extraction network can enable the generation network to provide a high-quality restoration effect while maintaining the facial identity features unchanged. The facial key point features constrained by the second feature extraction network can assist the generation network in capturing the positions of facial key points, focusing on the restoration effect of facial key points. Therefore, the generation network trains the initial facial image restoration model based on the facial identity features constrained by the extraction network and the facial key point features constrained by the second feature extraction network, so that the target facial image restoration model can provide a high-quality restoration effect when restoring the facial image to be restored, without relying on the detection of different facial parts. The facial image restoration model provided by the embodiment of the present application is an end-to-end model, which can automate feature learning, reduce manual intervention, simplify model design, improve the overall performance of the system, and has better generalization ability.

[0013] Existing solutions usually need to fuse the three-dimensional shape information or two-dimensional segmentation map information of the image (that is, classify each pixel in the image into a certain part). The three-dimensional shape information and the two-dimensional segmentation map information have a relatively large impact on the final restoration result, that is, the final result is more dependent on the accuracy of the prediction of the three-dimensional shape information and the two-dimensional segmentation map information, and it is more likely to have inaccurate results due to prediction errors. However, the feature point map (such as facial identity features and facial key point features) in the embodiment of the present application is a relatively coarse geometric constraint and will deviate the feature points to a certain extent in actual training, making the model more robust to prediction errors. In addition, the facial image restoration model provided in the embodiment of the present application adopts the form of learnable latent coding to provide high-quality image features, and the model can be implemented on embedded devices.

[0014] The training method of the facial image restoration model provided in the embodiment of the present application can be applied to electronic devices such as mobile phones, tablet computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), servers, imaging devices (such as cameras, camcorders, etc.). The embodiment of the present application does not impose any restrictions on the specific types of electronic devices.

[0015] See also Figure 1 , Figure 1 A flow chart of a method for training a facial image restoration model provided by an embodiment of the present application is shown. The initial facial image restoration model to be trained includes a first feature extraction network, a second feature extraction network connected to the first feature extraction network, and a generation network. As an example but not a limitation, the method is applied to an electronic device, and the method includes the following steps: Step 101: Obtain training samples and facial key point sample images corresponding to the training samples.

[0016] The training samples include facial sample images and restored reference images corresponding to the facial sample images, and the image quality of the restored reference images is higher than that of the facial sample images. The facial sample images may refer to low-quality data used for training and evaluating facial image restoration models. The restored reference images may refer to accurate labels or high-quality data used for training and evaluating facial image restoration models.

[0017] In order to distinguish the facial image restoration model to be trained and the trained facial image restoration model, the facial image restoration model to be trained can be referred to as the initial facial image restoration model, and the trained facial image restoration model (i.e., the trained facial image restoration model) can be referred to as the target facial image restoration model. It should be understood that the facial image restoration model to be trained and the trained facial image restoration model can also be named in other ways, and this application does not limit this.

[0018] In order to better optimize the model parameters of the initial facial image restoration model, reduce the risk of overfitting, and improve the generalization ability of the model, multiple training samples can be used to train the initial facial image restoration model. Of course, it is understandable that one training sample can also be used to train the initial facial image restoration model, and this application does not limit this. A training sample includes a facial sample image and a restored reference image corresponding to the facial sample image.

[0019] It should be noted that the face corresponding to the training sample can be any face with identity features, such as a human face or an animal face.

[0020] As an example but not limitation, if a face restoration model is used to restore a human face image, then the face corresponding to the training sample is a human face, and the face sample images of different training samples can be face sample images of the same human face or face sample images of different human faces. Using the training samples corresponding to the human face to train the initial face image restoration model can enable the trained target face image restoration model to restore human face images. If a face restoration model is used to restore the face images of a certain type of animal, then the face corresponding to the training sample is the face of the animal, and the face sample images of different training samples can be face sample images of the same animal in the animal, or face sample images of different animals in the animal, and using the training samples corresponding to the animal's face to train the initial face image restoration model can enable the trained target face image restoration model to restore animal face images.

[0021] The above training samples and the corresponding facial key point sample images correspond to the same face and have the same size. The facial key points in the facial key point sample images can be the eyes, nose, mouth and other key points of the face. In order to reduce the influence of other facial areas (i.e., areas other than the facial key points) on model training, the pixel values ​​of the pixels in the facial key point sample images where the facial key points are located are 255, and the pixel values ​​of the pixels in the facial key point sample images where the facial key points are located are 0.

[0022] Step 102, inputting the facial sample image and the facial key point sample image into the initial facial image restoration model to trigger the generation network to predict a restored sample image corresponding to the facial sample image based on the facial identity features of the facial sample image constrained by the first feature extraction network and the facial key point features of the facial key point sample image constrained by the second feature extraction network.

[0023] Among them, the above-mentioned inputting of the facial sample image and the facial key point sample image into the initial facial image restoration model may refer to inputting the facial sample image into the first feature extraction network and inputting the facial key point sample image into the second feature extraction network. Inputting the facial sample image into the first feature extraction network can extract and constrain the facial identity features of the facial sample image, so that the facial identity features of the restored sample image are consistent with the facial identity features of the facial sample image, that is, the facial identity features are kept unchanged. Inputting the facial key point sample image into the second feature extraction network can extract and constrain the facial key point features of the facial key point sample image, which can assist the generation network in capturing the positions of the facial key points, focusing on the restoration effect of the facial key points.

[0024] The generative network restores the initial facial image model based on the facial identity features constrained by the first feature extraction network and the facial key point features constrained by the second feature extraction network. It can restore the facial image while keeping the facial identity features unchanged, thereby providing high-quality restoration effects. The facial key point features constrained by the second feature extraction network can assist the generative network in capturing the positions of facial key points, focusing on the restoration effect of facial key points.

[0025] In a possible implementation, the first feature extraction network includes a downsampling residual block, a first fusion module and an upsampling residual block, and the generation network includes a convolution layer, and the convolution layer corresponds to a potential code, and the potential code is used to adjust the weight of the convolution layer of the generation network. On this basis, the above step 102 includes: The facial sample image is input into the down-sampling residual block to obtain a down-sampling feature map; the facial key point sample image is input into the second feature extraction network to obtain a facial key point feature map, and the facial key point attention of the down-sampling feature map is enhanced based on the facial key point feature map to obtain an image enhancement feature map, wherein the facial key point feature map includes the facial key point features of the facial key point sample image; the image enhancement feature map is input into the first fusion module to trigger the first fusion module to fuse the image enhancement feature map with the down-sampling feature map to obtain a fused feature map; the down-sampling feature map and the fused feature map are input into the up-sampling residual block to obtain an up-sampling feature map; the down-sampling feature map, the up-sampling feature map and the potential code are input into the generation network to obtain a restored sample image.

[0026] The second feature extraction network can achieve attention enhancement on the facial key points of the downsampled feature map by multiplying the facial key point feature map in the facial key point sample image and the downsampled feature map. The first fusion module can fuse the image enhancement feature map and the downsampled feature map, which can refer to splicing the image enhancement feature map and the downsampled feature map.

[0027] In another possible implementation, the first feature extraction network includes a downsampling residual block, a second fusion module and an upsampling residual block, and the generation network includes a convolution layer, and the convolution layer corresponds to a potential code, and the potential code is used to adjust the weight of the convolution layer of the generation network. On this basis, the above step 102 includes: The facial sample image is input into the down-sampling residual block to obtain a down-sampling feature map; the facial key point sample image is input into the second feature extraction network to obtain a facial key point feature map, the facial key point feature map includes the facial key point features of the facial key point sample image; the facial key point feature map is input into the second fusion module to trigger the second fusion module to perform attention enhancement of facial key points on the down-sampling feature map based on the facial key point feature map to obtain an image enhancement feature map, and the image enhancement feature map is fused with the down-sampling feature map to obtain a fused feature map; the down-sampling feature map and the fused feature map are input into the up-sampling residual block to obtain an up-sampling feature map; the down-sampling feature map, the up-sampling feature map and the potential code are input into the generation network to obtain a restored sample image.

[0028] The second fusion module can achieve attention enhancement on the facial key points of the downsampled feature map by multiplying the facial key point feature map in the facial key point sample image and the downsampled feature map. The second fusion module fusing the image enhancement feature map with the downsampled feature map can refer to splicing the image enhancement feature map with the downsampled feature map.

[0029] It should be understood that the downsampled feature map may refer to a feature map extracted by a downsampled residual block, which is an image that may include global features of a facial sample image. Global features of a facial sample image include, but are not limited to, color features, shape features, facial identity features, etc. Facial identity features may refer to features that can be individually identified, including but not limited to texture features, structural features of facial key points, etc. The upsampled feature map may refer to a feature map extracted by an upsampled residual block.

[0030] As an example but not limitation, the structure of the first feature extraction network can be a Unet structure or other network structures, which is not limited in this application. The generation network can be a pre-trained StyleGAN network, which can provide high-quality image features including low-frequency information, medium-frequency information and high-frequency information, thereby achieving better image restoration effects. The above-mentioned generation network may include a style convolutional network and a conditional control network. The convolutional layer included in the generation network may refer to the convolutional layer included in the style convolutional network.

[0031] The latent coding provided in this embodiment is a learnable parameter. The optimal latent coding (i.e., the trained latent coding) can be obtained through model training, which enables the model to retain high-quality generation priors and be applied on embedded devices.

[0032] As an example but not limitation, the first feature extraction network may include M downsampled residual blocks, M-1 first fusion modules or M-1 second fusion modules having the same resolution as the first M-1 downsampled residual blocks, and M-1 upsampled residual blocks corresponding to the first M-1 downsampled residual blocks, where M is an integer greater than 1. On this basis, the second feature extraction network may include M-1 downsampled residual blocks, and the generation network may include M-1 network blocks, each of which includes a style convolution network and a conditional control network. Among them, in the first feature extraction network, the resolution of the M downsampled residual blocks decreases successively, the resolution of the M-1 first fusion modules or the M-1 second fusion modules decreases successively, and the resolution of the M-1 upsampled residual blocks increases successively, and the first M-1 downsampled residual blocks may refer to downsampled residual blocks whose resolution is arranged in the M-1 position.

[0033] As an example but not a limitation, M is 4, and the first feature extraction network includes four downsampling residual blocks, three second fusion modules and three upsampling residual blocks. Based on the first feature extraction network, the facial sample image can be downsampled four times, that is, four downsampling residual blocks and three upsampling residual blocks are used. This means that this embodiment will discard some weights in the underlying generation prior, and only needs a high-scale generation prior, which can control the computational complexity of the first feature extraction network.

[0034] like Figure 2 The structure example diagram of the initial facial image restoration model provided by the embodiment of the present application is shown. For the sake of distinction, A1, A2, A3, and A4 are used to represent the four down-sampled residual blocks in the first feature extraction network, and the resolutions of the four down-sampled residual blocks are reduced in sequence. B1, B2, and B3 are used to represent the three second fusion modules in the first feature extraction network. C1, C2, and C3 are used to represent the three up-sampled residual modules in the first feature extraction network. D1, D2, and D3 are used to represent the three down-sampled residual blocks in the second feature extraction network. E1, E2, and E3 are used to represent the three network modules in the generation network. E11 is used to represent the style convolution network in E1, E12 is used to represent the conditional control network in E1, E21 is used to represent the style convolution network in E2, E22 is used to represent the conditional control network in E2, E31 is used to represent the style convolution network in E3, and E32 is used to represent the conditional control network in E3. The resolutions of A1, B1, C3, D1, E31, and E32 are the same, the resolutions of A2, B2, C2, D2, E21, and E22 are the same, and the resolutions of A3, B3, C1, D3, E31, and E32 are the same. Figure 2 When the initial facial image restoration model is shown, the facial sample image is input into A1, and the downsampled features output by A1 are Figure 1Input B1 and A2, input the facial key point sample image into D1, and output the facial key point features of D1 Figure 1 Input B1 and D2, B1 is based on facial key point features Figure 1 Downsample features Figure 1 Enhance the attention of facial key points to obtain image enhancement features Figure 1 , and enhance the image features Figure 1 and downsampled features Figure 1 Fusion is performed to obtain fusion features Figure 1 , and the fusion features Figure 1 Input C3; A2 downsamples the input features Figure 1 After processing, output downsampled features Figure 1 , and the downsampled features Figure 1 Input B2 and A3, D2 input facial key point features Figure 1 After processing, output facial key point features Figure 2 , and the facial key point features Figure 2 Input B2, B2 is based on facial key point features Figure 2 Downsample features Figure 2 Enhance the attention of facial key points to obtain image enhancement features Figure 2 , and enhance the image features Figure 2 and downsampled features Figure 2 Fusion is performed to obtain fusion features Figure 2 , and the fusion features Figure 2 Input C2; A3 downsamples the input features Figure 2 After processing, output downsampled features Figure 3 , and the downsampled features Figure 3 Input B3 and A4, D3 for the input facial key point features Figure 2 After processing, output facial key point features Figure 3 , and the facial key point features Figure 3 Input B3, B3 is based on facial key point features Figure 3 Downsample features Figure 3 Enhance the attention of facial key points to obtain image enhancement features Figure 3 , and enhance the image features Figure 3 and downsampled features Figure 3 Fusion is performed to obtain fusion features Figure 3 , and the fusion features Figure 3 Input C1; A4 downsamples the input features Figure 3 After processing, output downsampled features Figure 4 , and the downsampled features Figure 4 Input C1 and E11; C1 downsamples features Figure 4 and fusion features Figure 3After processing, we get the upsampled features Figure 1 , and the upsampled features Figure 1 Input C2 and E12; C2 upsamples features Figure 1 and fusion features Figure 2 After processing, we get the upsampled features Figure 2 , and the upsampled features Figure 2 Input C3 and E22; C3 upsamples features Figure 2 and fusion features Figure 1 After processing, we get the upsampled features Figure 3 , and the upsampled features Figure 3 Input E32; E11 downsamples the input features Figure 4 After processing the latent code corresponding to E11, the first high-quality feature is obtained Figure 1 , and the first high-quality feature Figure 1 Input E12; E12 is the first high-quality feature of the input Figure 1 and upsampled features Figure 1 After processing, the second highest quality feature is obtained Figure 1 , and the second highest quality feature Figure 1 Input E21; E21 has the second highest quality feature for the input Figure 1 After processing the potential code corresponding to E21, the first high-quality feature is obtained Figure 2 , and the first high-quality feature Figure 2 Input E22; E22 has the first high quality feature of the input Figure 2 and upsampled features Figure 2 After processing, the second highest quality feature is obtained Figure 2 , and the second highest quality feature Figure 2 Input E31; E31 is the second highest quality feature of the input Figure 2 The first high-quality feature after processing the latent code corresponding to E31 Figure 3 , and the first high-quality feature Figure 3 Input E32; E32 is the first high-quality feature of the input Figure 3 and upsampled features Figure 3 After processing, the restored sample image is obtained.

[0035] like Figure 3 The figure shows a structural example of a down-sampling residual block (i.e., a down-sampling residual block in the first feature extraction network and the second feature extraction network) provided in an embodiment of the present application. The down-sampling residual block is completed using a convolutional layer with a stride of 2 during down-sampling. Figure 4The structure example diagram of the upsampling residual block provided in the embodiment of the present application is shown. The upsampling residual block uses an upsample layer of the nearest neighbor interpolation method when upsampling. The downsampling residual block uses a convolution layer with a stride equal to 2 when downsampling and the upsampling residual block uses an upsample layer of the nearest neighbor interpolation method when upsampling, which can facilitate subsequent model deployment without sacrificing high-quality effects.

[0036] like Figure 5 The figure shows an example of the structure of the style convolutional network provided in the embodiment of the present application. The role of the style convolutional network is to adjust the style of the features extracted by the generative network, so that high-quality prior information of various styles can be obtained. At the same time, the richness of the texture is enhanced by adding random noise, and the potential code is transformed through the fully connected layer (FC layer) to control the weight of the convolutional layer. Figure 6 The figure shows an example structure diagram of the conditional control network provided in the embodiment of the present application. The conditional control network performs convolution transformation on the feature map extracted by the first extraction network at different resolutions, which can limit the generation network to provide high-quality features, thereby achieving the purpose of keeping the facial identity features unchanged. Figure 7 The figure shows an example of the structure of the fusion module provided in the embodiment of the present application. The fusion module can focus on restoring the key point details of the face by fusing the geometric prior (i.e., the facial key point features extracted by the second feature extraction network) and the global features extracted by the first feature extraction network, and fuse them in a splicing manner, so that the fusion module can more flexibly fuse the above two parts of the extracted features. Optionally, the Concat function can be used to implement feature splicing.

[0037] Step 103: Based on the restored sample image and the restored reference image, the initial facial image restoration model is trained to obtain a target facial image restoration model.

[0038] The target facial image restoration model is used to restore the facial image to be restored.

[0039] During the training process of the initial facial image restoration model, the model parameters of the initial facial image restoration model can be continuously updated based on the difference between the restored sample image and the restored reference image. The corresponding model parameters when the difference between the restored sample image and the restored reference image is minimized are the model parameters of the target facial image restoration model.

[0040] Among them, the above-mentioned training of the initial facial image restoration model can be to train the first feature extraction network, the second feature extraction network and the generation network. In this case, the model parameters of the above-mentioned initial facial image restoration model include the model parameters of the first feature extraction network, the model parameters of the second feature extraction network and the model parameters of the generation network; it can also be to train the first feature extraction network and the second feature extraction network. In this case, the generation network in the initial facial image restoration model is pre-trained, and the model parameters of the above-mentioned initial facial image restoration model include the model parameters of the first feature extraction network and the model parameters of the second feature extraction network; when the convolution layer of the generation network corresponds to a latent code, and the latent code is used to adjust the weight of the convolution layer of the generation network, the first feature extraction network, the second feature extraction network and the latent code can be trained to obtain the trained first feature extraction network, the trained target feature extraction network and the trained latent code, and the model parameters of the above-mentioned initial facial image restoration model include the model parameters of the first feature extraction network and the model parameters of the second feature extraction network. After obtaining the trained latent code, the trained latent code can be merged into the corresponding convolutional layer, that is, the weight of the corresponding convolutional layer is first adjusted using the trained latent code to obtain the adjusted weight, and the weight of the convolutional layer is updated to the adjusted weight, which can make the structure of the generated network simpler.

[0041] In order to facilitate the use of the model in embedded devices, this embodiment designs the latent code as a learnable parameter. When the trained latent code is obtained and deployed, the trained latent code can be merged into the corresponding convolution layer. When the added random noise is deployed, it can be set to a fixed value as the bias of the style convolution network. In this way, the structure of the target facial image restoration model can be made simpler while achieving high-quality restoration. Figure 8 The figure shows another structural example diagram of the style convolutional network provided in an embodiment of the present application, which is a structural example diagram after random noise and trained latent coding are deployed in the style convolutional network.

[0042] The deployment process of random noise and trained latent code can be expressed by the following formulas (1) and (2). Formula (1) represents Figure 5 The feature map processing process of the style convolutional network is shown in Figure 2. Formula (2) represents Figure 8 The feature map processing process of the style convolutional network shown.

[0043] (1) (2) in, represents convolution, In formula (1), and Represent the weights and biases of the FC layer, which are used to calculate the transformed potential code and (i.e., the weight of the convolutional layer) is adjusted, and the adjusted (The feature map received by the convolutional layer) is convolved and upsampled, and finally activated with random noise (using Figure 5 The Leaky ReLu activation function shown in ) is used to obtain the result Y. Since the embodiment of the present application uses a learnable latent code for training, in actual deployment, the formula (1) for The noise can also be merged into the bias of the convolution layer, which can greatly reduce the amount of calculation and the difficulty of deployment. By removing the convolution with variable weights, formula (1) is transformed into formula (2), and the calculation results of these two formulas are the same. is the potential encoding pair after training After adjustment, the adjusted weight is obtained. , represents the potential code after training. N in formula (1) and formula (2) represents the multiple of upsampling, and N is an integer greater than 1. As an example but not a limitation, N is 2.

[0044] In a possible implementation, when the initial facial image restoration model is trained based on the restored sample image and the restored reference image, the first feature extraction network, the second feature extraction network and the generation network may be trained. In this case, the model parameters of the initial facial image restoration model include the model parameters of the first feature extraction network, the model parameters of the second feature extraction network and the model parameters of the generation network. Alternatively, only the first feature extraction network and the second feature extraction network may be trained. In this case, the generation network is pre-trained, and the model parameters of the initial facial image restoration model include the model parameters of the first feature extraction network, the model parameters of the second feature extraction network and the potential code. The model parameters may include the weights and biases of the corresponding networks.

[0045] Based on the pre-trained generative network, the generative network can provide generative priors, thereby providing high-quality low-, medium-, and high-frequency image features to achieve better image restoration effects. Generative priors usually refer to feature information extracted from high-quality images. For example, a generative network can be used to map Gaussian distributions to high-quality data distributions, thereby obtaining high-quality generative priors on known distributions.

[0046] The facial image restoration model provided in the embodiment of the present application is a facial image restoration model based on deep learning, which combines geometric priors (i.e., facial key point features) and generative priors to provide stable high-quality image restoration results. At the same time, a learnable latent code is used to optimize an optimal latent code during training, retaining some high-quality generative priors. After adjusting the weights of the convolutional layer of the generative network, the target facial image restoration model can be used on some embedded devices that are not easy to deploy and have poor performance.

[0047] In a possible implementation, a loss value can be used to characterize the difference between the restored sample image and the restored reference image. On this basis, the above step 103 includes: calculating a first loss value, the first loss value is the loss value between the restored sample image and the restored reference image; obtaining a first perceptual feature map and a second perceptual feature map, the first perceptual feature map includes the perceptual features of the restored sample image, and the second perceptual feature map includes the perceptual features of the restored reference image; calculating a second loss value, the second loss value is the loss value between the first perceptual feature map and the second perceptual feature map; obtaining a first facial identity feature map and a second facial identity feature map, the first facial identity feature map includes the facial identity features of the restored sample image, and the second identity feature map includes the facial identity features of the restored reference image; calculating a third loss value, the third loss value is the loss value between the first facial identity feature map and the second facial identity feature map; and training the initial facial image restoration model based on the first loss value, the second loss value, and the third loss value.

[0048] The first loss value represents the pixel-level difference between the restored sample image and the restored reference image. The second loss value is the perceptual loss, which focuses on the high-level features of the image (such as texture features, shape features, etc.). The perceptual loss can better capture the subtle differences perceived by the human visual system, thereby generating high-quality images. The third loss value is the identity feature loss, which is used to maintain the original facial identity features of the facial sample image, so that the restored sample image and the facial sample image are as consistent as possible in terms of facial identity features.

[0049] Optionally, a Visual Geometry Group (VGG) may be used to extract the first perceptual feature map and the second perceptual feature map respectively. An Arcface network may be used to extract the first face identity feature map and the second face identity feature map respectively.

[0050] Optionally, a preset loss function may be used to calculate the first loss value, the second loss value, and the third loss value. This application does not limit the type of loss function.

[0051] As an example but not limitation, the first loss value may be calculated by formula (3), the second loss value may be calculated by formula (4), and the third loss value may be calculated by formula (5).

[0052] (3) in, represents the first loss value, and Respectively represent the height and width of the restored sample image or the restored reference image (the restored sample image and the restored reference image have the same height and width), represents the restored sample image, represents the restored reference image, represents a sample face image, Represents the coordinates of pixels in the restored sample image and the restored reference image, Indicates that the coordinates in the restored sample image are The pixel value of the pixel, Indicates that the coordinates in the restored reference image are The pixel value of the pixel.

[0053] (4) in, represents the second loss value, and Respectively represent the height and width of the first perceptual feature map or the second perceptual feature map (the first perceptual feature map and the second perceptual feature map have the same height and width), represents the first perceptual feature map, represents the second perceptual feature map, represents the coordinates of the pixels in the first perceptual feature map and the second perceptual feature map, Indicates that the coordinates in the first perceptual feature map are The pixel value of the pixel, Indicates that the coordinates in the second perceptual feature map are The pixel value of the pixel.

[0054] (5) in, represents the third loss value, and represent the height and width of the first face identity map or the second face identity map respectively (the first face identity map and the second face identity map have the same height and width), represents a first facial identity feature map, represents the second facial identity feature map, represents the coordinates of the pixels in the first face identity feature map and the second face identity feature map, Indicates that the coordinates in the first facial identity feature map are The pixel value of the pixel, Indicates that the coordinates in the second facial identity feature map are The pixel value of the pixel.

[0055] In one embodiment, when the initial facial image restoration model is trained based on the first loss value, the second loss value, and the third loss value, the first loss value, the second loss value, and the third loss value can be weighted and summed based on the weight of the first loss value, the weight of the second loss value, and the weight of the third loss value to obtain a total loss value, and the initial facial image restoration model is trained based on the total loss value. Optionally, the weight of the first loss value, the weight of the second loss value, and the weight of the third loss value can be set according to the scene requirements. As an example and not limitation, the weight of the first loss value is 1, the weight of the second loss value is 1, and the weight of the third loss value is 10.

[0056] As an example but not a limitation, in the training process of the initial face image restoration model, the optimizer used may be the Adam optimizer, the learning rate is 3e-4, and the batch size (batch_size) is 16, to train the initial model. Optionally, other optimizers, other learning rates, and other batch sizes may be selected for training according to actual needs.

[0057] During the training process, oscillation needs to be controlled. When the total loss value is too large, the gradient used for parameter update during back propagation will be relatively large. Adjusting a smaller learning rate can solve the training collapse problem caused by such a large gradient update, but it will greatly slow down the training speed. In one embodiment, a gradient clipping algorithm can be used to balance the training collapse problem and the problem of slowing down the training speed, stabilize the training process, and obtain a full-gain applicable facial image restoration model. Specifically, the maximum norm (norm) of the gradient of each parameter can be scaled to prevent it from affecting normal training, as shown in formula (6): (6) in, represents a preset maximum gradient, which can be set to 10 as an example but not a limitation. Represents the gradient with the largest absolute value among a batch of gradients, Represents the gradient of the total loss value compared to the parameter w in a gradient descent, Represents the clipped gradient (i.e. It should be understood that the parameter w can be a model parameter of the first feature extraction network and the second feature extraction network, or can be a potential code corresponding to the convolution layer in the generation network.

[0058] The embodiment of the present application uses the facial identity features of the facial sample image constrained by the first feature extraction network to enable the generation network to provide a high-quality restoration effect while keeping the facial identity features unchanged. The facial key point features constrained by the second feature extraction network can assist the generation network in capturing the positions of facial key points, focusing on the restoration effect of facial key points. Therefore, the generation network trains the initial facial image restoration model based on the facial identity features constrained by the extraction network and the facial key point features constrained by the second feature extraction network, which enables the target facial image restoration model to provide a high-quality restoration effect when restoring the facial image to be restored.

[0059] It should be noted that the facial image restoration model based on deep learning has a great influence on the restoration effect by the matching degree of the image, for example: whether the low-quality image conforms to the actual acquired image domain distribution, and whether the high-quality image conforms to the subjective visual perception of the evaluator. According to the actual image acquisition scenario, because the faces captured in reality will be affected by various conditions such as lighting, noise, and strong light, the actual situations that need to be processed to cause low-quality images are not uniform. Different situations have different impacts on the restoration process. For example: daytime lighting and noise have less impact on image effects than nighttime; faces with longer exposure times are more likely to be affected by motion blur than images with shorter exposure times; when the image is subsequently sharpened with a high intensity, the edges on the image are enhanced and the noise is amplified. Conversely, when the noise reduction parameters are larger, the image noise is smaller, but the loss of details is greater. The embodiments of the present application are implemented by Fig. 9 The method for generating training samples shown takes into account various situations that may be encountered in actual use scenarios when preparing training samples, and can make the low-quality images used in the training samples as close as possible to the actually collected images.

[0060] See also Fig. 9 , Fig. 9 A flow chart of a method for generating training samples provided in an embodiment of the present application is shown. As an example but not a limitation, the method is applied to an electronic device, and the method includes the following steps: Step 901, obtaining a facial image.

[0061] This application does not limit the source of the facial image. As an example and not a limitation, the facial image may be a facial image taken by an electronic device, or may refer to a facial image obtained from other devices, or may be a facial image stored in the electronic device itself.

[0062] Step 902: Determine a corresponding facial sample image based on the facial image.

[0063] The image quality of the facial sample image is lower than the image quality of the facial image. The facial sample image and the facial image correspond to the same face.

[0064] In a possible implementation, the determining the corresponding facial sample image based on the facial image includes: performing K degradation processes on the facial image to obtain K degraded images, wherein the K degradation processes have different degrees of degradation, and K is an integer greater than 1; and fusing different regions of the K degraded images to obtain the facial sample image.

[0065] It should be noted that the present application does not limit the manner of degrading the facial image. As an example and not a limitation, the facial image may be subjected to blurring, downsampling, noise addition, compression, noise reduction, upsampling, etc. in sequence to obtain a degraded image.

[0066] In a possible implementation, a method for determining any degraded image may include: adding noise to a facial image to obtain a noisy image; performing non-local mean denoising on the noisy image to obtain a first denoised image; performing total variation denoising on the first denoised image to obtain a second denoised image; and determining a corresponding degraded image based on the second denoised image.

[0067] After obtaining the second denoised image, the electronic device may determine the second denoised image as the corresponding degraded image, or may perform random blurring and random sharpening on the second denoised image to obtain the corresponding degraded image. In the above determination method, different degrees of degradation may be achieved by adding noise of different intensities, performing different degrees of noise reduction, etc., to obtain degraded images with different degrees of degradation.

[0068] The non-local means (NLM) denoising algorithm utilizes the non-local similarity in the image to remove noise by weighted averaging similar pixel blocks while retaining the image details as much as possible. The total variation (TV) denoising algorithm smoothes the image to remove noise while maintaining the edge features of the image. This embodiment can simulate the traditional image signal processing algorithms that may be applied in actual devices by using different types of denoising algorithms. The above-mentioned random blurring and random sharpening are also a kind of simulation. By simulating the actual image processing process, the degraded image can be made more consistent with the actual captured image.

[0069] In a possible implementation, the above-mentioned fusing different regions of K degraded images to obtain a facial sample image includes: obtaining K first mask images, wherein regions with pixel values ​​of 1 in the K first mask images are different, and sizes of the K first mask images are the same as sizes of the K degraded images; fusing the K first mask images with the K degraded images to obtain K fused images, wherein different first mask images correspond to different degraded images; and fusing the K fused images to obtain a facial sample image.

[0070] When K first mask images are fused with K degraded images, one first mask image is fused with one degraded image, and different first mask images are fused with different degraded images, so K fused images can be obtained after fusion. It should be understood that when K first mask images are fused with K degraded images, the first mask images and degraded images to be fused together can be randomly selected, and this application does not limit this. As an example but not a limitation, K is 3, the three first mask images are first mask image 1, first mask image 2, and first mask image 3, and the three degraded images are degraded image 1, degraded image 2, and degraded image 3, respectively. The first mask image 1 can be fused with the degraded image 1 to obtain fused image 1, the first mask image 2 can be fused with the degraded image 2 to obtain fused image 2, and the first mask image 3 can be fused with the degraded image 3 to obtain fused image 3.

[0071] Optionally, the area with a pixel value of 1 in the K first mask images corresponds to the entire area of ​​the degraded image (that is, if the area with a pixel value of 1 in the K first mask images is combined, the size of the area is the same as the size of the degraded image). On this basis, the fused image and the degraded image can correspond to the same face and have the same size. As an example but not a limitation, K is 2, the area with a pixel value of 1 in the first mask image 1 corresponds to area 1 in the degraded image, the area with a pixel value of 1 in the second mask image 2 corresponds to area 2 in the degraded image, and area 2 is the area in the degraded image other than area 1.

[0072] In a possible implementation, the above-mentioned obtaining of K first mask images includes: generating K second mask images, the pixel values ​​in the second mask images are all 0, and the sizes of the K second mask images are the same as the sizes of the K degraded images; generating L groups of target coordinate information, each group of target coordinate information corresponds to an area, the L groups of target coordinate information correspond to different areas, and L is an integer greater than or equal to K; allocating the L groups of target coordinate information to the K second mask images, and each second mask image is allocated at least one group of target coordinate information; for any second mask image, updating the pixel value of the area corresponding to the second mask image of the at least one group of allocated target coordinate information to 1, so as to obtain the corresponding first mask image.

[0073] Among them, the pixel values, sizes and coordinate information of the K second mask images are the same, and L groups of target coordinate information can be generated based on the coordinate information of the second mask image (for example, the coordinate information of the second mask image is divided into L groups of target coordinate information, which can be randomly divided or a division rule can be pre-specified), and the area corresponding to the L groups of target coordinate information corresponds to the entire area of ​​the second mask image (that is, if the areas corresponding to the L groups of target coordinate information are combined, the size of the obtained area is the same as the size of the second mask image).

[0074] When the electronic device allocates L sets of target coordinate information to K second mask images, it can be allocated randomly or according to a pre-set allocation rule, which is not limited in this application. Optionally, the allocation rule can be set according to the scene requirements.

[0075] The added noise includes but is not limited to the flat area noise and Gaussian noise actually collected.

[0076] In actual scenarios, the degree of degradation corresponding to different areas in an image may be different. For example, the quality of facial images taken at night is low, which is more serious. To simulate this situation, different degradation parameters can be used to generate multiple different degraded images, and then different areas of the multiple degraded images can be fused through randomly generated masks.

[0077] Optionally, the facial image may be degraded to different degrees by setting degradation parameters such as different blurring degrees, different downsampling multiples, adding noise of different intensities, different compression degrees, different noise reduction degrees and different upsampling multiples during blurring, etc.

[0078] In order to reduce the learning difficulty of the initial facial image restoration model, the facial image can be aligned to a standard template before the facial image is degraded and denoised. The standard template can refer to a pre-set facial key point template. Figure 10-1 The following is an example of a facial image: Figure 10-2 The following is an example of a standard template: Figure 10-3 The figure shows an example of a facial image after alignment. The positions of the facial key points in the aligned facial image are the same as those in the standard template. Figure 10-4 The figure shows an example of a facial sample image, which is obtained by fusing different regions of two degraded images. Figure 10-4 The area inside the red box is a local area in one degraded image, and the area outside the red box is a local area in another degraded image.

[0079] Optionally, the face image can be aligned to the standard template by affine transformation. If there are missing parts that cannot be interpolated in the affine transformation, they can be filled using a reflection filling algorithm.

[0080] Step 903: Perform denoising processing on the facial image to obtain a first denoised image and a second denoised image.

[0081] The denoising intensity of the first denoised image is greater than the denoising intensity of the second denoised image.

[0082] In order to improve the mid- and low-frequency performance of facial data without further increasing the high-frequency texture of the model (because too much high-frequency texture details on the facial image can easily cause anomalies in the final result), this embodiment can use two TV noise reductions with different noise reduction intensities to achieve the above effect. TV noise reduction mainly smoothes the image to remove noise while maintaining the edge features of the image. TV noise reduction can be achieved by minimizing the energy functional in formula (7).

[0083] (7) in, represents the energy functional, represents the denoised image, represents the L1 norm of the gradient of the denoised image, represents a face image, represents the regularization coefficient, Used to balance the trade-off between noise reduction and fidelity, represents pixels and 2 represents the L2 norm.

[0084] Step 904: Determine the mid- and low-frequency information of the first denoised image based on the first denoised image and the second denoised image.

[0085] In one embodiment, the low- and medium-frequency information in the first denoised image can be obtained by subtracting the pixel value of the corresponding pixel in the second denoised image from the pixel value in the first denoised image, wherein the first denoised image and the second denoised image have the same size.

[0086] Step 905: frequency enhancement is performed on the facial image based on the medium and low frequency information to obtain a corresponding restored reference image.

[0087] The image quality of the restored reference image is higher than that of the facial image, and the facial sample image and the restored reference image are training samples corresponding to the facial image.

[0088] After frequency enhancement of the facial image based on the mid- and low-frequency information, the restored reference image can be made more three-dimensional than the facial image. Figure 10-5 The following is an example of a restored reference image compared to Figure 10-3 The facial image shown is more three-dimensional.

[0089] In a possible implementation, after generating the training samples corresponding to the facial image, the facial image restoration model may be trained so that the trained facial image restoration model can better restore the facial image to be restored. As an example but not a limitation, the training samples generated in this embodiment may be used to train Figure 1 The initial facial image restoration model of the illustrated embodiment enables the target facial image restoration model to better restore the facial image to be restored.

[0090] The facial sample image generated by the embodiment of the present application conforms to the actual snapshot scene, and the generated restored reference image can improve the mid- and low-frequency performance of the facial data without further strengthening the high-frequency texture of the model.

[0091] See also Fig.11 , Fig.11 The flowchart of the facial image restoration method provided by the embodiment of the present application is shown. As an example but not a limitation, the method is applied to an electronic device in which a target facial image restoration model is deployed. The method comprises the following steps: Step 1101: Obtain a facial image to be restored and a facial key point image corresponding to the facial image to be restored.

[0092] The present application does not limit the source of the facial image to be restored. As an example but not a limitation, the facial image to be restored may be a facial image taken by an electronic device, or may refer to a facial image obtained from other devices, or may be a facial image stored in the electronic device itself.

[0093] The above-mentioned facial image to be restored and the facial key point image corresponding thereto correspond to the same face and have the same size. The facial key points in the facial key point image may be the key points of the face such as the eyes, nose, mouth, etc. In order to reduce the influence of other facial areas (i.e., areas other than the facial key points) on the restoration of the facial image, the pixel value of the pixels in the facial key point image in the area where the facial key points are located is 255, and the pixel value of the pixels in the facial key point sample image in the area where the facial key points are located is 0. This application does not limit the method for obtaining the facial key point image.

[0094] Step 1102: Determine a restored facial image corresponding to the facial image to be restored based on the facial image to be restored, the facial key point image corresponding to the facial image to be restored, and the target facial image restoration model.

[0095] The target facial image restoration model includes a first feature extraction network, a second feature extraction network connected to the first feature extraction network, and a generation network.

[0096] As an example, the target facial image restoration model can be used by a smart device such as a mobile phone or tablet computer or a server through Figure 1 The method shown is trained, or obtained by the smart device or server from device A, and device A is executed Figure 1 The method shown is used to train a device to obtain a target facial image restoration model.

[0097] Optionally, the above step 1102 includes: In one case, if the target facial image restoration model is deployed in the smart device or server, the facial image to be restored and the facial key point image are input into the target facial image restoration model to obtain the restored facial image. In another case, if the target facial image restoration model is not deployed in the smart device or server, the smart device sends the facial image to be restored and the facial key point image to the device deployed with the target facial image restoration model, and then the device deployed with the target facial image restoration model obtains the restored facial image based on the facial image to be restored and the facial key point image, and sends it to the smart device or server.

[0098] As an example but not a limitation, in practical applications, the target facial image restoration model can be used to perform facial image restoration on facial images collected in real time.

[0099] It should be noted that the schemes not detailed in this embodiment can refer to the description of the relevant schemes in the above embodiments, and will not be repeated here. It should be understood that the size of the sequence number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0100] Corresponding to the training method of the facial image restoration model described in the above embodiment, Fig.12 The structural schematic diagram of the training device of the face image restoration model provided by the embodiment of the present application is shown. The initial face image restoration model includes: a first feature extraction network, a second feature extraction network connected to the first feature extraction network, and a generation network. For ease of explanation, only the part related to the embodiment of the present application is shown.

[0101] Reference Fig.12 , the device comprises: The sample acquisition module 1201 is used to acquire training samples and facial key point sample images corresponding to the training samples, wherein the training samples include facial sample images and restored reference images corresponding to the facial sample images, and the image quality of the restored reference images is higher than that of the facial sample images; The image processing module 1202 is used to input the facial sample image and the facial key point sample image into the initial facial image restoration model to trigger the generation network to predict a restored sample image corresponding to the facial sample image based on the facial identity features of the facial sample image constrained by the first feature extraction network and the facial key point features of the facial key point sample image constrained by the second feature extraction network; The model training module 1203 is used to train the initial facial image restoration model based on the restored sample image and the restored reference image to obtain a target facial image restoration model, and the target facial image restoration model is used to perform facial image restoration on the facial image to be restored.

[0102] Optionally, the convolution layer of the generator network corresponds to a potential code, and the potential code is used to adjust the weight of the convolution layer of the generator network. The above-mentioned model training module 1203 is specifically used for: Based on the restored sample image and the restored reference image, the first feature extraction network, the second feature extraction network and the latent code are trained to obtain a trained first feature extraction network, a trained target feature extraction network and a trained latent code.

[0103] Optionally, the first feature extraction network includes a downsampling residual block, a first fusion module and an upsampling residual block, and the image processing module 1202 is specifically used for: Inputting the facial sample image into the down-sampling residual block to obtain a down-sampling feature map, wherein the down-sampling feature map includes facial identity features of the facial sample image; Inputting the facial key point sample image into the second feature extraction network to obtain a facial key point feature map, and performing attention enhancement of facial key points on the downsampled feature map based on the facial key point feature map to obtain an image enhancement feature map, wherein the facial key point feature map includes facial key point features of the facial key point sample image; Inputting the image enhancement feature map into the first fusion module to trigger the first fusion module to fuse the image enhancement feature map and the down-sampling feature map to obtain a fused feature map; Inputting the downsampled feature map and the fused feature map into the upsampled residual block to obtain an upsampled feature map; The down-sampled feature map, the up-sampled feature map and the potential code are input into the generation network to obtain the restored sample image.

[0104] Optionally, the first feature extraction network includes a downsampling residual block, a second fusion module and an upsampling residual block, and the image processing module 1202 is specifically used for: Inputting the facial sample image into the down-sampling residual block to obtain a down-sampling feature map, wherein the down-sampling feature map includes facial identity features of the facial sample image; Inputting the facial key point sample image into the second feature extraction network to obtain a facial key point feature map, wherein the facial key point feature map includes facial key point features of the facial key point sample image; Inputting the facial key point feature map into the second fusion module to trigger the second fusion module to perform attention enhancement of facial key points on the downsampled feature map based on the facial key point feature map to obtain an image enhancement feature map, and fusing the image enhancement feature map with the downsampled feature map to obtain a fused feature map; Inputting the downsampled feature map and the fused feature map into the upsampled residual block to obtain an upsampled feature map; The down-sampled feature map, the up-sampled feature map and the potential code are input into the generation network to obtain the restored sample image.

[0105] Optionally, the above model training module 1203 is specifically used for: Calculating a first loss value, where the first loss value is a loss value between the restored sample image and the restored reference image; Acquire a first perceptual feature map and a second perceptual feature map, wherein the first perceptual feature map includes perceptual features of the restored sample image, and the second perceptual feature map includes perceptual features of the restored reference image; Calculate a second loss value, where the second loss value is a loss value between the first perceptual feature map and the second perceptual feature map; Acquire a first facial identity feature map and a second facial identity feature map, wherein the first facial identity feature map includes facial identity features of the restored sample image, and the second facial identity feature map includes facial identity features of the restored reference image; Calculating a third loss value, where the third loss value is a loss value between the first facial identity feature map and the second facial identity feature map; The initial facial image restoration model is trained based on the first loss value, the second loss value and the third loss value.

[0106] Optionally, the sample acquisition module 1201 includes: An image acquisition unit, used for acquiring a facial image; a sample determination unit, configured to determine a corresponding facial sample image based on the facial image, wherein the image quality of the facial sample image is lower than the image quality of the facial image; a denoising unit, configured to perform denoising on the facial image to obtain a first denoised image and a second denoised image, wherein the denoising intensity of the first denoised image is greater than the denoising intensity of the second denoised image; a medium and low frequency determination unit, configured to determine medium and low frequency information of the first denoised image based on the first denoised image and the second denoised image; A frequency enhancement unit is used to perform frequency enhancement on the facial image based on the medium and low frequency information to obtain a corresponding restored reference image, wherein the image quality of the restored reference image is higher than the image quality of the facial image.

[0107] Optionally, the sample determination unit includes: A degradation subunit, configured to perform K degradation processes on the facial image to obtain K degraded images, wherein the K degradation processes have different degradation degrees, and K is an integer greater than 1; The fusion subunit is used to fuse different regions of the K degraded images to obtain the face sample image.

[0108] Optionally, the above-mentioned degradation subunit is specifically used for: adding noise to the facial image to obtain a noise image; Performing non-local mean denoising on the noisy image to obtain a first denoised image; Performing total variation denoising on the first denoised image to obtain a second denoised image; Based on the second denoised image, the corresponding degraded image is determined.

[0109] Optionally, the above fusion subunit is specifically used for: Acquire K first mask images, wherein regions with pixel values ​​of 1 in the K first mask images are different, and sizes of the K first mask images are the same as sizes of the K degraded images; fusing the K first mask images with the K degraded images to obtain K fused images, where different first mask images correspond to different degraded images; The K fused images are fused to obtain the face sample image.

[0110] Optionally, the above fusion subunit is specifically used for: Generate K second mask images, wherein pixel values ​​in the second mask images are all 0, and sizes of the K second mask images are the same as sizes of the K degraded images; Generate L groups of target coordinate information, each group of target coordinate information corresponds to an area, the L groups of target coordinate information correspond to different areas, and L is an integer greater than or equal to K; Allocating the L groups of target coordinate information to the K second mask images, each of the second mask images being allocated at least one group of target coordinate information; For any of the second mask images, the pixel value of the allocated at least one set of target coordinate information in the area corresponding to the second mask image is updated to 1 to obtain the corresponding first mask image.

[0111] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0112] Fig.13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.13 As shown, the electronic device 13 of this embodiment includes: at least one processor 1300 ( Fig.13 Only one is shown in the figure), a memory 1301 and a computer program 1302 stored in the memory 1301 and executable on the at least one processor 1300, wherein the processor 1300 implements the steps of any of the above-mentioned method embodiments when executing the computer program 1302.

[0113] The electronic device may include, but is not limited to, a processor 1300 and a memory 1301. Those skilled in the art will appreciate that Fig.13 It is only an example of the electronic device 13 and does not constitute a limitation on the electronic device 13. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0114] The processor 1300 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0115] In some embodiments, the memory 1301 may be an internal storage unit of the electronic device 13, such as a hard disk or memory of the electronic device 13. In other embodiments, the memory 1301 may also be an external storage device of the electronic device 13, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 13. Further, the memory 1301 may also include both an internal storage unit of the electronic device 13 and an external storage device. The memory 1301 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 1301 may also be used to temporarily store data that has been output or is to be output.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0117] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0118] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0119] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0121] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A training method for a facial image restoration model, characterized in that: The initial facial image restoration model includes: a first feature extraction network, a second feature extraction network and a generation network respectively connected to the first feature extraction network, and the training method includes: Acquire a training sample and a facial key point sample image corresponding to the training sample, wherein the training sample includes a facial sample image and a restored reference image corresponding to the facial sample image, and the image quality of the restored reference image is higher than that of the facial sample image; Inputting the facial sample image and the facial key point sample image into the initial facial image restoration model to trigger the generation network to predict a restored sample image corresponding to the facial sample image based on the facial identity features of the facial sample image constrained by the first feature extraction network and the facial key point features of the facial key point sample image constrained by the second feature extraction network; Based on the restored sample image and the restored reference image, the initial facial image restoration model is trained to obtain a target facial image restoration model, and the target facial image restoration model is used to perform facial image restoration on the facial image to be restored.

2. The training method according to claim 1, characterized in that: The convolution layer of the generating network corresponds to a potential code, and the potential code is used to adjust the weight of the convolution layer of the generating network. The initial facial image restoration model is trained based on the restored sample image and the restored reference image to obtain a target facial image restoration model, including: Based on the restored sample image and the restored reference image, the first feature extraction network, the second feature extraction network and the latent code are trained to obtain a trained first feature extraction network, a trained target feature extraction network and a trained latent code.

3. The training method according to claim 2, characterized in that: The first feature extraction network includes a downsampling residual block, a first fusion module and an upsampling residual block, and the step of inputting the facial sample image into the initial facial image restoration model to trigger the generation network to predict a restored sample image corresponding to the facial sample image based on the facial identity features of the facial sample image constrained by the first feature extraction network and the facial key point features of the facial key point sample image constrained by the second feature extraction network, comprises: Inputting the facial sample image into the down-sampling residual block to obtain a down-sampling feature map, wherein the down-sampling feature map includes facial identity features of the facial sample image; Inputting the facial key point sample image into the second feature extraction network to obtain a facial key point feature map, and performing attention enhancement of facial key points on the downsampled feature map based on the facial key point feature map to obtain an image enhancement feature map, wherein the facial key point feature map includes facial key point features of the facial key point sample image; Inputting the image enhancement feature map into the first fusion module to trigger the first fusion module to fuse the image enhancement feature map and the down-sampling feature map to obtain a fused feature map; Inputting the downsampled feature map and the fused feature map into the upsampled residual block to obtain an upsampled feature map; The down-sampled feature map, the up-sampled feature map and the potential code are input into the generation network to obtain the restored sample image.

4. The training method according to claim 2, characterized in that: The first feature extraction network includes a downsampling residual block, a second fusion module and an upsampling residual block, and the step of inputting the facial sample image into the initial facial image restoration model to trigger the generation network to predict a restored sample image corresponding to the facial sample image based on the facial identity features of the facial sample image constrained by the first feature extraction network and the facial key point features of the facial key point sample image constrained by the second feature extraction network, comprises: Inputting the facial sample image into the down-sampling residual block to obtain a down-sampling feature map, wherein the down-sampling feature map includes facial identity features of the facial sample image; Inputting the facial key point sample image into the second feature extraction network to obtain a facial key point feature map, wherein the facial key point feature map includes facial key point features of the facial key point sample image; Inputting the facial key point feature map into the second fusion module to trigger the second fusion module to perform attention enhancement of facial key points on the downsampled feature map based on the facial key point feature map to obtain an image enhancement feature map, and fusing the image enhancement feature map with the downsampled feature map to obtain a fused feature map; Inputting the downsampled feature map and the fused feature map into the upsampled residual block to obtain an upsampled feature map; The down-sampled feature map, the up-sampled feature map and the potential code are input into the generation network to obtain the restored sample image.

5. The training method according to any one of claims 1 to 4, characterized in that: The training of the initial facial image restoration model based on the restored sample image and the restored reference image comprises: Calculating a first loss value, where the first loss value is a loss value between the restored sample image and the restored reference image; Acquire a first perceptual feature map and a second perceptual feature map, wherein the first perceptual feature map includes perceptual features of the restored sample image, and the second perceptual feature map includes perceptual features of the restored reference image; Calculate a second loss value, where the second loss value is a loss value between the first perceptual feature map and the second perceptual feature map; Acquire a first facial identity feature map and a second facial identity feature map, wherein the first facial identity feature map includes facial identity features of the restored sample image, and the second facial identity feature map includes facial identity features of the restored reference image; Calculating a third loss value, where the third loss value is a loss value between the first facial identity feature map and the second facial identity feature map; The initial facial image restoration model is trained based on the first loss value, the second loss value and the third loss value.

6. The training method according to any one of claims 1 to 4, characterized in that: The obtaining of training samples comprises: Get a facial image; Based on the facial image, determining a corresponding facial sample image, wherein the image quality of the facial sample image is lower than the image quality of the facial image; Performing denoising processing on the facial image to obtain a first denoised image and a second denoised image, wherein the denoising intensity of the first denoised image is greater than the denoising intensity of the second denoised image; Determining mid- and low-frequency information of the first denoised image based on the first denoised image and the second denoised image; The frequency of the facial image is enhanced based on the medium and low frequency information to obtain a corresponding restored reference image, wherein the image quality of the restored reference image is higher than that of the facial image.

7. The training method according to claim 6, characterized in that: The determining, based on the facial image, a corresponding facial sample image comprises: Performing K times of degradation processing on the facial image to obtain K degraded images, wherein the K times of degradation processing have different degrees of degradation, and K is an integer greater than 1; Different regions of the K degraded images are fused to obtain the face sample image.

8. The training method according to claim 7, characterized in that: Any method for determining the degraded image includes: adding noise to the facial image to obtain a noise image; Performing non-local mean denoising on the noisy image to obtain a first denoised image; Performing total variation denoising on the first denoised image to obtain a second denoised image; Based on the second denoised image, the corresponding degraded image is determined.

9. The training method according to claim 7, characterized in that: The different regions of the K degraded images are fused to obtain the face sample image, including: Acquire K first mask images, wherein regions with pixel values ​​of 1 in the K first mask images are different, and sizes of the K first mask images are the same as sizes of the K degraded images; fusing the K first mask images with the K degraded images to obtain K fused images, where different first mask images correspond to different degraded images; The K fused images are fused to obtain the face sample image.

10. The training method according to claim 9, characterized in that: The step of acquiring K first mask images comprises: Generate K second mask images, wherein pixel values ​​in the second mask images are all 0, and sizes of the K second mask images are the same as sizes of the K degraded images; Generate L groups of target coordinate information, each group of target coordinate information corresponds to an area, the L groups of target coordinate information correspond to different areas, and L is an integer greater than or equal to K; Allocating the L groups of target coordinate information to the K second mask images, each of the second mask images being allocated at least one group of target coordinate information; For any of the second mask images, the pixel value of the allocated at least one set of target coordinate information in the area corresponding to the second mask image is updated to 1 to obtain the corresponding first mask image.

Images