Face image super-resolution method, terminal device and storage medium

Abstract

The invention discloses a face image super-resolution method, terminal equipment and a storage medium, and the method comprises the steps: carrying out the super-resolution processing of a to-be-processed face image based on a super-resolution network, and obtaining a super-resolution face image corresponding to the to-be-processed face image; wherein the super-resolution network is obtained by training a feature dictionary in a pre-trained feature dictionary model.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a super-resolution method for face images, a terminal device, and a storage medium. Background Art

[0002] Super-resolution (SR), also known as super-resolution, is a subset of super-resolution. Face super-resolution technology can break through the limitations of hardware imaging resolution and further improve the clarity of the image. It is widely used in current mobile phone camera functions.

[0003] Common face super-resolution technologies mainly rely on some face prior information, such as geometric information, reference frame information, or generated face prior. However, common face image super-resolution methods all have the defect of poor super-resolution effect, which reduces the super-resolution performance of terminal devices. Summary of the invention

[0004] The embodiments of the present application provide a super-resolution method for facial images, a terminal device, and a storage medium, which can improve the super-resolution effect of facial images and thereby enhance the super-resolution performance of the terminal device.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a super-resolution method for a face image, the method comprising:

[0007] The super-resolution processing is performed on the face image to be processed based on the super-resolution network to obtain the super-resolution face image corresponding to the face image to be processed; wherein the super-resolution network is obtained by feature dictionary training in a pre-trained feature dictionary model.

[0008] In a second aspect, an embodiment of the present application provides a terminal device, the terminal device comprising: a processing unit,

[0009] The processing unit is used to perform super-resolution processing on the face image to be processed based on a super-resolution network to obtain a super-resolution face image corresponding to the face image to be processed; wherein the super-resolution network is obtained by feature dictionary training in a pre-trained feature dictionary model.

[0010] In a third aspect, an embodiment of the present application provides a terminal device, wherein the terminal device includes a processor and a memory; wherein:

[0011] The memory is used to store a computer program that can be run on the processor;

[0012] The processor is used to execute the method described in the first aspect above when running the computer program.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the method described in the first aspect above is implemented.

[0014] The embodiment of the present application provides a super-resolution method for facial images, a terminal device, and a storage medium. The terminal device performs super-resolution processing on the facial image to be processed based on a super-resolution network to obtain a super-resolution facial image corresponding to the facial image to be processed; wherein the super-resolution network is obtained by training a feature dictionary in a pre-trained feature dictionary model. That is to say, in the embodiment of the present application, the super-resolution network for super-resolution processing of the facial image to be processed is trained using a pre-trained feature dictionary. Since effective feature information in high-definition facial images can be mined during the training of the feature dictionary, the feature dictionary has rich facial details, and more facial details can be implicitly applied to image reconstruction during the training of the super-resolution network guided by the feature dictionary. It can be seen that in the embodiment of the present application, the super-resolution network trained with a learned feature dictionary has higher robustness, and then the super-resolution network is used for super-resolution processing of facial images, which can obtain more accurate super-resolution effects, improve the super-resolution effects of facial images, and effectively improve the super-resolution performance of the terminal device. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of face super-resolution based on face geometric prior information;

[0016] Figure 2 Schematic diagram of face super-resolution based on reference prior;

[0017] Figure 3 Schematic diagram of face super-segmentation based on generated face prior;

[0018] Figure 4 A schematic diagram of the implementation flow of the super-resolution method for face images proposed in the embodiment of the present application;

[0019] Figure 5 Schematic diagram of the composition structure of the super-resolution network;

[0020] Figure 6 A schematic diagram of super-resolution processing based on a super-resolution network;

[0021] Figure 7 Schematic diagram of the composition structure of the feature dictionary model;

[0022] Figure 8Schematic diagram of the composition structure of the initial network;

[0023] Fig. 9 Schematic diagram of the training process of the feature dictionary;

[0024] Fig.10 Schematic diagram of the process of super-resolution network training guided by feature dictionary;

[0025] Fig.11 Schematic diagram of the structure of the terminal device proposed in the embodiment of the present application Figure 1 ;

[0026] Fig.12 Schematic diagram of the structure of the terminal device proposed in the embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It is understood that the specific embodiments described herein are only used to explain the related applications, rather than to limit the applications. It should also be noted that, for ease of description, only the parts related to the related applications are shown in the drawings.

[0028] Super-resolution (SR), abbreviated as super-resolution, refers to the process of using optics and related optical knowledge to restore image details and other data information based on known image information. Simply put, it increases the resolution of the image to prevent its image quality from deteriorating.

[0029] Face super-resolution is a subset of the super-resolution field. Portrait photography is a key scene for taking photos, and the clarity and naturalness of the face will directly affect the success rate and user experience. Face super-resolution technology can break through the limitations of hardware imaging resolution and further improve the clarity of the finished image. It is widely used in current mobile phone camera functions.

[0030] Common face super-resolution techniques mainly rely on some face prior information, such as geometric information, reference frame information, or generated face prior.

[0031] For example, Figure 1 is a schematic diagram of face super-resolution based on face geometric prior information, such as Figure 1 As shown in the figure, when using the geometric prior information of the face to perform super-resolution processing on the face image, upsampling processing can be performed first to extract the features of the facial features, and then the geometric prior information can be introduced into the super-resolution process. Then, the feature information can be constrained to finally obtain the reconstructed image.

[0032] For example, Figure 2 is a schematic diagram of face super-resolution based on reference priors, such as Figure 2As shown in the figure, when using reference prior information for super-resolution processing of face images, you can first use the database for matching to obtain reference prior information, and then you can use the reference prior information, combine it with the facial features library, and integrate it into the super-resolution network to finally obtain the reconstructed image.

[0033] For example, Figure 3 Schematic diagram of face super-resolution based on generating face priors, such as Figure 3 As shown, when using the generated face prior information to perform super-resolution processing on the face image, it is necessary to use a generative network to generate the prior information, and then use the generated prior information to perform super-resolution processing to finally obtain a reconstructed image.

[0034] For the common face super-resolution methods mentioned above, the main principle of the face super-resolution method based on face geometric prior information is to use the face structure information (the positions of the facial features are relatively fixed) to guide the super-resolution network, but the structure information comes from low-definition images and its accuracy is not high, resulting in poor super-resolution results. The main principle of the face super-resolution method based on reference prior is to use the fusion of a single high-definition face information to migrate high-definition information. Its disadvantage is that the amount of prior information of a single frame of face is small and not representative, which will lead to abnormal results. The main principle of the face super-resolution method based on generated face prior is to use the face generation network trained with big data to generate details for low-definition images. Its disadvantage is that it is easy to over-generate, resulting in serious image distortion.

[0035] In other words, common super-resolution methods for facial images all have the defect of poor super-resolution effect, which reduces the super-resolution performance of terminal devices.

[0036] In order to solve the above problems, in an embodiment of the present application, the terminal device performs super-resolution processing on the face image to be processed based on the super-resolution network to obtain the face image after super-resolution corresponding to the face image to be processed; wherein the super-resolution network is obtained by training the feature dictionary in the pre-trained feature dictionary model. That is to say, in an embodiment of the present application, the super-resolution network for super-resolution processing of the face image to be processed is trained using a pre-trained feature dictionary. Since the effective feature information in the high-definition face image can be mined during the training of the feature dictionary, the feature dictionary has rich face details, and more face details can be implicitly applied to the reconstruction of the image by guiding the training of the super-resolution network through the feature dictionary. It can be seen that in an embodiment of the present application, the super-resolution network trained with the learned feature dictionary has higher robustness, and then the super-resolution network is used for super-resolution processing of face images, which can obtain more accurate super-resolution effect, improve the super-resolution effect of face images, and effectively improve the super-resolution performance of the terminal device.

[0037] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0038] An embodiment of the present application provides a super-resolution method for a face image, which can be applied to a terminal device. Figure 4 Schematic diagram of the implementation process of the super-resolution method for face images proposed in the embodiment of the present application, such as Figure 4 As shown, the terminal device may perform a super-resolution method for a face image, including the following steps:

[0039] Step 101, performing super-resolution processing on the face image to be processed based on a super-resolution network to obtain a super-resolution face image corresponding to the face image to be processed; wherein the super-resolution network is obtained by feature dictionary training in a pre-trained feature dictionary model.

[0040] In an embodiment of the present application, the terminal device may choose to perform super-resolution processing on the face image to be processed based on a super-resolution network, thereby obtaining a super-resolution face image corresponding to the face image to be processed.

[0041] It should be noted that, in the embodiments of the present application, the super-resolution network may be obtained by training a feature dictionary in a pre-trained feature dictionary model.

[0042] That is, in an embodiment of the present application, a feature dictionary model including a feature dictionary may be trained in advance, and then the feature dictionary in the trained feature model may be used to guide the training of the super-resolution network.

[0043] It can be understood that in the embodiments of the present application, the facial image to be processed before super-resolution processing can be a low-quality facial image or a low-definition facial image, and the super-resolution facial image obtained by super-resolution processing can be a high-quality facial image or a high-definition facial image.

[0044] It should be noted that in the embodiments of the present application, the facial image may be an image including the contours of facial features or the features of facial features, that is, whether it is a low-definition facial image or a high-definition facial image, it includes relevant information about the facial features.

[0045] It can be understood that in the embodiments of the present application, compared with low-definition face images (or low-definition face images), the resolution, clarity, detail information, feature information and other image quality-related parameters of the high-definition face images (or high-quality face images) are higher than those of the low-definition face images (or low-definition face images).

[0046] Furthermore, in an embodiment of the present application, the super-resolution network may include an encoder and a decoder, wherein the encoder may be used to extract feature tensors from an input image, and the decoder may reconstruct the image based on the input feature tensors.

[0047] Exemplarily, in some embodiments, Figure 5 It is a schematic diagram of the composition structure of the super-resolution network. Figure 5 As shown, the super-resolution network may include an encoder and a decoder. After the face image to be processed is input into the encoder, the encoder may map the image to a feature space to obtain a corresponding feature tensor; after the feature tensor is input into the decoder, the decoder may complete the image reconstruction based on the feature tensor to obtain a reconstructed image after super-resolution processing, that is, a super-resolution face image.

[0048] It should be noted that in an embodiment of the present application, when super-resolution processing is performed on a facial image to be processed based on a super-resolution network to obtain a super-resolution facial image corresponding to the facial image to be processed, the facial image to be processed can be first input into an encoder to output a first feature tensor corresponding to the facial image to be processed; then the first feature tensor can be input into a decoder to output a super-resolution facial image.

[0049] Exemplarily, in some embodiments, the resolution of the face image to be processed is lower than that of the corresponding super-resolution face image.

[0050] Exemplarily, in some embodiments, Figure 6 It is a schematic diagram of super-resolution processing based on super-resolution network, such as Figure 6 As shown, after the face image to be processed is input into the encoder, the encoder can map the face image to be processed to the feature space to obtain the corresponding first feature tensor; then, the first feature tensor is input into the decoder, and the decoder can complete the image reconstruction based on the first feature tensor to obtain the corresponding super-resolution face image.

[0051] Furthermore, in an embodiment of the present application, a feature dictionary model may be pre-trained based on a face data set. The present application does not specifically limit the pre-training method, for example, the pre-training method may be VQAVE.

[0052] Exemplarily, in some embodiments, the face data set may include HD SLR face data, and the terminal device may use VQAVE to perform pre-training on the HD SLR face data to extract effective high-definition face details, such as skin texture, hair, etc., from the HD SLR face data to obtain a feature dictionary.

[0053] Accordingly, in an embodiment of the present application, after the training of the feature dictionary model is completed, the feature dictionary in the trained feature dictionary model can be used to guide the learning of the super-resolution network.

[0054] It should be noted that, in the embodiments of the present application, an initial network can be trained based on a face image dataset and a feature dictionary model, so as to obtain a super-resolution network.

[0055] It should be noted that, in an embodiment of the present application, a facial image data set may include high-definition facial images and corresponding low-definition facial images, wherein the resolution of the high-definition facial images is greater than that of the low-definition facial images.

[0056] It is understood that in the embodiment of the present application, the face image data set used for model training may include high-definition face images and low-definition face images in one-to-one correspondence. The high-definition face images can be used to train the feature dictionary model including the feature dictionary, and the high-definition face images combined with the corresponding low-definition face images can be used to train the initial network, and finally obtain the super-resolution network.

[0057] Furthermore, in an embodiment of the present application, the feature dictionary model may include an encoder, a decoder, and a feature dictionary. The encoder may be used to extract feature tensors from an input image, the feature dictionary may be used to perform vector matching, and the decoder may reconstruct an image based on the input feature tensor.

[0058] It should be noted that, in the embodiments of the present application, during the training of the feature dictionary model, the encoder and the decoder may be respectively composed of convolutional networks.

[0059] It should be noted that, in the embodiment of the present application, during the training process of the feature dictionary model, the feature dictionary may be randomly initialized by Gaussian distribution.

[0060] Exemplarily, in some embodiments, Figure 7 It is a schematic diagram of the composition structure of the feature dictionary model, such as Figure 7 As shown, the feature dictionary model may include an encoder, a feature dictionary, and a decoder. After the high-definition face image is input to the encoder, the encoder may map the image to the feature space to obtain the corresponding feature tensor; before the feature tensor is input to the decoder, it may be matched through the feature dictionary to obtain a new feature tensor with richer feature information; after the new feature tensor is input to the decoder, the decoder may complete the image reconstruction based on the new feature tensor to obtain a reconstructed image after super-resolution processing.

[0061] Furthermore, in an embodiment of the present application, when training a feature dictionary model based on a high-definition face image, the high-definition face image can be first input into an encoder to output a second feature tensor corresponding to the high-definition face image; then, the second feature tensor is matched based on the feature dictionary to determine a third feature tensor; then, the third feature tensor is input into a decoder to output a first reconstructed image corresponding to the high-definition face image; finally, the first loss can be determined based on the second feature tensor, the third feature tensor, the high-definition face image and the first reconstructed image, and the encoder, decoder and feature dictionary can be corrected according to the first loss.

[0062] It can be understood that in an embodiment of the present application, when training a feature dictionary model including an encoder, a decoder and a feature dictionary, for an input high-definition face image, the encoder can perform feature extraction on the high-definition face image to obtain a second feature tensor. Then, the feature dictionary can perform vector matching on the second feature tensor to obtain a third feature tensor with richer feature information. Then, the decoder can reconstruct the image based on the input third feature tensor and output a first reconstructed image.

[0063] Furthermore, in an embodiment of the present application, when matching the second feature tensor based on the feature dictionary to determine the third feature tensor, the vector of the second feature tensor can be firstly matched based on the dictionary vector in the feature dictionary to determine the matching vector corresponding to the vector of the second feature tensor; and then the second feature tensor is updated according to the matching vector to determine the third feature tensor.

[0064] It should be noted that, in an embodiment of the present application, when the second feature tensor is subjected to vector matching processing through a feature dictionary, the best matching dictionary vector is found for each vector of the second feature tensor in the dictionary vector of the feature dictionary, that is, the corresponding matching vector, and then the matching vector is used to replace the vector of the second feature tensor, and finally a new feature tensor, that is, the third feature tensor, is formed.

[0065] It can be understood that in an embodiment of the present application, when performing vector matching on the vectors of the second feature tensor based on the dictionary vectors in the feature dictionary, Euclidean distance can be used for matching, that is, the dictionary vector with the shortest Euclidean distance with the vectors of the second feature tensor is the corresponding matching vector.

[0066] Furthermore, in an embodiment of the present application, when training a feature dictionary model, after obtaining the second feature tensor, the third feature tensor and the first reconstructed image corresponding to the high-definition face image using the encoder, the feature dictionary and the decoder, the first loss can be determined based on the second feature tensor, the third feature tensor, the high-definition face image and the first reconstructed image, and the encoder, the decoder and the feature dictionary can be corrected according to the first loss.

[0067] It should be noted that in an embodiment of the present application, when training a feature dictionary model including an encoder, a decoder and a feature dictionary, a loss function can be calculated based on the second feature tensor, the third feature tensor and the first reconstructed image corresponding to the obtained high-definition face image, wherein any one or more loss functions can be used to determine the first loss.

[0068] It can be understood that in the embodiments of the present application, the calculated first loss can be used to constrain the consistency between the input high-definition face image and the output first reconstructed image, and to correct the structural parameters and weight parameters of the encoder, feature dictionary and decoder, thereby completing the training of the entire feature dictionary model.

[0069] Exemplarily, in some embodiments, the second eigentensor Z h and the third eigentensor Z p Determine the loss function L′ d and L′ c , as follows:

[0070]

[0071]

[0072] Exemplarily, in some embodiments, the high-definition face image I h and the first reconstructed image Determine the loss function L′ l1 , L′ per , L′ adv , as follows:

[0073]

[0074]

[0075]

[0076] Exemplarily, in some embodiments, the loss function L′ can be obtained based on the calculation d , L′ c , L′ l1 , L′ per , L′ adv , determine the first loss, as follows:

[0077] L Dict =L′ l1 +λ per L′ per +λ adv L′ adv +λ d L′ d+λ c L′ c (6)

[0078] Among them, λ per ,λ adv ,λ d ,λ c Can represent the weights between different loss functions.

[0079] Furthermore, in an embodiment of the present application, the initial network may include two parts: a feature guidance subnetwork and a super-resolution subnetwork, wherein the feature guidance subnetwork includes an encoder and a feature dictionary, and the super-resolution subnetwork may include an encoder and a decoder.

[0080] It should be noted that in the embodiments of the present application, the encoder and feature dictionary in the feature guidance subnetwork, as well as the encoder and decoder in the super-resolution subnetwork, are all obtained after training the feature dictionary model. Among them, the encoder can be used to extract the feature tensor of the input image, the feature dictionary can be used for vector matching, and the decoder can reconstruct the image based on the input feature tensor.

[0081] It should be noted that in an embodiment of the present application, during the training of the initial network, the encoder and decoder can be respectively composed of convolutional networks. Although the encoder, decoder and feature dictionary are obtained through the training of the feature dictionary model, when training the initial network, the encoder and decoder still need to be trained and updated, but the feature dictionary that has been trained will not be updated.

[0082] Exemplarily, in some embodiments, Figure 8 is a schematic diagram of the composition structure of the initial network, such as Figure 8As shown, the initial network may include a feature guidance subnetwork and a super-resolution subnetwork, wherein the feature guidance subnetwork may be composed of an encoder and a feature dictionary, and the super-resolution subnetwork may be composed of an encoder and a decoder. After the high-definition face image is input into the encoder in the feature guidance subnetwork, the encoder may map the high-definition face image to the feature space to obtain the corresponding feature tensor; then the feature dictionary may be used for matching to obtain a new feature tensor with richer feature information; at the same time, the low-definition face image corresponding to the high-definition face image may be input into the encoder in the super-resolution subnetwork, and the encoder may map the low-definition face image to the feature space to obtain the corresponding feature tensor. In the feature space, the new feature tensor corresponding to the high-definition face image and the feature tensor corresponding to the low-definition face image may be constrained to keep the two as consistent as possible, and obtain the new feature tensor corresponding to the low-definition face image. In this way, the facial detail features stored in the feature dictionary may be implicitly sent to the super-resolution subnetwork. Finally, the new feature tensor corresponding to the low-definition face image can be input into the decoder in the super-resolution sub-network. The decoder can complete the image reconstruction based on the new feature tensor to obtain the reconstructed image after super-resolution processing.

[0083] Furthermore, in an embodiment of the present application, when the initial network is trained according to the face image data set and the feature dictionary model to obtain a super-resolution network, the high-definition face image can be first input into the feature guidance sub-network to output the fourth feature tensor corresponding to the high-definition face image; at the same time, the low-definition face image is input into the encoder in the super-resolution sub-network to output the fifth feature tensor; then, feature constraints are performed based on the fourth feature tensor and the fifth feature tensor to determine the constrained feature tensor, and the constrained feature tensor is input into the decoder in the super-resolution sub-network to determine the second reconstructed image; finally, the second loss can be determined based on the fourth feature tensor, the constrained feature tensor, the high-definition face image and the second reconstructed image, and the encoder and decoder are corrected according to the second loss function to obtain a super-resolution network.

[0084] Furthermore, in an embodiment of the present application, when a high-definition facial image is input into a feature guidance subnetwork and a fourth feature tensor corresponding to the high-definition facial image is output, the high-definition facial image can be input into an encoder in the feature guidance subnetwork to output a sixth feature tensor corresponding to the high-definition facial image; then, the sixth feature tensor can be matched based on the feature dictionary to determine the fourth feature tensor.

[0085] It can be understood that in an embodiment of the present application, when training an initial network including a feature guidance subnetwork and a super-resolution subnetwork, for a high-definition face image input into the feature guidance subnetwork, the encoder in the feature guidance subnetwork can perform feature extraction on the high-definition face image to obtain a sixth feature tensor, and then the feature dictionary can perform vector matching on the sixth feature tensor to obtain a fourth feature tensor with richer feature information.

[0086] It can be understood that in an embodiment of the present application, when training the initial network including a feature guidance subnetwork and a super-resolution subnetwork, for the low-definition face image input to the super-resolution subnetwork, the encoder in the super-resolution subnetwork can perform feature extraction on the low-definition face image to obtain the fifth feature tensor.

[0087] Further, in an embodiment of the present application, after respectively obtaining the fourth feature tensor and the fifth feature tensor, feature constraints can be performed on the fourth feature tensor and the fifth feature tensor in the feature space to obtain a constrained feature tensor corresponding to the low-definition face image. Finally, the decoder in the super-resolution subnetwork can reconstruct the image based on the input constrained feature tensor and output a second reconstructed image.

[0088] Furthermore, in an embodiment of the present application, when matching the sixth feature tensor based on the feature dictionary to determine the fourth feature tensor, the vector of the sixth feature tensor can be firstly vector matched based on the dictionary vector in the feature dictionary to determine the matching vector corresponding to the vector of the sixth feature tensor; and then the sixth feature tensor is updated according to the matching vector to determine the fourth feature tensor.

[0089] It should be noted that, in an embodiment of the present application, when the sixth feature tensor is subjected to vector matching processing through a feature dictionary, the best matching dictionary vector is found for each vector of the sixth feature tensor in the dictionary vector of the feature dictionary, that is, the corresponding matching vector, and then the matching vector is used to replace the vector of the sixth feature tensor, and finally a new feature tensor, that is, the fourth feature tensor, is formed.

[0090] It can be understood that in an embodiment of the present application, when performing vector matching on the vectors of the sixth feature tensor based on the dictionary vectors in the feature dictionary, Euclidean distance can be used for matching, that is, the dictionary vector with the shortest Euclidean distance with the vectors of the sixth feature tensor is the corresponding matching vector.

[0091] Furthermore, in an embodiment of the present application, when training the initial network, a fourth feature tensor corresponding to the high-definition face image is obtained through the feature guidance subnetwork, and a fifth feature tensor corresponding to the low-definition face image is obtained through the super-resolution subnetwork, and the fourth feature tensor and the fifth feature tensor are constrained to obtain a constrained feature tensor. After the second image is generated based on the constrained feature tensor, a second loss can be determined based on the fourth feature tensor, the constrained feature tensor, the high-definition face image, and the second reconstructed image, and the encoder and decoder are corrected according to the second loss function to obtain a super-resolution network.

[0092] It should be noted that in an embodiment of the present application, when training the initial network including a feature guidance subnetwork and a super-resolution subnetwork, the loss function can be calculated based on the fourth feature tensor corresponding to the obtained high-definition face image, the constrained feature tensor after constraint, and the second reconstructed image, wherein any one or more loss functions can be used to determine the second loss.

[0093] It can be understood that in the embodiments of the present application, the calculated second loss can be used to constrain the consistency between the input high-definition face image and the output second reconstructed image, correct the structural parameters and weight parameters of the encoder and decoder, thereby completing the training of the entire initial network and obtaining a super-resolution network.

[0094] Exemplarily, in some embodiments, the fourth eigentensor Z p And the constrained feature tensor Z l Determine the loss function L′ d and L′ c , as follows:

[0095]

[0096] Exemplarily, in some embodiments, the high-definition face image I h and the second reconstructed image Determine the loss function L′ l1 , L′ per , L′ adv , as follows:

[0097]

[0098]

[0099]

[0100] Exemplarily, in some embodiments, the loss function obtained by calculation can be L′ l1 , L′per , L′ adv , determine the second loss.

[0101] In summary, through the super-resolution method of facial images proposed in step 101 above, the input facial image to be processed can be reconstructed using a super-resolution network to obtain a corresponding super-resolution facial image. The super-resolution network is obtained by training a feature dictionary in a pre-trained feature dictionary model. Since effective feature information in high-definition facial images can be mined during the learning process of the feature dictionary, the super-resolution network trained using the learned feature dictionary has higher robustness and can obtain a more accurate super-resolution effect.

[0102] It should be noted that the super-resolution method for facial images proposed in the embodiment of the present application, compared with the method based on geometric features and reference features, can obtain more accurate and robust face priors, and can more stably improve the clarity of face super-resolution, because the super-resolution network for facial image super-resolution is obtained by training with a feature dictionary.

[0103] It should be noted that, compared with the method based on generating face priors, the super-resolution method of facial images proposed in the embodiment of the present application does not directly use the generation network to restore details, and can effectively suppress the distortion of the repaired image caused by over-generation.

[0104] It can be seen that the super-resolution method of facial images proposed in the embodiments of the present application can effectively improve the quality of the input facial images to be processed, especially the facial clarity under extreme imaging conditions such as at night and in dim light.

[0105] Exemplarily, in some embodiments, the figure is a schematic diagram of the effect of super-resolution implementation. Figure 1 , the picture shows the effect of super resolution Figure 2 It can be seen that, compared with the input face image to be processed, after super-resolution processing using the super-resolution method for face images proposed in the embodiment of the present application, the face clarity of the reconstructed image obtained is significantly improved.

[0106] It should be noted that the super-resolution method for facial images proposed in the embodiment of the present application is not limited to being deployed on mobile phones. Any smart device with a CPU, GPU or NPU chip can be deployed using this solution.

[0107] The embodiment of the present application provides a super-resolution method for facial images, wherein a terminal device performs super-resolution processing on a facial image to be processed based on a super-resolution network to obtain a super-resolution facial image corresponding to the facial image to be processed; wherein the super-resolution network is obtained by training a feature dictionary in a pre-trained feature dictionary model. That is to say, in the embodiment of the present application, the super-resolution network for super-resolution processing of the facial image to be processed is trained using a pre-trained feature dictionary. Since effective feature information in high-definition facial images can be mined during the training of the feature dictionary, the feature dictionary has rich facial details, and more facial details can be implicitly applied to image reconstruction during the training of the super-resolution network guided by the feature dictionary. It can be seen that in the embodiment of the present application, the super-resolution network trained with a learned feature dictionary has higher robustness, and then the super-resolution network is used for super-resolution processing of facial images, which can obtain a more accurate super-resolution effect, improve the super-resolution effect of facial images, and effectively improve the super-resolution performance of the terminal device.

[0108] Based on the above embodiments, yet another embodiment of the present application proposes a face super-resolution method based on a feature dictionary, that is, a super-resolution method for face images.

[0109] It should be noted that in an embodiment of the present application, the terminal device can first use VQAVE to perform pre-training on HD SLR face data (face data set) to extract effective high-definition face details from the HD SLR face, such as skin texture, hair, etc., that is, a feature dictionary.

[0110] Further, in the embodiment of the present application, after the training of the feature dictionary is completed, the feature dictionary can be used to guide the learning of the super-resolution network. That is, these effective details (feature dictionary) can be used to guide the super-resolution network to enhance the details of the low-definition face.

[0111] That is to say, in the embodiment of the present application, the super-resolution network used for super-resolution processing of face images is trained using a pre-trained feature dictionary.

[0112] Furthermore, in an embodiment of the present application, the training process of the super-resolution network may include two stages: feature dictionary training and super-resolution network training.

[0113] Exemplarily, in some embodiments, Fig. 9 This is a schematic diagram of the training process of the feature dictionary, such as Fig. 9 As shown, in the process of training the feature dictionary, the encoder can first be used to map the high-definition face image H1 to the feature space to obtain the feature tensor Z1, wherein the encoder is composed of a common convolutional network.

[0114] Next, the feature tensor Z1 is matched through the feature dictionary before being input into the decoder.

[0115] For example, the best matching dictionary vector is found for each vector of Z1 in the feature dictionary to form a new feature tensor Z2, where the Euclidean distance is used for matching.

[0116] Next, Z2 is input into the encoder to obtain the reconstructed image H2.

[0117] It should be noted that, in the embodiment of the present application, during the training process of the feature dictionary, the entire network and the feature dictionary may be trained by constraining the consistency between the input image H1 and the reconstructed image H2.

[0118] It should be noted that, in the embodiments of the present application, during the training process, the feature dictionary may be randomly initialized by Gaussian distribution.

[0119] It is understandable that in the embodiments of the present application, a model composed of an encoder, a decoder and a feature dictionary can be used as a feature dictionary model. The encoder can be used to extract feature tensors from an input image, the feature dictionary can be used to perform vector matching, and the decoder can reconstruct an image based on the input feature tensor.

[0120] It should be noted that in the embodiment of the present application, during the training process of the feature dictionary model, the specific training loss function is shown in the following formula:

[0121]

[0122]

[0123] L′ l1 =‖H1-H2‖1 (13)

[0124]

[0125] L′ adv =[logD(H1)-log(1-D(H2))] (15)

[0126] Exemplarily, in some embodiments, the loss function L′ can be obtained based on the calculation d , L′ c , L′ l1 , L′ per , L′ adv , determine the first loss, as follows:

[0127] L Dict =L′ l1 +λper L′ per +λ adv L′ adv +λ d L′ d +λ c L′ c (16)

[0128] Among them, λ per ,λ adv ,λ d ,λ c Can represent the weights between different loss functions.

[0129] Exemplarily, in some embodiments, Fig.10 The flowchart of the feature dictionary guiding the super-resolution network training is as follows: Fig.10 As shown, the trained initial network can include a feature dictionary guidance module (feature guidance sub-network) and a super-resolution module (super-resolution sub-network).

[0130] Exemplarily, in some embodiments, the feature dictionary guidance module is mainly composed of an encoder and a feature dictionary. The weights and model parameters of these two parts come from the training of the feature dictionary in the first stage. The input of this part is a high-definition face image H3. H3 is obtained after passing through the encoder to obtain Z3, and Z3 is matched with the feature dictionary to obtain the encoded feature tensor Z4. The encoded feature tensor Z4 is used to guide the training of the super-resolution module, that is, to guide the training of the super-resolution network.

[0131] Exemplarily, in some embodiments, the super-resolution module may be composed of an encoder and a decoder. The weights and model parameters of these two parts also come from the training of the feature dictionary in the first stage. The input of this part is a low-quality input image (low-definition face image) L1 corresponding to the high-definition face image H3.

[0132] Exemplarily, in some embodiments, L1 is first input into the encoder to obtain the encoded feature tensor Z5. In the feature space, the constraints Z4 and Z5 are consistent, and in this way, the facial detail features stored in the feature dictionary are implicitly fed into the super-resolution network. At the same time, the constrained Z5 is mapped back to the image space through the decoder to obtain the reconstructed image H4.

[0133] It should be noted that in the embodiment of the present application, during the training process of the initial model, the specific training loss function is shown in the following formula:

[0134] L′ l1 =‖H3-H4‖1 (17)

[0135]

[0136] L′adv =[logD(H3)-log(1-D(H4))] (19)

[0137]

[0138] In summary, through the super-resolution method of facial images proposed in the embodiment of the present application, a super-resolution network can be used to reconstruct the input facial image to be processed to obtain the corresponding super-resolution facial image. Among them, the super-resolution network is obtained by training the feature dictionary in the pre-trained feature dictionary model. Since effective feature information in high-definition facial images can be mined during the learning process of the feature dictionary, the super-resolution network trained with the learned feature dictionary has higher robustness and can obtain a more accurate super-resolution effect.

[0139] It should be noted that the super-resolution method for facial images proposed in the embodiment of the present application, compared with the method based on geometric features and reference features, can obtain more accurate and robust face priors, and can more stably improve the clarity of face super-resolution, because the super-resolution network for facial image super-resolution is obtained by training with a feature dictionary.

[0140] It should be noted that, compared with the method based on generating face priors, the super-resolution method of facial images proposed in the embodiment of the present application does not directly use the generation network to restore details, and can effectively suppress the distortion of the repaired image caused by over-generation.

[0141] It can be seen that the super-resolution method of facial images proposed in the embodiments of the present application can effectively improve the quality of the input facial images to be processed, especially the facial clarity under extreme imaging conditions such as at night and in dim light.

[0142] Exemplarily, in some embodiments, the figure is a schematic diagram of the effect of super-resolution implementation. Figure 1 , the picture shows the effect of super resolution Figure 2 It can be seen that, compared with the input face image to be processed, after super-resolution processing using the super-resolution method for face images proposed in the embodiment of the present application, the face clarity of the reconstructed image obtained is significantly improved.

[0143] It should be noted that the super-resolution method for facial images proposed in the embodiment of the present application is not limited to being deployed on mobile phones. Any smart device with a CPU, GPU or NPU chip can be deployed using this solution.

[0144] The embodiment of the present application provides a super-resolution method for facial images, wherein a terminal device performs super-resolution processing on a facial image to be processed based on a super-resolution network to obtain a super-resolution facial image corresponding to the facial image to be processed; wherein the super-resolution network is obtained by training a feature dictionary in a pre-trained feature dictionary model. That is to say, in the embodiment of the present application, the super-resolution network for super-resolution processing of the facial image to be processed is trained using a pre-trained feature dictionary. Since effective feature information in high-definition facial images can be mined during the training of the feature dictionary, the feature dictionary has rich facial details, and more facial details can be implicitly applied to image reconstruction during the training of the super-resolution network guided by the feature dictionary. It can be seen that in the embodiment of the present application, the super-resolution network trained with a learned feature dictionary has higher robustness, and then the super-resolution network is used for super-resolution processing of facial images, which can obtain a more accurate super-resolution effect, improve the super-resolution effect of facial images, and effectively improve the super-resolution performance of the terminal device.

[0145] Based on the above embodiment, in another embodiment of the present application, Fig.11 Schematic diagram of the structure of the terminal device proposed in the embodiment of the present application Figure 1 ,like Fig.11 As shown, the terminal device 10 proposed in the embodiment of the present application may include a processing unit 111,

[0146] The processing unit 111 is used to perform super-resolution processing on the face image to be processed based on a super-resolution network to obtain a super-resolution face image corresponding to the face image to be processed; wherein the super-resolution network is obtained by feature dictionary training in a pre-trained feature dictionary model.

[0147] Furthermore, in an embodiment of the present application, the super-resolution network includes an encoder and a decoder; the processing unit 111 is also used to input the face image to be processed into the encoder, and output a first feature tensor corresponding to the face image to be processed; input the first feature tensor into the decoder, and output the super-resolution face image.

[0148] Furthermore, in an embodiment of the present application, the processing unit 111 is also used to train the feature dictionary model based on a face image data set; train an initial network based on the face image data set and the feature dictionary model to obtain the super-resolution network; wherein the face image data set includes high-definition face images and corresponding low-definition face images, and the resolution of the high-definition face images is greater than that of the low-definition face images.

[0149] Furthermore, in an embodiment of the present application, the feature dictionary model includes the encoder, the decoder and the feature dictionary, and the processing unit 111 is also used to input the high-definition face image into the encoder, and output a second feature tensor corresponding to the high-definition face image; perform matching processing on the second feature tensor based on the feature dictionary to determine a third feature tensor; input the third feature tensor into the decoder, and output a first reconstructed image corresponding to the high-definition face image; determine a first loss based on the second feature tensor, the third feature tensor, the high-definition face image and the first reconstructed image, and correct the encoder, the decoder and the feature dictionary according to the first loss.

[0150] Furthermore, in an embodiment of the present application, the processing unit 111 is also used to perform vector matching on the vector of the second feature tensor based on the dictionary vector in the feature dictionary to determine the matching vector corresponding to the vector of the second feature tensor; update the second feature tensor according to the matching vector to determine the third feature tensor.

[0151] Furthermore, in an embodiment of the present application, the initial network includes a feature guidance subnetwork and a super-resolution subnetwork, the super-resolution subnetwork includes the encoder and the decoder, and the processing unit 111 is also used to input the high-definition face image into the feature guidance subnetwork, and output a fourth feature tensor corresponding to the high-definition face image; input the low-definition face image into the encoder in the super-resolution subnetwork, and output a fifth feature tensor; perform feature constraints based on the fourth feature tensor and the fifth feature tensor to determine the constrained feature tensor, and input the constrained feature tensor into the decoder in the super-resolution subnetwork to determine a second reconstructed image; determine a second loss based on the fourth feature tensor, the constrained feature tensor, the high-definition face image, and the second reconstructed image, and correct the encoder and the decoder according to the second loss function to obtain the super-resolution network.

[0152] Furthermore, in an embodiment of the present application, the feature guidance subnetwork includes the encoder and the feature dictionary, and the processing unit 111 is also used to input the high-definition face image into the encoder in the feature guidance subnetwork, and output the sixth feature tensor corresponding to the high-definition face image; match the sixth feature tensor based on the feature dictionary to determine the fourth feature tensor.

[0153] In the embodiments of the present application, further, Fig.12 Schematic diagram of the structure of the terminal device proposed in the embodiment of the present application Figure 2 ,like Fig.12As shown, the terminal device 10 proposed in the embodiment of the present application may also include a processor 121 and a memory 122 storing executable instructions of the processor 121. Further, the software testing device 10 may also include a communication interface 123 and a bus 124 for connecting the processor 121, the memory 122 and the communication interface 123.

[0154] In the embodiment of the present application, the above-mentioned processor 121 can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other, and the embodiment of the present application is not specifically limited. The software testing device 10 can also include a memory 122, which can be connected to the processor 121, wherein the memory 122 is used to store executable program code, the program code includes computer operation instructions, and the memory 122 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0155] In the embodiment of the present application, the bus 124 is used to connect the communication interface 123, the processor 121 and the memory 122, and the mutual communication between these devices.

[0156] In the embodiment of the present application, the memory 122 is used to store instructions and data.

[0157] Furthermore, in an embodiment of the present application, the above-mentioned processor 121 is used to perform super-resolution processing on the face image to be processed based on a super-resolution network to obtain a super-resolution face image corresponding to the face image to be processed; wherein the super-resolution network is obtained by feature dictionary training in a pre-trained feature dictionary model.

[0158] In practical applications, the memory 122 may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or a combination of the above types of memories, and provide instructions and data to the processor 121.

[0159] In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.

[0160] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk, etc., which can store program code.

[0161] The embodiment of the present application provides a terminal device, which performs super-resolution processing on a face image to be processed based on a super-resolution network to obtain a super-resolution face image corresponding to the face image to be processed; wherein the super-resolution network is obtained by training a feature dictionary in a pre-trained feature dictionary model. That is to say, in the embodiment of the present application, the super-resolution network for super-resolution processing of the face image to be processed is trained using a pre-trained feature dictionary. Since effective feature information in high-definition face images can be mined during the training of the feature dictionary, the feature dictionary has rich face details, and more face details can be implicitly applied to image reconstruction by guiding the training of the super-resolution network through the feature dictionary. It can be seen that in the embodiment of the present application, the super-resolution network trained with a learned feature dictionary has higher robustness, and then the super-resolution network is used for super-resolution processing of face images, which can obtain more accurate super-resolution effects, improve the super-resolution effects of face images, and effectively improve the super-resolution performance of the terminal device.

[0162] An embodiment of the present application provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the super-resolution method for facial images as described above.

[0163] Specifically, the program instructions corresponding to the super-resolution method for a facial image in this embodiment can be stored on a storage medium such as a CD, a hard disk, or a USB flash drive. When the program instructions corresponding to the super-resolution method for a facial image in the storage medium are read or executed by an electronic device, the following steps are included:

[0164] The super-resolution processing is performed on the face image to be processed based on the super-resolution network to obtain the super-resolution face image corresponding to the face image to be processed; wherein the super-resolution network is obtained by feature dictionary training in a pre-trained feature dictionary model.

[0165] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0166] The present application is described with reference to implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0167] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0168] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the steps in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0169] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A super-resolution method for face images, characterized in that: The method comprises: The super-resolution processing is performed on the face image to be processed based on the super-resolution network to obtain the super-resolution face image corresponding to the face image to be processed; wherein the super-resolution network is obtained by feature dictionary training in a pre-trained feature dictionary model.

2. The method according to claim 1, characterized in that The super-resolution network includes an encoder and a decoder; the super-resolution network is used to perform super-resolution processing on the face image to be processed to obtain a super-resolution face image corresponding to the face image to be processed, including: Inputting the face image to be processed into the encoder, and outputting a first feature tensor corresponding to the face image to be processed; The first feature tensor is input into the decoder, and the super-resolution face image is output.

3. The method according to claim 2, characterized in that The method further comprises: Training the feature dictionary model based on a face image dataset; An initial network is trained according to the facial image data set and the feature dictionary model to obtain the super-resolution network; wherein the facial image data set includes high-definition facial images and corresponding low-definition facial images, and the resolution of the high-definition facial images is greater than that of the low-definition facial images.

4. The method according to claim 3, characterized in that The feature dictionary model includes the encoder, the decoder and the feature dictionary, and the step of training the feature dictionary model according to the high-definition face image includes: Inputting the high-definition face image into the encoder, and outputting a second feature tensor corresponding to the high-definition face image; Performing matching processing on the second feature tensor based on the feature dictionary to determine a third feature tensor; Inputting the third feature tensor into the decoder, and outputting a first reconstructed image corresponding to the high-definition face image; A first loss is determined based on the second feature tensor, the third feature tensor, the high-definition face image, and the first reconstructed image, and the encoder, the decoder, and the feature dictionary are corrected according to the first loss.

5. The method according to claim 4, characterized in that The matching the second feature tensor based on the feature dictionary to determine a third feature tensor includes: Performing vector matching on the vector of the second feature tensor based on the dictionary vector in the feature dictionary to determine a matching vector corresponding to the vector of the second feature tensor; The second feature tensor is updated according to the matching vector to determine the third feature tensor.

6. The method according to any one of claims 3 to 5, characterized in that: The initial network includes a feature guidance subnetwork and a super-resolution subnetwork, the super-resolution subnetwork includes the encoder and the decoder, and the initial network is trained according to the face image data set and the feature dictionary model to obtain the super-resolution network, including: Inputting the high-definition face image into the feature guidance subnetwork, and outputting a fourth feature tensor corresponding to the high-definition face image; Inputting the low-definition face image into the encoder in the super-resolution sub-network, and outputting a fifth feature tensor; Performing feature constraints based on the fourth feature tensor and the fifth feature tensor to determine a constrained feature tensor, and inputting the constrained feature tensor into the decoder in the super-resolution subnetwork to determine a second reconstructed image; A second loss is determined based on the fourth feature tensor, the constrained feature tensor, the high-definition face image, and the second reconstructed image, and the encoder and the decoder are corrected according to the second loss function to obtain the super-resolution network.

7. The method according to claim 6, characterized in that The feature guidance sub-network includes the encoder and the feature dictionary, and the inputting the high-definition face image into the feature guidance sub-network and outputting the fourth feature tensor corresponding to the high-definition face image includes: Inputting the high-definition face image into the encoder in the feature guidance subnetwork, and outputting a sixth feature tensor corresponding to the high-definition face image; The sixth feature tensor is matched based on the feature dictionary to determine the fourth feature tensor.

8. A terminal device, characterized in that: The terminal device comprises: a processing unit, The processing unit is used to perform super-resolution processing on the face image to be processed based on a super-resolution network to obtain a super-resolution face image corresponding to the face image to be processed; wherein the super-resolution network is obtained by feature dictionary training in a pre-trained feature dictionary model.

9. A terminal device, characterized in that: The terminal device comprises: a processor and a memory; wherein, The memory is used to store a computer program that can be run on the processor; The processor is configured to execute the method according to any one of claims 1 to 7 when running the computer program.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images