Image processing method, electronic equipment and storage medium

Abstract

The invention provides an image processing method, electronic equipment and a storage medium, and relates to the field of image processing.The method comprises the steps that a first interface is displayed, and a face image is obtained in response to a first operation on a first control in the first interface; processing the face image by using a trained inverse rendering model to obtain a face normal image, a face albedo image and an environment image; and performing enhancement processing on the face image, the face normal image, the face albedo image and the environment image by using a trained face enhancement model to obtain a face target image. In the implementation mode, the processing result of the inverse rendering model on the face image is used as the face prior information, the face enhancement model can effectively utilize the face prior information to accurately recover the face, and even under the complex light and shadow condition, generation of unnatural textures can be avoided, and a high-quality shot image can be output.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method, electronic device and storage medium. Background Art

[0002] With the popularity of electronic devices with shooting functions in life, people use electronic devices to shoot has become a daily behavior. However, during the shooting process, the image quality is often poor due to reasons such as dim light, lens shaking, and movement of the subject.

[0003] Although many methods for enhancing image quality have been proposed in current technology, these methods can also cause new problems while improving image quality, especially in face image processing. For example, the enhanced image may have face blur and pseudo texture, which seriously affects the quality of face images. Therefore, a new image processing method is urgently needed to solve these problems. Summary of the invention

[0004] The present application provides an image processing method, an electronic device and a storage medium, which use the processing result of the inverse rendering model on the face image as the face prior information. The face enhancement model can effectively use the face prior information to accurately restore the face, avoid the generation of unnatural textures even under complex lighting conditions, and output high-quality captured images.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions: In a first aspect, an image processing method is provided, which includes: displaying a first interface, the first interface including a first control; obtaining a face image in response to a first operation on the first control; processing the face image using a trained inverse rendering model to obtain a face normal image, a face albedo image, and an environment image; and enhancing the face image, the face normal image, the face albedo image, and the environment image using a trained face enhancement model to obtain a face target image.

[0006] Optionally, the first operation is used to instruct to start shooting, and the first operation may be a click operation on a shooting control.

[0007] Optionally, the face normal image is used to represent the normal feature corresponding to the portrait in the face image, or in other words, to represent the normal direction of the face surface in the face image.

[0008] Optionally, the face albedo image is used to represent the albedo features corresponding to the portrait in the face image, or in other words, to represent features such as color and texture of the face surface in the face image.

[0009] Optionally, the environment image is used to represent environment contents other than the portrait, or in other words, to represent the influence of lighting of the shooting environment.

[0010] Optionally, the definition of the face target image is greater than the definition of the face image.

[0011] Optionally, the face target image is rich in details and does not have problems such as strong smearing, artifacts, and pseudo textures.

[0012] In this implementation, the face and environment in the face image are separated by the trained inverse rendering model to obtain the face normal image, face albedo image and environment image, that is, accurate prior information. The trained face enhancement model makes full use of this prior information during the enhancement process, which can effectively restore the details of the face image, eliminate the smearing, artifacts, pseudo textures, etc. in the face image, and improve the clarity and quality of the face image.

[0013] In combination with the first aspect, in certain implementations of the first aspect, in response to a first operation on a first control, obtaining a facial image includes: in response to a first operation on the first control, obtaining an original image: determining the clarity of the original image; detecting that the clarity of the original image is less than a preset clarity threshold, and cropping the original image to obtain a facial image.

[0014] In this implementation, the original image is effectively screened by definition, so that the determined face images are all face images that need to be enhanced. On the one hand, it is convenient to process only some face images that need to be enhanced in the future, which can improve the image processing efficiency; on the other hand, it avoids enhancing face images that do not need to be enhanced, saving the resources of electronic equipment and reducing power consumption; at the same time, it can also avoid enhancing face images that do not need to be enhanced, which may lead to artifacts and false textures.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the trained inverse rendering model includes a first network, a second network, and a third network, and the trained inverse rendering model is used to process a face image to obtain a face normal image, a face albedo image, and an environment image, including: using the first network to process the face image to obtain a face normal image; using the second network to process the face image and the face normal image to obtain a face albedo image; using the third network to process the face image, the face normal image, and the face albedo image to obtain an environment image.

[0016] Optionally, the first network is VQVAE, and / or the second network is VQVAE.

[0017] Optionally, the third network may be VQVAE or U-Net.

[0018] In this implementation, the VQ technology is introduced into each network contained in the inverse rendering model, which can extract accurate prior information, and help the subsequent face enhancement model to effectively use this prior information to enhance the face image, avoid the generation of false textures, artifacts, unnatural effects, etc., and greatly improve the image quality.

[0019] In combination with the first aspect, in certain implementations of the first aspect, the first network includes a first vector codebook module, the first vector codebook module includes a first discrete vector set, and the face image is processed using the first network to obtain a face normal image, including: using the first network to extract a first latent vector in the face image; searching for a vector close to the first latent vector in the first discrete vector set; and generating a face normal image based on the vector close to the first latent vector.

[0020] Optionally, the first latent vector may include 3D structure information and geometric information corresponding to the face.

[0021] In this implementation, vector quantization through VQVAE can effectively map the information in the continuous latent space into a discrete space, which helps to improve the computational efficiency and generalization ability of the first network. In this way, the first network can extract rich 3D structural information and geometric information from the input face image, laying the foundation for the subsequent enhancement of the face image, that is, it helps to more accurately restore the 3D structural features and geometric features of the face when the face image is enhanced.

[0022] In combination with the first aspect, in certain implementations of the first aspect, the second network includes a second vector codebook module, the second vector codebook module includes a second discrete vector set, and the face image and the face normal image are processed using the second network to obtain a face albedo image, including: splicing the face image and the face normal image to obtain a first spliced ​​image; extracting a second latent vector in the first spliced ​​image using the second network; searching for a vector close to the second latent vector in the second discrete vector set; and generating a face albedo image based on the vector close to the second latent vector.

[0023] Optionally, the second latent vector may include texture information and skin information corresponding to the face.

[0024] In this implementation, vector quantization through VQVAE can effectively map the information in the continuous latent space into a discrete space, which helps to improve the computational efficiency and generalization ability of the second network. In this way, the second network can extract accurate texture information and skin information from the input face image and face normal image, laying the foundation for the subsequent enhancement of the face image, that is, it helps to more accurately restore the texture features and skin features of the face when the face image is enhanced, improve the details and realism of the face image, and avoid the generation of pseudo-textures, artifacts, unnatural effects, etc.

[0025] In combination with the first aspect, in certain implementations of the first aspect, the trained face enhancement model includes a third vector codebook module, the third vector codebook module includes a third discrete vector set, and the trained face enhancement model is used to enhance the face image, the face normal image, the face albedo image, and the environmental image to obtain a face target image, including: splicing the face image, the face normal image, the face albedo image, and the environmental image to obtain a second spliced ​​image; using the trained face enhancement model to extract the third latent vector in the second spliced ​​image; searching for a vector close to the third latent vector in the third discrete vector set; and generating a face target image based on the vector close to the third latent vector.

[0026] Optionally, the third latent vector may include 3D structure information, geometric information, texture information, skin information, ambient lighting information, light and shadow information, etc. corresponding to the face.

[0027] In this implementation, the face enhancement model can obtain accurate 3D structure information, geometry information, texture information, skin information, ambient lighting information, light and shadow information, etc. from prior information. Therefore, even in the case of poor quality of face image and complex facial lighting and shadow, it can correctly guide the restoration of the 3D structure, geometry, texture, skin, ambient lighting, light and shadow and other features of the face image based on this information, while effectively avoiding the generation of pseudo-texture and artifacts, thereby improving the image quality and the robustness of the face enhancement model, making the face enhancement model more widely applicable.

[0028] In combination with the first aspect, in certain implementations of the first aspect, a trained face enhancement model is used to enhance a face image, a face normal image, a face albedo image, and an environment image to obtain a face target image, including: using a trained light and shadow enhancement model to determine a fill-light environment image corresponding to the environment image; using a trained face enhancement model to enhance a face image, a face normal image, a face albedo image, and a fill-light environment image to obtain a face target image.

[0029] In this implementation, the light and shadow enhancement model is used to intelligently fill in the ambient image, and then the face enhancement model is used to fine-tune the face image based on the filled-in ambient image. This processing not only enhances the facial details, but also achieves a comprehensive fill-in effect for the face and the surrounding environment, ensuring the balance of the face and ambient lighting, which is conducive to generating a face target image with higher clarity and stronger contrast.

[0030] In combination with the first aspect, in certain implementations of the first aspect, the image processing method further includes: using a first training set to train an initial inverse rendering network to obtain an inverse rendering model in training; the first training set includes high-definition sample images; using a second training set to train the inverse rendering model in training to obtain a trained inverse rendering model; the second training set includes low-definition sample images, and the low-definition sample images are obtained by degrading high-definition sample images.

[0031] In this implementation, the training of the inverse rendering model is divided into two stages. In the first stage, the initial inverse rendering network is trained with high-definition sample images, so that the vector codebook module can capture and save the prior information of the high-definition sample images; in the second stage, further training is performed with low-definition sample images, so that the inverse rendering model in training can learn how to accurately use the prior information to restore and output high-quality images. Through these two stages of training, the trained inverse rendering model has strong robustness, so that when facing face images in various complex light and shadow scenes, it can also accurately extract and use the prior information to generate high-quality images.

[0032] In combination with the first aspect, in certain implementations of the first aspect, the image processing method also includes: using a third training set to train an initial face enhancement network to obtain a face enhancement model in training; the third training set includes high-definition sample images; using a fourth training set to train the face enhancement model in training to obtain a trained face enhancement model; the fourth training set includes low-definition sample images, and normal sample images, albedo sample images, and environmental sample images corresponding to the low-definition sample images; the normal sample images and albedo sample images are obtained by processing the low-definition sample images using a trained inverse rendering model; and the environmental sample images are simulated by a trained light and shadow enhancement model.

[0033] In this implementation, the training of the face enhancement model is divided into two stages. In the first stage, the initial face enhancement network is trained through high-definition sample images, so that the vector codebook module in the initial face enhancement network can learn how to quantize the vector so that it can accurately use the prior information to restore and output high-quality face images. In the second stage, further training is performed through low-definition sample images, so that the face enhancement model in training can learn how to accurately use the prior information to restore and output high-quality images. Through these two stages of training, the trained face enhancement model has strong robustness, so that when faced with face images in various complex light and shadow scenes, it can also accurately use the prior information to generate high-quality images.

[0034] In combination with the first aspect, in certain implementations of the first aspect, the image processing method also includes: in the process of training the face enhancement model, calculating the target loss value according to a preset loss function; according to the target loss value, continuing to train the face enhancement model in training in a back propagation manner to obtain a trained face enhancement model.

[0035] Optionally, the target loss value is used to represent the loss between the high-definition sample image and the predicted image output by the face enhancement model in training, and the loss consists of three parts, the first part is the L1 loss, the second part is the perceptual loss, and the third part is the adversarial loss.

[0036] Among them, L1 loss is used to represent the loss between the high-definition sample image and the predicted image (or reconstructed image); perceptual loss is used to represent the feature loss between the high-definition sample image and the predicted image (or reconstructed image); adversarial loss is used to represent the loss between the real image and the image generated by the generator in the face enhancement model.

[0037] In this implementation, the back-propagation algorithm is used to adjust the model parameters, which helps the model converge to the optimal solution faster and improves the training speed of the face enhancement model. The losses of the three parts of the face enhancement model are taken into consideration when adjusting the model parameters, which can improve the performance of the face enhancement model in many aspects and help the face enhancement model cope with various low-quality images with complex lighting and shadows.

[0038] In combination with the first aspect, in certain implementations of the first aspect, the image processing method further includes: splicing the original image and the target face image to obtain a high-definition image corresponding to the original image.

[0039] In this implementation, the face image is enhanced specifically. On the one hand, the details of the face area can be effectively restored, and the smearing, artifacts, and pseudo textures existing in the face image can be eliminated, ensuring that the image quality of the face area is significantly improved; on the other hand, the face image is enhanced separately, and then the original image and the face target image are spliced ​​to obtain a high-definition image, which can effectively improve the image processing efficiency and reduce the resource consumption of electronic devices. In addition, users are usually more concerned about the faces in the captured images. The image processing method provided by this application can significantly improve the image quality of the face area while improving the image processing speed, which can greatly improve the user's favorability.

[0040] In a second aspect, the present application provides an image processing device, which includes one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the image processing device to execute any one of the methods in the technical solution provided in the first aspect.

[0041] In a third aspect, the present application provides an electronic device, comprising: a processor, a memory, and an interface; the processor, the memory, and the interface cooperate with each other so that the electronic device executes any one of the methods in the technical solution provided in the first aspect.

[0042] In a fourth aspect, the present application provides a chip, including a processor. The processor is used to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation thereof.

[0043] Optionally, the chip also includes a memory, and the memory is connected to the processor via a circuit or wire.

[0044] Optionally, the chip also includes a communication interface.

[0045] In a fifth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes any one of the methods in the technical solution of the first aspect.

[0046] In a sixth aspect, the present application provides a computer program product, the computer program product comprising: a computer program code, when the computer program code runs on an electronic device, the electronic device executes any one of the methods in the technical solution of the first aspect.

[0047] The technical effects obtained by the above-mentioned second, third, fourth, fifth and sixth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a schematic diagram of a scene for capturing an image shown in an exemplary embodiment of the present application; Figure 2 is a schematic diagram showing a captured image according to an exemplary embodiment of the present application; Figure 3 is a flowchart of an image processing method shown in an exemplary embodiment of the present application; Figure 4 It is a flowchart of a method for acquiring a face image shown in an exemplary embodiment of the present application; Figure 5 is a schematic diagram of a face image shown in an exemplary embodiment of the present application; Figure 6 is a flowchart of another image processing method shown in an exemplary embodiment of the present application; Figure 7 is a schematic diagram of a trained inverse rendering model structure shown in an exemplary embodiment of the present application; Figure 8 is a schematic diagram of a face image, a face normal image, a face albedo image, and an environment image shown in an exemplary embodiment of the present application; Fig. 9 is a schematic diagram of an inverse rendering model processing flow diagram shown in an exemplary embodiment of the present application; Fig.10 is a schematic diagram of a first network structure shown in an exemplary embodiment of the present application; Fig.11 is a schematic diagram of a second network structure shown in an exemplary embodiment of the present application; Fig.12 is a schematic diagram of a face enhancement model processing flow diagram shown in an exemplary embodiment of the present application; Fig.13 is a structural schematic diagram of a face enhancement model shown in an exemplary embodiment of the present application; Fig.14 is a schematic diagram of a process of training an inverse rendering model according to an exemplary embodiment of the present application; Fig.15 is a schematic diagram of a light and shadow enhancement model enriching a high-definition sample image according to an exemplary embodiment of the present application; Fig.16 is a schematic diagram of a process of training a face enhancement model according to an exemplary embodiment of the present application; Fig.17 is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of the present application; Fig.18is a software structure block diagram of an electronic device shown in an exemplary embodiment of the present application; Fig.19 is a schematic diagram of an image processing device shown in an exemplary embodiment of the present application; Fig. 20 It is a schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In the embodiments of the present application, the following terms "first", "second", "third" and "fourth" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features.

[0050] In order to facilitate understanding of the technical solutions in the embodiments of the present application, some of the terms involved in the embodiments of the present application are explained before introducing the technical solutions in the embodiments of the present application.

[0051] 1. Convolutional Neuron Network (CNN) A convolutional neural network is a deep neural network with a convolutional structure. A convolutional neural network contains a feature extractor consisting of a convolutional layer and a subsampling layer. The feature extractor can be regarded as a filter. A convolutional layer refers to a neuron layer in a convolutional neural network that performs convolution processing on the input signal. In a convolutional layer of a convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units.

[0052] 2. RGB (Red, Green, Blue) color space The RGB color space can also be called the RGB domain, which refers to a color model related to the structure of the human visual system. According to the structure of the human eye, all colors are regarded as different combinations of red, green, and blue.

[0053] 3. RGB (Red, Green, Blue) channel values RGB channel values ​​can also be called RGB values, which refer to the intensity values ​​of the three color channels of red (Red), green (Green), and blue (Blue) used to define a specific color in the RGB color model. These intensity values ​​together determine the final displayed color.

[0054] 4. Backlight Backlighting is a condition where the subject (also called the photographed object) is located between the light source and the camera. In this condition, it is easy to cause the subject to be underexposed. Therefore, in general, users should try to avoid shooting objects in backlighting conditions.

[0055] 5. Albedo Albedo usually refers to the ratio of the reflected radiation flux to the incident radiation flux under the influence of solar radiation. It is an important variable reflecting many surface parameters, such as the surface's ability to absorb solar radiation.

[0056] In the embodiment of the present application, albedo refers to the ratio of the reflected radiation flux to the incident radiation flux of the portrait head under the illumination of light, which is used to reflect the absorption capacity of the surface layer of the portrait head (such as the face, scalp, etc.) to the illumination radiation.

[0057] 6. Normal In the embodiment of the present application, the normal direction refers to the direction of the normal line.

[0058] 7. Receptive Field In the deep neural network in the field of machine vision, there is a concept called receptive field, which is used to indicate the size of the range of perception of neurons at different positions within the network to the original image.

[0059] 8. A priori Priority refers to inherent knowledge or preset information about facial structure and features, which can help electronic devices improve accuracy and efficiency when processing facial images.

[0060] 9. Back Propagation (BP) The neural network can use the error back propagation algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges.

[0061] The back-propagation algorithm is a back-propagation movement dominated by error loss, which is used to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0062] The above is a brief introduction to the terms involved in the embodiments of the present application, which will not be repeated below.

[0063] With the popularity of electronic devices with shooting functions in our daily lives, people use electronic devices to shoot has become a daily behavior. However, during the shooting process, due to the influence of factors such as the hardware performance of the shooting device (such as the performance of the photosensitive element), the shooting environment (such as low light), lens shaking, telephoto camera shooting, and the movement of the subject, the image quality obtained is poor.

[0064] Although many methods for enhancing image quality have been proposed in current technology, these methods will also cause new problems while improving image quality, especially in the processing of face images. For example, the composition of the face is highly a priori, and traditional image enhancement methods (such as linear filtering methods, histogram equalization methods, sharpening enhancement methods, etc.) cannot utilize the a priori information of the face, resulting in the inability to obtain a better face contour when processing face images, and the inability to better enhance face details. Even when the light and shadow of the face are complex, it will introduce pseudo-textures that do not exist in the face and are abrupt and inconsistent. In other words, using traditional image enhancement methods to process face images will result in problems such as low clarity, insufficient details, strong smearing, noise, artifacts, and pseudo-textures in the processed face images.

[0065] Among them, lack of details refers to the loss of details. For example, the original subtle texture (such as skin texture, hair, etc.), edge, shape and other information in the face image are lost, resulting in the face image not looking clear enough or lacking three-dimensionality.

[0066] A strong smearing effect means that the colors and textures in the face image are over-smoothed, causing the face image to look like it has been smeared and lacks sharpness and clarity.

[0067] The existence of artifacts refers to the addition of non-real content to the processed face image.

[0068] For example, a face image has complex lighting and shadows. There is a dark shadow above the eyebrows of the face in the face image. When using traditional image enhancement methods for enhancement processing, it is impossible to accurately distinguish between real features and lighting effects, and the dark shadow is mistakenly enhanced as an eyebrow, resulting in two eyebrows appearing in the enhanced face image, one is a real eyebrow, and the other is an unreal eyebrow. The unreal eyebrow can be called an artifact. The face image processed in this way seriously affects the authenticity of the face image and the user experience.

[0069] The existence of pseudo-texture means that non-real texture information is added to the processed face image.

[0070] For example, the facial light and shadow of a face image are complex. There is a circle of unnatural light and shadow around the eyes in the face image. When using traditional image enhancement methods for enhancement processing, it is impossible to accurately distinguish between real features and light and shadow effects. This circle of light and shadow is mistakenly enhanced as the outline of glasses, resulting in glasses appearing in the enhanced face image, but the glasses do not really exist. The processed face image seriously affects the authenticity of the face image and the user experience.

[0071] In view of this, an embodiment of the present application provides an image processing method, or a facial image enhancement method, which can not only eliminate problems such as facial blur, facial smearing, noise, artifacts, pseudo texture, etc., but also supplement details and perform super-resolution reconstruction. Even in complex light and shadow conditions, it can process clear, complete, high-resolution, and high-quality facial images from poor quality images, thereby improving the user's shooting experience.

[0072] The image processing method may specifically include displaying a first interface, and acquiring a facial image in response to a first operation on a first control in the first interface; processing the facial image using a trained inverse rendering model to obtain a facial normal image, a facial albedo image, and an environmental image; enhancing the facial image, the facial normal image, the facial albedo image, and the environmental image using a trained face enhancement model to obtain a facial target image; the clarity of the facial target image is greater than the clarity of the facial image.

[0073] The image processing method provided in the embodiment of the present application uses a trained inverse rendering model to extract the face normal image, face albedo image and environment image corresponding to the captured image, thereby obtaining accurate face prior information, laying a solid foundation for subsequent face enhancement processing. In the enhancement stage, when the face image is enhanced using the trained face enhancement model, the face prior information can be used to accurately restore the face, effectively improve the face contour, enhance the face details, and avoid the generation of unnatural textures even under complex lighting conditions. Ultimately, the image processing method can output high-quality captured images, which are not only clear and detailed, but also free of problems such as strong smearing, artifacts, and pseudo-textures.

[0074] Before introducing the image processing method provided in the embodiment of the present application, the application scenario of the image processing method provided in the embodiment of the present application is illustrated with reference to the accompanying drawings.

[0075] Application scenario 1: Photo shooting The image processing method provided in the embodiment of the present application can be applied to photographing scenes. For example, it can be applied to photographing images in various scenes. Figure 1 , Figure 1 It is a schematic diagram of a scene for shooting an image shown in an exemplary embodiment of the present application.

[0076] In the embodiment of the present application, the electronic device is described as a mobile phone. For example, the display screen of the mobile phone displays the main interface, such as Figure 1The main interface UI1 shown in (a) in FIG. The main interface UI1 includes icons of multiple applications, such as a "video" application icon, a "gallery" application icon, a "camera" application icon, etc. When a user wants to take a picture, he can click the icon of the "camera" application, and the mobile phone responds to the click operation and runs the camera application. At the same time, the display screen of the mobile phone jumps from the main interface UI1 to the shooting interface UI2.

[0077] like Figure 1 The shooting interface UI2 shown in (b) of FIG. 1 includes a preview area 102, a shooting control 103, a preview thumbnail 104, and multiple shooting mode options. The preview area 102 is used to display a preview image; the shooting control 103 is used to instruct the mobile phone to shoot an image when receiving a shooting operation (such as clicking the shooting control 103); and the preview thumbnail 104 is used to display a thumbnail corresponding to the currently shot image.

[0078] The various shooting mode options may include: aperture mode, night scene mode, portrait mode, photo mode, video mode, professional mode, etc. It is understandable that the camera application is generally in the photo mode by default when it is opened.

[0079] For example, the light source of the shooting environment where the photographed object 101 is located is sunlight, and the user wants to shoot the photographed object 101, so he or she points the camera of the mobile phone at the photographed object 101. The camera captures the image corresponding to the photographed object 101 and displays the preview image in the preview area 102. When the user is satisfied with the effect of the preview image, he or she can click the shooting control 103. In response to the user's operation of clicking the shooting control 103, the mobile phone captures the preview image displayed in the current preview area 104, saves it as a photo after processing, and stores it in the mobile phone's gallery.

[0080] When the user wants to view the captured image, he can click the icon of the "Gallery" application, and the mobile phone responds to the click and displays the captured image. Alternatively, the user can click the preview thumbnail 104, and the mobile phone responds to the click and quickly displays the current captured image.

[0081] It is worth noting that in this shooting scene, the subject 101 is exactly between the light source and the camera, making the area where the subject 101 is located a dark area. At this time, the captured image obtained using relevant technologies will usually cause the face of the subject 101 in the captured image to have low clarity, insufficient details, strong smearing, artifacts, pseudo textures and other problems.

[0082] See also Figure 2 , Figure 2FIG. 1 is a schematic diagram showing a captured image according to an exemplary embodiment of the present application. Figure 2 The display interface UI3 shown in (a) is an interface for displaying images in the gallery application. In the embodiment of the present application, the display interface UI3 displays a captured image 105 obtained using relevant technologies. Obviously, the face of the subject 101 in the captured image 105 has problems such as low clarity (also called blur), insufficient details, and artifacts.

[0083] Through the image processing method provided in the embodiment of the present application, the trained inverse rendering model is used to extract the face normal image, face albedo image and environment image corresponding to the captured image, thereby obtaining accurate face prior information, laying a solid foundation for subsequent face enhancement processing. In the enhancement stage, when the face image is enhanced using the trained face enhancement model, the face prior information can be used to accurately restore the face, effectively improve the face contour, enhance the face details, and avoid the generation of unnatural textures even under complex lighting conditions. Ultimately, the image processing method can output high-quality captured images, which are not only clear and detailed, but also free of smearing, artifacts, pseudo-textures and other problems.

[0084] like Figure 2 The display interface UI4 shown in (b) is an interface for displaying images in the gallery application. In the embodiment of the present application, the display interface UI4 displays the captured image 106 obtained by using the image processing method provided in the embodiment of the present application. Obviously, compared with the captured image 105, the captured image 106 has higher quality. The face of the subject 101 in the captured image 106 is not only high in clarity and rich in details, but also free from problems such as strong smearing, artifacts, and false textures.

[0085] Application scenario 2: Video recording The image processing method provided in the embodiment of the present application can be applied to video recording scenarios. For example, it can be applied to recording videos in various scenarios.

[0086] Exemplarily, after the camera application is run, a plurality of shooting mode options are displayed in the shooting interface. The mobile phone switches the shooting mode from the photo mode to the video mode in response to the user's sliding operation on the plurality of shooting mode options. In the preview state, the preview image captured by the mobile phone can be displayed in real time in the preview area. When the user is satisfied with the effect of the preview image, the user can click the shooting control, and the mobile phone responds to the user's operation of clicking the shooting control, captures the preview image displayed in the current preview area in real time, and runs the program corresponding to the image processing method provided in the embodiment of the present application, obtains the shot video, and stores it in the gallery of the mobile phone.

[0087] For example, when the subject takes a selfie in a backlit environment, the area where the subject is located becomes a dark area due to the strong surrounding light. At this time, if the relevant technology is used for processing, the face of the subject in the acquired video will have problems such as low clarity, insufficient details, strong smearing, artifacts, and pseudo textures. If the image processing method provided by the embodiment of the present application is used for processing, the image in the captured video can be first split using the inverse rendering model to obtain accurate face prior information. In the enhancement stage, when the face image is enhanced using the face enhancement model, the face prior information can be used to accurately restore the face, effectively improve the face contour, enhance the face details, and avoid the generation of unnatural textures even under complex light and shadow conditions. Finally, the image processing method can output a high-quality video, which is not only high in clarity and rich in details, but also does not have problems such as strong smearing, artifacts, and pseudo textures.

[0088] Application scenario three: video call scenario The image processing method provided in the embodiment of the present application can also be applied to video call scenarios. For example, when a user makes a video call in a backlit environment, the area where the user is located becomes a dark area due to the strong surrounding light. At this time, if the relevant technology is used for processing, the user's face photographed by the electronic device during the video call will have problems such as low clarity, insufficient details, strong smearing, artifacts, and false textures. If the image processing method provided in the embodiment of the present application is used for processing, high-quality video can be output.

[0089] Application scenario 4: Image editing / image beautification (or photo editing) scenario The image processing method provided in the embodiment of the present application can also be applied to the image beautification scene. For example, the images stored in the gallery have problems such as low clarity, insufficient details, strong smearing, artifacts, and pseudo-textures. When the user wants to beautify the stored images (pre-shot images, or images downloaded from the Internet, etc.), click the control in the display interface (such as automatic optimization controls, intelligent photo editing controls, brightening controls, etc.), and the electronic device responds to the click operation and processes the image through the image processing method provided in the embodiment of the present application, which can improve the image clarity, enrich the image details, eliminate the problems such as strong smearing, artifacts, and pseudo-textures in the image, and obtain a high-quality image.

[0090] It should be understood that the application scenarios of the above-mentioned image processing methods are only examples and do not limit the actual application scenarios of the present application. The image processing method provided in the embodiments of the present application can also be applied but not limited to the following scenarios: scenarios of enhancing the processing of images and / or videos in video surveillance, scenarios of enhancing the processing of preview images in various applications, scenarios of enhancing the processing of images in video conferencing applications, scenarios of enhancing the processing of images in long and short video applications, scenarios of enhancing the processing of images in video live broadcast applications, scenarios of enhancing the processing of images in intelligent mirror application scenarios, etc.

[0091] The image processing method provided in the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0092] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of an image processing method according to an exemplary embodiment of the present application. The method comprises: S101. Display a first interface, where the first interface includes a first control.

[0093] For example, the user can instruct the electronic device to run the camera application by clicking the icon of the "camera" application. Generally, after running the camera application, the main camera of the electronic device, that is, the rear camera, is started by default. At the same time, the display interface of the electronic device displays the first interface, such as Figure 1 In the shooting interface UI2 shown in (b), the first interface may include a preview area, a first control, a preview thumbnail, and multiple shooting mode options.

[0094] Among them, the preview area can be used to display the preview image captured in real time by the rear camera; the first control can be a shooting control, which is used to instruct the electronic device to shoot an image when a shooting operation (such as clicking the first control) is received.

[0095] Optionally, embodiments of the present application also provide other ways to run the camera application. For example, when the electronic device is in a lock screen state, the user can instruct the electronic device to run the camera application by double-clicking the volume down button on the electronic device. Alternatively, when the electronic device is in a lock screen state, the lock screen interface includes an icon for the camera application, and the user instructs the electronic device to run the camera application by clicking the icon for the camera application. Alternatively, when the electronic device is running other applications, the other applications have permission to call the camera application; the user can instruct the electronic device to run the camera application by clicking the corresponding controls in other applications. For example, when the electronic device is running an instant messaging application, the user can instruct the electronic device to run the camera application by selecting the control of the camera function.

[0096] It should be understood that the above is an example of the operation of running the camera application; the electronic device can also be instructed to run the camera application through voice instructions or other operations; this application does not make any limitation on this. It should also be understood that running the camera application can refer to starting the camera application.

[0097] S102: In response to a first operation on a first control, obtaining a facial image.

[0098] Exemplarily, the first operation is used to instruct to start shooting, and the first operation may be a click operation on a shooting control.

[0099] It can be understood that the above description uses the example of a click operation as the first operation; in an embodiment of the present application, the first operation may also be an operation of shooting by voice instruction; or, the first operation may also be an operation of shooting by face recognition instruction; or, the first operation may also be an operation of shooting by gesture recognition instruction; or, the first operation may also be an operation of pressing a physical button (such as a volume button) to indicate shooting, etc.; the present application does not impose any limitation on this.

[0100] The camera application responds to various types of first operations triggered by the user and starts shooting, that is, the electronic device uses the camera to collect the original image. The camera can be any camera among the main camera, the front camera, the telephoto camera, the wide-angle camera, etc. It should be understood that the embodiment of the present application does not impose any restrictions on the type of camera.

[0101] It should be noted that the main camera has the characteristics of large light intake, high resolution, and moderate field of view. The main camera is generally used as the default camera of electronic devices. The telephoto camera has a longer focal length and a smaller field of view, which is suitable for shooting objects that are far away from the electronic device, that is, distant objects. The wide-angle camera has a shorter focal length and a larger field of view, which is suitable for shooting objects that are close to the electronic device, that is, nearby objects.

[0102] In a possible implementation, when the original image is an image captured in real time by a camera of an electronic device, the original image may be an RGB domain image, which means that the image is located in the RGB domain color space. The original image may also be a Raw domain image, which means that the image is acquired in the Raw color space, that is, the Raw domain image means that the image is located in the Raw color space. It should be understood that the original image may also be other color images such as a multispectral image, which is not limited in the embodiments of the present application.

[0103] In another possible implementation, the original image may also be an image pre-stored in the electronic device, for example, an image taken by the user and pre-stored in the gallery of the electronic device, or an image downloaded from the Internet, or an image received from other electronic devices, and the present application embodiment does not impose any restrictions on this.

[0104] The original image in the embodiment of the present application may contain a face. In one possible implementation, the original image containing the face may be directly used as the face image. In another possible implementation, the original image may be cropped to obtain the face image. For example, the face region in the original image is identified, the face region is cropped, and the image containing the face portion is retained to obtain the face image.

[0105] It is worth noting that the embodiment of the present application does not limit the number of faces. For example, the original image or face image may contain one or more faces.

[0106] S103, using the trained inverse rendering model to process the face image to obtain a face normal image, a face albedo image and an environment image.

[0107] The face normal image is used to represent the normal features corresponding to the portrait in the face image, or in other words, to represent the normal direction of the face surface in the face image. The face normal image can be used to capture the 3D information of the face, which is very important for the subsequent analysis of the shape and structure of the face.

[0108] The face albedo image is used to represent the albedo features corresponding to the portrait in the face image, or in other words, to represent the color, texture and other features of the face surface in the face image. The face albedo image is very important for the subsequent accurate analysis of the inherent attributes of the face (such as skin color, spots, wrinkles, etc.).

[0109] The environmental image is used to represent environmental content other than portraits, or in other words, to represent the influence of lighting in the shooting environment.

[0110] Among them, the normal feature can be understood as the direction of the normal at each point on the concave and convex surface of the object. For example, in the present application, the normal feature is the direction of the normal at each point on the surface of the face. The normal image of the face can mark the direction of the normal through the RGB color channel. For example, the RGB value of each pixel can be used to encode the three components (X, Y, Z) of the normal vector, where the red channel can represent the X component of the normal vector, the green channel represents the Y component, and the blue channel represents the Z component. In this way, the RGB value of each pixel corresponds to a vector in a three-dimensional space, indicating the normal direction of the point.

[0111] The albedo feature refers to the ratio of the light flow scattered in all directions by the fully illuminated part of the object surface to the light flow incident on the object surface. For example, for a face image, the albedo feature refers to the ratio of the light flow scattered in all directions by the fully illuminated part of the face skin to the light flow incident on the face skin. The albedo feature of the face can tell us how the face skin surface reflects light.

[0112] Exemplarily, the trained inverse rendering model can be used to process the face image, that is, the trained inverse rendering model can be used to separate the face and the environment in the face image to obtain the face normal image, the face albedo image and the environment image. In the embodiment of the present application, the face normal image, the face albedo image and the environment image can be understood as the prior information corresponding to the face image, which provides key information about the 3D structure, contour, color, texture, skin, ambient lighting, light and shadow of the face, which is crucial for the subsequent restoration and processing of the face image.

[0113] S104, using the trained face enhancement model to enhance the face image, the face normal image, the face albedo image and the environment image to obtain a face target image.

[0114] Exemplarily, after the face image, face normal image, face albedo image and environment image are spliced ​​together, they are input into the trained face enhancement model for processing to obtain the face target image. Since the accurate prior information corresponding to the face image has been obtained through the inverse rendering model, this prior information provides the accurate contour, rich texture, true color, clear 3D structure and precise ambient lighting conditions of the face, etc. Therefore, the trained face enhancement model can make full use of this prior information in the enhancement process, effectively restore the details of the face image, eliminate the smearing, artifacts, pseudo-textures, etc. in the face image, and improve the clarity and quality of the face image.

[0115] Among them, the face albedo image reflects the skin information of the face, does not produce unnecessary textures, and is not affected by factors such as lighting changes and shadows. When the face enhancement model uses the face albedo image for enhancement processing, it can effectively enhance the details in the face image and improve the quality of the face image. This enhancement not only improves the visual clarity of the face image, but also maintains the authenticity and naturalness of the face. Therefore, the enhanced face image is not only clear and detailed, but also avoids problems such as strong smearing, artifacts and pseudo-textures, giving users a better shooting experience.

[0116] Optionally, the definition of the face target image is greater than the definition of the face image.

[0117] Optionally, the face target image is rich in details and does not have problems such as strong smearing, artifacts, and pseudo textures.

[0118] The image processing method provided in the embodiment of the present application uses a trained inverse rendering model to separate the face and the environment in the face image, and obtains a face normal image, a face albedo image, and an environment image, that is, accurate prior information. The trained face enhancement model makes full use of this prior information during the enhancement process, can effectively restore the details of the face image, eliminate the smearing, artifacts, pseudo-textures, etc. in the face image, and improve the clarity and quality of the face image.

[0119] See also Figure 4 , Figure 4 FIG. 1 is a flow chart of a method for obtaining a face image according to an exemplary embodiment of the present application. The method includes: S1021. In response to a first operation on a first control, obtain an original image.

[0120] Exemplarily, the first operation may be a click operation on a shooting control, and the camera application responds to various types of first operations triggered by the user and starts shooting, that is, the electronic device uses the camera to capture the original image.

[0121] S1022: Determine the clarity of the original image.

[0122] The clarity of the original image refers to the clarity of details in the image, which may include edge sharpness, texture clarity, etc. It should be understood that an original image with high clarity has rich details and obvious edges of human faces.

[0123] In one example, edge information in the original image can be detected by an edge detection algorithm, and the edge information can include the strength and number of edges, and then the clarity of the original image is evaluated by counting the strength and number of edges. The edge detection algorithm can include a Sobel operator algorithm, a Canny operator algorithm, and the like.

[0124] In another example, the original image can be converted to the frequency domain, such as by Fourier transform, and the high-frequency components in the converted original image can be analyzed. It should be understood that the more high-frequency components there are, the higher the clarity of the original image; conversely, the lower the high-frequency components are, the lower the clarity of the original image is.

[0125] In yet another example, image quality assessment indicators (such as sharpness index), edge contrast, etc. may be used to quantify the clarity of the original image.

[0126] In yet another example, a Normalized Root Mean Square Similarit (NASS) value corresponding to the original image may be determined.

[0127] NRSS is a non-reference image quality assessment method used to evaluate the quality of the original image. The NRSS value refers to the quality score of the original image calculated using the NRSS method. When the NRSS value is larger, it means that the quality of the original image is better, clearer and less blurred; when the NRSS value is smaller, it means that the quality of the original image is worse, less clear, and more blurred.

[0128] S1023: Detect that the clarity of the original image is less than a preset clarity threshold, and crop the original image to obtain a face image.

[0129] For example, if the clarity of the original image is detected to be greater than or equal to the preset clarity threshold, it proves that the clarity of the original image is high, the face details in the original image are rich, there are no problems such as strong smearing, artifacts, and false textures, and no enhancement processing is required. If the clarity of the original image is detected to be less than the preset clarity threshold, it proves that the clarity of the original image is low, the face details in the original image may be insufficient, there may be problems such as strong smearing, artifacts, and false textures, and enhancement processing is required.

[0130] It is understandable that the size of the preset clarity threshold can be set and modified as needed, and the embodiment of the present application does not impose any limitation on this.

[0131] Exemplarily, when it is detected that the clarity of the original image is less than a preset clarity threshold, the face region is located (or identified) in the original image. For example, a face detection algorithm (such as a deep learning model, a face feature classifier, etc.) can be used to detect the face region in the original image. Based on the detected face region, a bounding box of the face region is determined; the region within the bounding box is cropped from the original image to obtain a face image.

[0132] See also Figure 5 , Figure 5 FIG. 1 is a schematic diagram of a face image shown in an exemplary embodiment of the present application. Figure 5 The original image shown is detected to have a definition less than a preset definition threshold, and the original image on the left is cropped to obtain the face image on the right. It is worth noting that the oblique stripes in the original image and the face image are both used to represent the environment.

[0133] In this implementation, the original image is effectively screened by definition, so that the determined face images are all face images that need to be enhanced. On the one hand, it is convenient to process only some face images that need to be enhanced in the future, which can improve the image processing efficiency; on the other hand, it avoids enhancing face images that do not need to be enhanced, saving the resources of electronic equipment and reducing power consumption; at the same time, it can also avoid enhancing face images that do not need to be enhanced, which may lead to artifacts and false textures.

[0134] See also Figure 6 , Figure 6 FIG. 1 is a flow chart of another image processing method according to an exemplary embodiment of the present application. The method comprises: S201, obtaining a face image.

[0135] Exemplarily, reference may be made to the aforementioned description on acquiring a facial image, which will not be repeated here.

[0136] It is worth noting that the number of facial images obtained can be one frame or multiple frames, and there is no limitation on this. In the embodiment of the present application, obtaining one frame of facial image is used as an example for explanation.

[0137] S202: Process the face image using the trained inverse rendering model to obtain a face normal image, a face albedo image, and an environment image.

[0138] The trained inverse rendering model is used to separate the face and the environment in the face image to obtain a face normal image, a face albedo image and an environment image. The trained inverse rendering model is obtained by first training the initial inverse rendering network based on the first training set to obtain the inverse rendering model in training, and then training the inverse rendering model in training using the second training set. The first training set includes high-definition sample images, and the second training set includes low-definition sample images, and the low-definition sample images are degraded from the high-definition sample images. The training process of the inverse rendering model will be described in detail later, which is omitted here for the time being.

[0139] The trained inverse rendering model may include a first network, a second network, and a third network. Optionally, the first network may be a vector quantized variational autoencoder (VQVAE), and / or the second network may be a VQVAE.

[0140] Optionally, the third network may be VQVAE or U-Net.

[0141] Among them, VQVAE is a model that combines the Variational Autoencoder (VAE) and Vector Quantization (VQ) technology. VQ technology can map the continuous vector space to a discrete code book to achieve compression and discretization of image data. Specifically, VQ defines a fixed set of vectors as a code book. For each input vector, VQ will find the closest vector in the code book for mapping, thereby reducing the dimension of the image data and improving computational efficiency.

[0142] In the embodiment of the present application, VQ technology is introduced into the inverse rendering model, that is, VQ technology is introduced into each network contained in the inverse rendering model. The benefits of doing so are multifaceted. First, VQ can extract features from facial images and quantize these features into a set of discrete codes, thereby reducing the dimension of the feature space and improving the computational efficiency of the inverse rendering model. Secondly, through vector quantization, VQ helps to enhance the robustness of feature representation, thereby improving the performance of the inverse rendering model. In addition, the application of VQ in the inverse rendering module can improve the reconstruction and enhancement effects of facial images, ensuring that the generated images are more natural and retain more details. Therefore, using the inverse rendering module to process facial images not only improves the efficiency of image processing, but also extracts accurate prior information, which helps the subsequent face enhancement model to effectively use this prior information to enhance the facial image, avoid the generation of pseudo-textures, artifacts, unnatural effects, etc., and greatly improve the quality of the image.

[0143] The following takes the first network as VQVAE, the second network as VQVAE, and the third network as U-Net as an example to explain the inverse rendering model in detail.

[0144] See also Figure 7 , Figure 7 FIG. 1 is a schematic diagram of a trained inverse rendering model structure shown in an exemplary embodiment of the present application. Figure 7 As shown, the inverse rendering model provided in the embodiment of the present application includes three interconnected networks, which are respectively called a first network, a second network and a third network.

[0145] Exemplarily, the first network is connected to the second network and the third network respectively, and the second network is connected to the third network. The first network includes an input terminal and an output terminal, the second network includes a first input terminal, a second input terminal and an output terminal, and the third network includes a first input terminal, a second input terminal, a third input terminal and an output terminal.

[0146] Exemplarily, the input end of the first network, the first input end of the second network and the first input end of the third network all serve as the input end of the inverse rendering model; the output end of the first network and the second input end of the second network are respectively connected to the second input end of the third network, and the output end of the second network is connected to the third input end of the third network; the output end of the third network serves as the output end of the inverse rendering model, and the output end of the first network and the output end of the second network also serve as the output end of the inverse rendering model.

[0147] based on Figure 7 The structure of the inverse rendering model shown is as follows: a face image is inputted at the input end of the first network, the first network processes the face image, and outputs a face normal image through the output end of the first network; a face image is inputted at the first input end of the second network, a face normal image is inputted at the second input end, the second network processes the face image and the face normal image, and outputs a face albedo image through the output end of the second network; a face image is inputted at the first input end of the third network, a face normal image is inputted at the second input end, and a face albedo image is inputted at the third input end, the third network processes the face image, the face normal image, and the face albedo image, and outputs an environment image through the output end of the third network.

[0148] Optionally, when a face image is input into the inverse rendering model, the input end of the first network, the first input end of the second network and the first input end of the third network respectively receive the face image, and the first network is used to determine the face normal image corresponding to the face image, so that the output end of the first network can provide the face normal image to the second input end of the second network and the second input end of the third network respectively; the second network is used to determine the face albedo image corresponding to the face image based on the face image and the face normal image, so that the output end of the second network can provide the face albedo image to the third input end of the third network; the third network is used to determine the environment image corresponding to the face image based on the face image, the face normal image and the face albedo image; thereafter, the face normal image determined by the first network, the face albedo image determined by the second network, and the environment image determined by the third network will all be output as outputs of the inverse rendering model.

[0149] It should be understood that the above is only a schematic diagram of the structure of an inverse rendering model, and the embodiments of the present application do not impose any limitation on this.

[0150] See also Figure 8 , Figure 8 FIG. 1 is a schematic diagram of a face image, a face normal image, a face albedo image, and an environment image shown in an exemplary embodiment of the present application. Exemplarily, the inverse rendering model separates the face and the environment in the face image to obtain the following: Figure 8The face normal image, face albedo image and environment image shown. It is worth noting that the dotted lines in the face normal image, the thick oblique lines in the face albedo image, and the oblique lines in the environment image are only used to distinguish the face normal image, the face albedo image and the environment image, and do not constitute a limitation on the face normal image, the face albedo image and the environment image actually output by the inverse rendering model.

[0151] It should be understood that, under normal circumstances, the background in the face normal image actually output by the inverse rendering model is gray, and the face part has color changes to represent the direction of the normal of the face surface, where red represents the component of the normal in the X-axis direction, green represents the component of the normal in the Y-axis direction, and blue represents the component of the normal in the Z-axis direction.

[0152] The background of the face albedo image actually output by the inverse rendering model is black, the face part is clearly visible, and it accurately reflects the facial skin information. There is no unnecessary texture and it is not affected by factors such as lighting changes and shadows.

[0153] The environment image actually output by the inverse rendering model shows the environment and lighting conditions of the subject at the time of shooting.

[0154] Combine the following Figure 8 The images shown and the structure of the inverse rendering model are described in detail for the processing process of the inverse rendering model.

[0155] See also Fig. 9 , Fig. 9 FIG. 1 is a schematic diagram of a reverse rendering model processing flow diagram of an exemplary embodiment of the present application. The process includes: S2021. Use the first network to process the face image to obtain a face normal image.

[0156] In one example, a first latent vector in a face image is extracted using a first network; a vector close to the first latent vector is searched in a first discrete vector set; and a face normal image is generated based on the vector close to the first latent vector.

[0157] like Fig. 9 As shown, the first network may include a first encoder, a first vector codebook module, and a first decoder. The first encoder is used to map the input face image to a continuous latent space, which can be understood as the first network is used to extract latent vectors in the face image. For example, the first encoder is used to extract features (such as edge features, contour features, geometric features, etc.) in the face image, and these features are encoded into a continuous latent vector, namely, the first latent vector. Among them, the first latent vector may include 3D structural information and geometric information corresponding to the face.

[0158] The first vector codebook module is obtained by extracting the latent vectors in the face normal image corresponding to the high-definition sample image from the inverse rendering model under training during the training process of the inverse rendering model. The first vector codebook module may include a first discrete vector set. In other words, the first discrete vector set includes latent vectors in the face normal image corresponding to multiple high-definition sample images. The continuous first latent vector extracted by the first encoder can be quantized into discrete latent vectors through the first vector codebook module.

[0159] Exemplarily, according to the first latent vector extracted by the first encoder, a vector close to the first latent vector is searched in the first discrete vector set, or a vector closest to the first latent vector is searched in the first discrete vector set. For example, the distance (such as Euclidean distance) between the first latent vector and each vector in the first discrete vector set is calculated, and the vector with the smallest distance is determined as the vector close to the first latent vector.

[0160] The first decoder is used to map the quantized latent vector (i.e., the vector close to the first latent vector found in the first discrete vector set) back to the face image space. The purpose of this process is to use the quantized latent vector to reconstruct a face image as close to the original as possible. Exemplarily, a vector close to the first latent vector is used as the input of the first decoder. The first decoder maps the vector close to the first latent vector back to the face image space. During the mapping process, the first decoder generates a face normal image.

[0161] See also Fig.10 , Fig.10 FIG. 1 is a schematic diagram of a first network structure shown in an exemplary embodiment of the present application. Fig.10 As shown, the first network includes a first encoder, a first vector codebook module and a first decoder. The first encoder is used to extract the first latent vector in the face image, and the first latent vector is input into the first vector codebook module. The first vector codebook module includes a first discrete vector set, also referred to as a first codebook in the embodiment of the present application. The first discrete vector set is composed of latent vectors in the face normal image corresponding to multiple high-definition sample images. Fig.10 It is represented by rectangular boxes of different forms and labeled with 0, 1...N.

[0162] The first vector codebook module may further include a first matching module (Matching Block), which compares the first potential vector with each vector in the first discrete vector set through the Matching Block, and determines the vector closest to the first potential vector in the first discrete vector set, such as Fig.10The first vector codebook module obtains quantized latent vectors based on these vectors and outputs the quantized latent vectors to the first decoder. Afterwards, the first decoder maps the quantized latent vectors back to the face image space, and generates a face normal image during the mapping process.

[0163] In this implementation, vector quantization through VQVAE can effectively map the information in the continuous latent space into a discrete space, which helps to improve the computational efficiency and generalization ability of the first network. In this way, the first network can extract rich 3D structural information and geometric information from the input face image, laying the foundation for the subsequent enhancement of the face image, that is, it helps to more accurately restore the 3D structural features and geometric features of the face when the face image is enhanced.

[0164] Optionally, in another example, the first network may be a CNN, which extracts edge features, contour features, geometric features, etc. in the face image through CNN; estimates depth information of each point on the face surface through CNN; calculates the normal vector of each point on the face surface based on the estimated depth information; converts the calculated normal vector into RGB channel values ​​for visualization in the image; and combines these RGB channel values ​​into a face normal image through CNN.

[0165] S2022. Use the second network to process the face image and the face normal image to obtain a face albedo image.

[0166] In one example, a face image and a face normal image are spliced ​​to obtain a first spliced ​​image; a second latent vector in the first spliced ​​image is extracted using a second network; a vector close to the second latent vector is searched in a second discrete vector set; and a face albedo image is generated based on the vector close to the second latent vector.

[0167] like Fig. 9 As shown, the second network may include a second encoder, a second vector codebook module, and a second decoder. The face image and the face normal image are used as inputs of the second network. Specifically, the face image and the face normal image are spliced ​​to obtain a first spliced ​​image and then input. For example, the RGB channels of the face image are spliced ​​with the RGB channels of the face normal image to obtain a new six-channel image, namely, the first spliced ​​image.

[0168] Afterwards, the first spliced ​​image is input into the second encoder, and the second encoder is used to map the input first spliced ​​image into a continuous latent space. It can be understood that the second network is used to extract the latent vector in the first spliced ​​image. For example, the second encoder is used to extract features (such as texture features, skin features, etc.) in the first spliced ​​image, and encode these features into a continuous latent vector, namely, the second latent vector. The second latent vector may include texture information and skin information corresponding to the face.

[0169] The second vector codebook module is obtained by extracting the latent vectors in the face albedo image corresponding to the high-definition sample image from the inverse rendering model in training during the process of training the inverse rendering model. The second vector codebook module may include a second discrete vector set. In other words, the second discrete vector set includes latent vectors in the face albedo image corresponding to multiple high-definition sample images. The continuous second latent vector extracted by the second encoder can be quantized into discrete latent vectors through the second vector codebook module.

[0170] Exemplarily, according to the second latent vector extracted by the second encoder, a vector close to the second latent vector is searched in the second discrete vector set, or a vector closest to the second latent vector is searched in the second discrete vector set. For example, the distance (such as Euclidean distance) between the second latent vector and each vector in the second discrete vector set is calculated, and the vector with the smallest distance is determined as the vector close to the second latent vector.

[0171] The second decoder is used to map the quantized vector (i.e., the vector close to the second latent vector found in the second discrete vector set) back to the face image space. Exemplarily, the vector close to the second latent vector is used as the input of the second decoder, and the second decoder maps the vector close to the second latent vector back to the face image space. During the mapping process, the second decoder generates a face albedo image.

[0172] See also Fig.11 , Fig.11 is a schematic diagram of a second network structure shown in an exemplary embodiment of the present application. Fig.11 As shown, the second network includes a second encoder, a second vector codebook module and a second decoder. The second encoder is used to extract the second latent vector in the first spliced ​​image, and the second latent vector is input into the second vector codebook module. The second vector codebook module includes a second discrete vector set, also referred to as a second codebook in the embodiment of the present application. The second discrete vector set is composed of latent vectors in the face albedo image corresponding to multiple high-definition sample images. Fig.11 It is represented by rectangular boxes of different forms and labeled with 0, 1...N.

[0173] The second vector codebook module may further include a second matching module (Matching Block), which compares the second potential vector with each vector in the second discrete vector set through the second Matching Block, and determines the vector closest to the second potential vector in the second discrete vector set, such as Fig.11 The vectors 4, 8, 7, 3, 5, 4, 2, 0, 5, etc. in the nine-square grid shown in the figure are used; the second vector codebook module obtains the quantized latent vector based on these vectors, and outputs the quantized latent vector to the second decoder. For example, the vector closest to the second latent vector is used to replace the original second latent vector to obtain the quantized latent vector. Afterwards, the second decoder maps the quantized latent vector back to the face image space, and generates a face albedo image during the mapping process.

[0174] In this implementation, vector quantization through VQVAE can effectively map the information in the continuous latent space into a discrete space, which helps to improve the computational efficiency and generalization ability of the second network. In this way, the second network can extract accurate texture information and skin information from the input face image and face normal image, laying the foundation for the subsequent enhancement of the face image, that is, it helps to more accurately restore the texture features and skin features of the face when the face image is enhanced, improve the details and realism of the face image, and avoid the generation of pseudo-textures, artifacts, unnatural effects, etc.

[0175] Optionally, in another example, the second network can be U-Net, which gradually extracts texture features, skin features, etc. in the first spliced ​​image through multiple convolutional layers; in the encoder part of U-Net, downsampling is performed through the pooling layer to reduce the size of the first spliced ​​image, increase the receptive field, and obtain more local information of the face; in the decoder part of U-Net, the spatial dimension of the image is gradually restored through upsampling, and in each upsampling step, the features in the encoder are combined with the features in the decoder through jump connections, so that more detail information can be retained; in the final stage of the decoder, the multi-channel feature map output by the decoder is converted into a final face albedo image.

[0176] S2023. Use a third network to process the face image, the face normal image, and the face albedo image to obtain an environment image.

[0177] In one example, the third network may be a U-Net. The face image, the face normal image, and the face albedo image are used as inputs of the third network. Specifically, the face image, the face normal image, and the face albedo image are spliced ​​to obtain a target spliced ​​image. For example, the RGB channels of the face image, the RGB channels of the face normal image, and the RGB channels of the face albedo image are spliced ​​to obtain a new nine-channel image, namely, the target spliced ​​image.

[0178] U-Net gradually extracts the ambient lighting features, light and shadow features, etc. in the target spliced ​​image through multiple convolutional layers. In the encoder part of U-Net, downsampling is performed through the pooling layer to reduce the size of the target spliced ​​image, increase the receptive field, and obtain more ambient lighting information and light and shadow information; in the decoder part of U-Net, the spatial dimension of the image is gradually restored through upsampling, and in each upsampling step, the features in the encoder are combined with the features in the decoder through jump connections, so that more ambient lighting information and light and shadow information can be retained; in the final stage of the decoder, the multi-channel feature map output by the decoder is converted into the final environmental image.

[0179] In another example, the third network may be VQVAE. The process of processing the face image, the face normal image, and the face albedo image by VQVAE to obtain the environment image may refer to the description in the above step S2022, which will not be repeated here.

[0180] S203, using the trained face enhancement model to enhance the face image, the face normal image, the face albedo image and the environment image to obtain a face target image.

[0181] The trained face enhancement model is obtained by first training the initial face enhancement network based on the third training set to obtain the face enhancement model in training, and then training the face enhancement model in training using the fourth training set. The third training set includes high-definition sample images, and the fourth training set includes low-definition sample images, as well as normal sample images, albedo sample images, and environmental sample images corresponding to the low-definition sample images; the normal sample images and albedo sample images are obtained by processing the low-definition sample images with the trained inverse rendering model; and the environmental sample images are simulated by the trained light and shadow enhancement model. The training process of the inverse rendering model will be described in detail later, which is omitted here for the time being.

[0182] The face enhancement model can be VQVAE or U-Net. In the embodiment of the present application, the face enhancement model is VQVAE as an example for illustration.

[0183] It should be understood that the benefits of introducing VQ technology into the face enhancement model are multifaceted. First, VQ can extract features from face images, face normal images, face albedo images, and environmental images, and quantize these features into a set of discrete codes, thereby reducing the dimension of the feature space and improving the computational efficiency of the face enhancement model. Secondly, through vector quantization, VQ helps to enhance the robustness of feature representation, thereby improving the performance of the face enhancement model. In addition, the application of VQ in the face enhancement model can improve the reconstruction and enhancement effects of face images, ensuring that the enhanced images are more natural and retain more details. Therefore, using the face enhancement model to enhance the face image not only improves the efficiency of image processing, but also effectively utilizes the accurate prior information extracted by the inverse rendering model to avoid the generation of pseudo-textures, artifacts, unnatural effects, etc., greatly improving the quality of the image.

[0184] See also Fig.12 , Fig.12 FIG. 1 is a schematic diagram of a face enhancement model processing flow diagram of an exemplary embodiment of the present application. The flow includes: S2031 , stitching the face image, the face normal image, the face albedo image, and the environment image to obtain a second stitched image.

[0185] Exemplarily, a face image, a face normal image, a face albedo image, and an environment image are used as inputs of a face enhancement model. Specifically, the face image, the face normal image, the face albedo image, and the environment image are spliced ​​to obtain a second spliced ​​image and then input. For example, the RGB channels of the face image, the RGB channels of the face normal image, the RGB channels of the face albedo image, and the RGB channels of the environment image are spliced ​​to obtain a new twelve-channel image, i.e., the second spliced ​​image. Afterwards, the second spliced ​​image is input into the face enhancement model for processing.

[0186] S2032. Extract a third latent vector in the second spliced ​​image using the trained face enhancement model.

[0187] Exemplarily, the trained face enhancement model may include a third encoder, a third vector codebook module, and a third decoder. The second spliced ​​image is input into the third encoder, and the third encoder is used to map the input second spliced ​​image to a continuous latent space. It can be understood that the third network is used to extract the latent vector in the second spliced ​​image. For example, the third encoder is used to extract features (such as edge features, contour features, geometric features, texture features, skin features, ambient lighting features, light and shadow features, etc.) in the second spliced ​​image, and these features are encoded into a continuous latent vector, that is, a third latent vector. Among them, the third latent vector may include 3D structure information, geometric information, texture information, skin information, ambient lighting information, light and shadow information, etc. corresponding to the face.

[0188] S2033. Search for a vector close to the third potential vector in the third discrete vector set.

[0189] The third vector codebook module is obtained by extracting the latent vector corresponding to the face in the high-definition sample image from the face enhancement model in training during the training process of the face enhancement model. The third vector codebook module may include a third discrete vector set. In other words, the third discrete vector set includes the latent vectors corresponding to the faces in multiple high-definition sample images. The continuous third latent vector extracted by the third encoder can be quantized into a discrete latent vector through the third vector codebook module.

[0190] Exemplarily, according to the third latent vector extracted by the third encoder, a vector close to the third latent vector is searched in the third discrete vector set, or a vector closest to the third latent vector is searched in the third discrete vector set. For example, the distance (such as Euclidean distance) between the third latent vector and each vector in the third discrete vector set is calculated, and the vector with the smallest distance is determined as the vector close to the third latent vector.

[0191] S2034. Generate a target face image based on a vector close to the third latent vector.

[0192] The third decoder is used to map the quantized vector (i.e., the vector close to the third latent vector found in the third discrete vector set) back to the face image space. Exemplarily, the vector close to the third latent vector is used as the input of the third decoder, and the third decoder maps the vector close to the third latent vector back to the face image space. During the mapping process, the third decoder generates a face target image.

[0193] See also Fig.13 , Fig.13 FIG. 1 is a schematic diagram of the structure of a face enhancement model shown in an exemplary embodiment of the present application. Fig.13As shown, the third network includes a third encoder, a third vector codebook module and a third decoder. The third encoder is used to extract the third latent vector in the second spliced ​​image, and the third latent vector is input into the third vector codebook module. The third vector codebook module includes a third discrete vector set, which is also called a third codebook in the embodiment of the present application. The third discrete vector set is composed of latent vectors corresponding to faces in multiple high-definition sample images. Fig.13 It is represented by rectangular boxes of different forms and labeled with 0, 1...N.

[0194] The third vector codebook module may further include a third matching module (Matching Block), which compares the third potential vector with each vector in the third discrete vector set through the third Matching Block, and determines the vector closest to the third potential vector in the third discrete vector set, such as Fig.13 The third vector codebook module obtains quantized latent vectors based on these vectors and outputs the quantized latent vectors to the third decoder. Afterwards, the third decoder maps the quantized latent vectors back to the face image space, and generates a face target image during the mapping process.

[0195] In this implementation, the face enhancement model can obtain accurate 3D structure information, geometry information, texture information, skin information, ambient lighting information, light and shadow information, etc. from prior information. Therefore, even in the case of poor quality of face image and complex facial lighting and shadow, it can correctly guide the restoration of the 3D structure, geometry, texture, skin, ambient lighting, light and shadow and other features of the face image based on this information, while effectively avoiding the generation of pseudo-texture and artifacts, thereby improving the image quality and the robustness of the face enhancement model, making the face enhancement model more widely applicable.

[0196] S204: splicing the original image and the target face image to obtain a high-definition image corresponding to the original image.

[0197] Exemplarily, the original image and the face target image can be spliced ​​using a pre-prepared mask, which is a binary image used to indicate which areas in the original image should be replaced by the face target image. It should be understood that in the mask, the white part usually represents the area of ​​the face target image, and the black part represents the area of ​​the original image that should be retained. The face target image is processed using the mask, for example, the mask can be used to perform a bitwise AND operation on the face target image to ensure that the pixels in the white part of the mask are retained.

[0198] The processed face target image is fused with the original image, for example, by a bitwise OR operation. The original image is used as the basis, and the corresponding parts of the face target image are merged through a mask. After fusion, the resulting image is the high-definition image corresponding to the original image.

[0199] Optionally, after the processed face target image is fused with the original image, some detail adjustments may be made to the splicing area, such as color balance, edge balance, etc., to ensure a natural splicing effect.

[0200] In this implementation, the face image is enhanced specifically. On the one hand, the details of the face area can be effectively restored, and the smearing, artifacts, and pseudo textures existing in the face image can be eliminated, ensuring that the image quality of the face area is significantly improved; on the other hand, the face image is enhanced separately, and then the original image and the face target image are spliced ​​to obtain a high-definition image, which can effectively improve the image processing efficiency and reduce the resource consumption of electronic devices. In addition, users are usually more concerned about the faces in the captured images. The image processing method provided by this application can significantly improve the image quality of the face area while improving the image processing speed, which can greatly improve the user's favorability.

[0201] Optionally, in one possible implementation, the image processing method provided in the embodiment of the present application can also use a trained light and shadow enhancement model to determine a fill-light environment image corresponding to the environment image; use a trained face enhancement model to enhance the face image, face normal image, face albedo image and fill-light environment image to obtain a face target image.

[0202] Among them, the trained light and shadow enhancement model can be trained based on the U-Net model. The light and shadow enhancement model can simulate the light and shadow conditions of the human face in various lighting environments. Therefore, the light and shadow enhancement model can be used to fill in the environmental image and generate a fill-light environmental image. For example, the original image is subjected to portrait posture estimation to determine the portrait posture. Among them, the portrait posture refers to the angle information of the face orientation, which can be represented by a rotation matrix, a rotation vector, a quaternion, an Euler angle (Yaw, Roll, Pitch), etc. Afterwards, the fill-light position is determined in the environmental image in combination with the portrait posture and the light source position in the environment shown in the environmental image; the environmental image is filled in at the fill-light position to obtain a fill-light environmental image.

[0203] Optionally, in a possible implementation manner, a new environment image may be used to replace the original environment image, and the new environment image may be determined as the fill-light environment image.

[0204] Afterwards, the face image, the face normal image, the face albedo image, and the fill-light environment image are stitched together to obtain a third stitched image; a fourth latent vector in the third stitched image is extracted using the trained face enhancement model; a vector close to the fourth latent vector is searched in the third discrete vector set; and a face target image is generated based on the vector close to the fourth latent vector. The specific implementation process can refer to the description in the aforementioned steps S2031 to S2034, which will not be repeated here.

[0205] In this implementation, the light and shadow enhancement model is used to intelligently fill in the ambient image, and then the face enhancement model is used to fine-tune the face image based on the filled-in ambient image. This processing not only enhances the facial details, but also achieves a comprehensive fill-in effect for the face and the surrounding environment, ensuring the balance of the face and ambient lighting, which is conducive to generating a face target image with higher clarity and stronger contrast.

[0206] Optionally, in a possible implementation, after determining the fill-light environment image, the trained light and shadow enhancement model can be used to render the face normal image, the face albedo image, and the fill-light environment image to generate a fill-light face image. It should be understood that based on the fill-light situation in the fill-light environment image, the normal in the face normal image, and the face albedo image, the corresponding shadow generated by the face in the fill-light environment can be calculated, and then displayed in the rendered fill-light face image. For example, if the fill-light is on the right side, the brightness of the right side of the face is increased, and the left side is relatively dark, then the shadow cast by the left side of the face in the environment can be displayed in the fill-light face image.

[0207] It is worth noting that the rendering in the embodiment of the present application refers to the process of converting a three-dimensional light energy transfer process into a two-dimensional image.

[0208] Afterwards, the trained face enhancement model is used to enhance the fill-light face image, face normal image, face albedo image and fill-light environment image to obtain the face target image.

[0209] In this implementation, the trained light and shadow enhancement model can achieve fill light, and no additional fill light device is required for the electronic device, nor does the user need to manually select the fill light area during the shooting process, which improves the user experience. And whether it is during the shooting process or after the shooting is completed, the face image can be filled with light by this method, which has high flexibility. In addition, compared with the traditional external light source or manual selection of fill light area method, the present application uses the light and shadow enhancement model to render the face normal image, the face albedo image and the fill light environment image to generate the fill light face image, ensuring that the fill light face image is more in line with the lighting conditions of the actual environment, making the fill light effect more natural and realistic.

[0210] It is worth noting that before using the inverse rendering model, it is necessary to train and generate the inverse rendering model. The training process of the inverse rendering model provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0211] See also Fig.14 , Fig.14 It is a schematic diagram of the process of training an inverse rendering model shown in an exemplary embodiment of the present application, which is as follows.

[0212] S301. Obtain a first training set.

[0213] The first training set includes high-definition sample images, which contain human faces. The high-definition sample images have high clarity and rich details, and do not have quality problems such as smearing, artifacts, and pseudo textures.

[0214] High-definition sample images can come from a variety of sources, such as pre-collecting high-definition images containing human faces through the camera of an electronic device, downloading high-definition images containing human faces from the Internet, or receiving high-definition images containing human faces transmitted by other electronic devices, etc.

[0215] It is worth noting that the number of high-definition sample images will affect the training effect and training speed of the inverse rendering model. In the embodiment of the present application, the number of high-definition sample images can be reasonably set to achieve the best balance between accuracy and training speed. The specific number is not limited.

[0216] S302: Train the initial inverse rendering network using the first training set to obtain an inverse rendering model in training.

[0217] Exemplarily, the high-definition sample images in the first training set are cropped to obtain high-definition face images, and the high-definition face images are input into the initial inverse rendering network. The initial inverse rendering network separates the face and the environment in the high-definition face image to obtain a high-definition face normal image, a high-definition face albedo image, and a high-definition environment image.

[0218] The structure of the initial inverse rendering network is consistent with the structure of the trained inverse rendering model. Therefore, the process of the initial inverse rendering network processing high-definition face images can refer to the process of processing face images using the trained inverse rendering model in the previous article, which will not be repeated here.

[0219] In the process of training the initial inverse rendering network using the first training set, the initial inverse rendering network can learn how to accurately extract the face normal image, face albedo image and environment image from the high-definition sample image. In addition, by first training the initial inverse rendering network with high-definition sample images, the vector codebook module in the initial inverse rendering network can learn the prior information corresponding to the high-definition sample image, such as the latent vector corresponding to the high-definition face normal image, the latent vector corresponding to the high-definition face albedo image and the latent vector corresponding to the high-definition environment image. Afterwards, the vector codebook module in the initial inverse rendering network is fixed to obtain the inverse rendering model in training.

[0220] Optionally, the training data can also be enriched, that is, the lighting scenes of the high-definition sample images can be enriched. Since the lighting scenes of the high-definition sample images are usually relatively simple, the light and shadow on the face are not complex enough, which will make the trained inverse rendering model unable to accurately handle various light and shadow changes when facing real complex lighting scenes, thereby generating artifacts, pseudo-textures and other problems. Therefore, in the embodiment of the present application, shadow enhancement technology can also be introduced in the process of training the inverse rendering model. For example, the color, brightness, etc. of specific areas of the face can be adjusted to generate more high-definition sample images with complex light and shadow effects.

[0221] Optionally, the light and shadow enhancement model can also be used to enrich the high-definition sample images. For example, a large number of various lighting environment maps are collected in advance from multiple channels; the original high-definition face image is processed by the inverse rendering model in training to obtain a high-definition face normal image and a high-definition face albedo image; the high-definition face normal image, high-definition face albedo image, and various lighting environment maps are rendered by the light and shadow enhancement model to simulate the light and shadow conditions of the face in various lighting environments, thereby generating high-definition face images in various lighting environments. Afterwards, the inverse rendering model in training is continued to be trained using high-definition face images in various lighting environments, so that the inverse rendering model in training can learn how to accurately handle various light and shadow changes when dealing with complex lighting scenes and avoid the generation of artifacts and pseudo-textures, thereby improving the robustness and adaptability of the inverse rendering model.

[0222] See also Fig.15 , Fig.15 1 is a schematic diagram of a light and shadow enhancement model enriching a high-definition sample image shown in an exemplary embodiment of the present application. Exemplarily, by processing the original high-definition face image through the inverse rendering model in training, a high-definition face normal image and a high-definition face albedo image can be obtained; select any lighting environment map (such as the grid represented in 15), and render the high-definition face normal image, the high-definition face albedo image, and any lighting environment map through the light and shadow enhancement model to generate the following Fig.15 The high-definition face image shown in this lighting environment.

[0223] In this implementation, the light and shadow enhancement model is used to enrich the high-definition sample images, which greatly ensures the light and shadow richness of the training data and increases the light and shadow complexity of the training data. This helps to improve the robustness and adaptability of the inverse rendering model, making it easier for the subsequently trained inverse rendering model to accurately handle various light and shadow changes when facing real and complex lighting scenes, avoiding problems such as artifacts and false textures, and effectively improving image quality.

[0224] S303: Obtain a second training set.

[0225] The second training set includes low-definition sample images, which are obtained by degrading high-definition sample images. Exemplarily, the generation process of these low-definition sample images can be implemented by a pre-trained degradation model, and the degradation model can simulate various degradation effects on the high-definition sample images. For example, the high-definition sample images are processed using the pre-trained degradation model to add at least one degradation effect of noise, blur, smearing, artifacts, pseudo textures, etc. to the high-definition sample images to obtain low-definition sample images.

[0226] S304: Use the second training set to train the inverse rendering model in training to obtain a trained inverse rendering model.

[0227] Exemplarily, the low-resolution sample images in the second training set are cropped to obtain low-resolution face images, which are input into the inverse rendering model in training. The inverse rendering model in training separates the face and the environment in the low-resolution face images to obtain a real face normal image, a real face albedo image and a real environment image.

[0228] The structure of the inverse rendering model being trained is consistent with the structure of the trained inverse rendering model. Therefore, the process of processing low-definition face images by the inverse rendering model being trained can refer to the process of processing face images using the trained inverse rendering model in the previous article, which will not be repeated here.

[0229] Since the vector codebook module of the inverse rendering model in training stores the prior information of the high-definition sample images, the pre-stored prior information of the high-definition sample images can be effectively used to assist the inverse rendering model in training to restore and output high-quality real face normal images, real face albedo images, and real environment images. In this process, the inverse rendering model in training learns how to output high-quality images based on the prior information of the high-definition sample images when facing low-definition sample images. When the inverse rendering model in training has mastered this ability, the vector codebook module at this time is fixed to obtain the trained inverse rendering model.

[0230] Generally speaking, human faces have strong a priori characteristics. For example, the facial features are very regular. The inverse rendering model in training learns these regularities in high-definition face images (that is, the prior information mentioned above). When processing low-definition face images, it can infer what the blurred areas in the low-definition face images should look like based on the learned regularities, thereby restoring and outputting high-quality images.

[0231] In this implementation, the training of the inverse rendering model is divided into two stages. In the first stage, the initial inverse rendering network is trained with high-definition sample images, so that the vector codebook module can capture and save the prior information of the high-definition sample images; in the second stage, further training is performed with low-definition sample images, so that the inverse rendering model in training can learn how to accurately use the prior information to restore and output high-quality images. Through these two stages of training, the trained inverse rendering model has strong robustness, so that when facing face images in various complex light and shadow scenes, it can also accurately extract and use the prior information to generate high-quality images.

[0232] It is worth noting that before using the face enhancement model, it is necessary to train and generate the face enhancement model. The training process of the face enhancement model provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0233] See also Fig.16 , Fig.16 It is a schematic diagram of the process of training a face enhancement model shown in an exemplary embodiment of the present application, and the details are as follows.

[0234] S401. Obtain a third training set.

[0235] The third training set may include high-definition sample images. It is worth noting that the high-definition sample images included in the third training set may be the same as or different from the high-definition sample images included in the first training set. In different situations, multiple high-definition images containing human faces may be collected through various methods to obtain the third training set.

[0236] S402: Using the third training set to train the initial face enhancement network to obtain a face enhancement model in training.

[0237] Exemplarily, the high-definition sample images in the third training set are cropped to obtain a high-definition face image; the high-definition face image is input into the trained inverse rendering model, and the inverse rendering model separates the face and the environment in the high-definition face image to obtain a high-definition face normal image, a high-definition face albedo image, and a high-definition environment image. The high-definition face image, the high-definition face normal image, the high-definition face albedo image, and the high-definition environment image are processed using the initial face enhancement network to obtain a high-definition face target image.

[0238] The structure of the initial face enhancement network is consistent with the structure of the trained face enhancement model. Therefore, the specific processing process of the initial face enhancement network can refer to the specific processing process of the trained face enhancement model mentioned above, which will not be repeated here.

[0239] In the process of training the initial face enhancement network using the third training set, the vector codebook module in the initial face enhancement network can learn how to quantize the vector so as to accurately use the prior information to restore and output high-quality face images. When the vector codebook module in the initial face enhancement network has this ability, the vector codebook module in the initial face enhancement network is fixed to obtain the face enhancement model in training.

[0240] In this implementation, the training of the face enhancement model is divided into two stages. In the first stage, the initial face enhancement network is trained through high-definition sample images, so that the vector codebook module in the initial face enhancement network can learn how to quantize the vector so that it can accurately use the prior information to restore and output high-quality face images. In the second stage, further training is performed through low-definition sample images, so that the face enhancement model in training can learn how to accurately use the prior information to restore and output high-quality images. Through these two stages of training, the trained face enhancement model has strong robustness, so that when faced with face images in various complex light and shadow scenes, it can also accurately use the prior information to generate high-quality images.

[0241] Optionally, the lighting scenes of the HD sample images in the third training set can also be enriched. For example, the color and brightness of specific areas of the face can be adjusted by using shadow enhancement technology to generate more HD sample images with complex light and shadow effects; for another example, the HD sample images can be enriched by using a light and shadow enhancement model.

[0242] In this implementation, the light and shadow enhancement model is used to enrich the high-definition sample images, which greatly ensures the light and shadow richness of the training data and increases the light and shadow complexity of the training data. This helps to improve the robustness and usability of the face enhancement model, so that the subsequently trained face enhancement model can accurately handle various light and shadow changes when faced with images of poor quality and complex face light and shadow, avoid problems such as artifacts and false textures, and effectively improve image quality.

[0243] S403: Obtain a fourth training set.

[0244] The fourth training set includes low-definition sample images. It is worth noting that the low-definition sample images included in the fourth training set may be the same as or different from the low-definition sample images included in the second training set.

[0245] The fourth training set may also include normal sample images, albedo sample images, and environment sample images corresponding to the low-definition sample images. The normal sample images and albedo sample images are obtained by processing the low-definition sample images with a trained inverse rendering model; the environment sample images can be obtained by processing the low-definition sample images with a trained inverse rendering model, or by simulating a trained light and shadow enhancement model. When the environment sample images are simulated with a trained light and shadow enhancement model, the light and shadow complexity of the training data of the face enhancement model can be enriched, which helps to improve the robustness and usability of the face enhancement model.

[0246] S404: Use the fourth training set to train the face enhancement model in training to obtain a trained face enhancement model.

[0247] Exemplarily, the low-definition sample images in the fourth training set are cropped to obtain low-definition face images; the low-definition face images, normal sample images, albedo sample images, and environment sample images are spliced ​​to obtain a fourth spliced ​​image; the fifth latent vector in the fourth spliced ​​image is extracted using the trained face enhancement model; a vector close to the fifth latent vector is searched in the vector codebook module of the face enhancement model being trained; and a high-definition face image is generated based on the vector close to the fifth latent vector. In the embodiment of the present application, the high-definition face image output by the face enhancement model being trained is referred to as a predicted image, or as a reconstructed image.

[0248] The structure of the face enhancement model in training is consistent with the structure of the trained face enhancement model. Therefore, the specific processing process of the face enhancement model in training can refer to the specific processing process of the trained face enhancement model in the previous article, which will not be repeated here.

[0249] When the target loss value corresponding to the face enhancement model in training converges, the trained face enhancement model is obtained. Exemplarily, in the process of training the face enhancement model, the target loss value is calculated according to a preset loss function; according to the target loss value, the face enhancement model in training is further trained in a back propagation manner to obtain the trained face enhancement model.

[0250] Exemplarily, the target loss value is used to represent the loss between a high-definition sample image and a predicted image output by a face enhancement model in training, and the loss consists of three parts: the first part is the L1 loss, the second part is the perceptual loss, and the third part is the adversarial loss.

[0251] Among them, L1 loss is used to represent the loss between the high-definition sample image and the predicted image (or reconstructed image); perceptual loss is used to represent the feature loss between the high-definition sample image and the predicted image (or reconstructed image); adversarial loss is used to represent the loss between the real image and the image generated by the generator in the face enhancement model.

[0252] The default loss function is as follows: , (1) , (2) In the above formulas (1) and (2), represents a high-definition sample image, represents a low-resolution sample image, Indicates that the face enhancement network The reconstructed image generated, represents L1 loss, represents the perceived loss, Represents resistance to loss, , , Represents the weights corresponding to the three losses. Among them, face enhancement network Refers to the network used by the face enhancement model in training.

[0253] Face Enhancement Network The implementation is as follows: , (3) In the above formula (3), represents the encoder of the face enhancement network, Represents the vector codebook module of the face enhancement network, represents the decoder of the face enhancement network, Represents prior information.

[0254] , as well as The calculation formula is as follows:

[0255]

[0256] , (4) In the above formula (4), represents the discriminator network.

[0257] For example, the calculation , , The target loss value is obtained by summing the values ​​of the face enhancement model in training and adjusting the face enhancement model in training in a back-propagation manner according to the target loss value. , , When the target loss value converges, the network parameters of the face enhancement network are fixed to obtain the trained face enhancement model.

[0258] For example, in the process of training the face enhancement model, the target loss value will gradually decrease and stabilize. In other words, as the number of training times increases, the target loss value no longer decreases significantly, but fluctuates within a smaller range, which indicates that the face enhancement network in training has learned how to accurately use prior information to restore and output high-quality face images, which means that the face enhancement model has been trained.

[0259] In an embodiment of the present application, when it is detected that the target loss value is greater than a preset loss threshold, the training process of the face enhancement model continues; when it is detected that the target loss value is less than or equal to the preset loss threshold, the training is stopped, and the network parameters of the face enhancement network are saved to obtain a trained face enhancement model.

[0260] In this implementation, the back-propagation algorithm is used to adjust the model parameters, which helps the model converge to the optimal solution faster and improves the training speed of the face enhancement model. The losses of the three parts of the face enhancement model are taken into consideration when adjusting the model parameters, which can improve the performance of the face enhancement model in many aspects and help the face enhancement model cope with various low-quality images with complex lighting and shadows.

[0261] The structure of the electronic device involved in the embodiments of the present application is briefly introduced below with reference to the accompanying drawings.

[0262] In the embodiment of the present application, the electronic device may be a mobile phone, a smart screen, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a handheld or laptop device, a media player, a smart projector, a smart TV, a desktop computer, an in-vehicle infotainment system, or other device equipped with a touch screen. The embodiment of the present application does not impose any restrictions on the specific type and form of the electronic device.

[0263] It should be understood that the software system, hardware system, device and chip in the embodiments of the present application can execute the training methods and image processing methods of the various models of the aforementioned embodiments of the present application, that is, the specific working processes of the following various products can refer to the corresponding processes in the aforementioned method embodiments.

[0264] Please refer to Fig.17 , Fig.171 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0265] It should be noted that Fig.17 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include Fig.17 More or fewer components than those shown, or the electronic device 100 may include Fig.17 Combinations of some of the components shown, or the electronic device 100 may include Fig.17 Subassemblies of some of the components shown. Fig.17 The components shown may be implemented by hardware, software, or a combination of hardware and software.

[0266] The processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). Different processing units may be independent devices or integrated devices.

[0267] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0268] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0269] The processor 110 can run the software code of the image processing method provided in the embodiment of the present application, process the facial image through the inverse rendering model, and use the processing result of the inverse rendering model as the facial prior information. The facial enhancement model can effectively use the facial prior information to accurately restore the face, and even under complex lighting conditions, it can avoid the generation of unnatural textures and output high-quality captured images.

[0270] The electronic device 100 can realize the display function through the GPU, the display screen 194 and the application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU can also be used to perform mathematical and posture calculations for graphics rendering, etc. The processor 110 may include one or more GPUs, which can generate or change display information by executing program instructions.

[0271] In the embodiment of the present application, the ability of the electronic device 100 to display various display interfaces depends on the display functions provided by the GPU, the display screen 194, and the application processor, for example, displaying preview images, captured images, captured videos, processed images, processed videos, etc.

[0272] In some embodiments, the electronic device 100 may include 1 or N display screens 194 , where N may be a positive integer greater than 1.

[0273] The display screen 194 in the embodiment of the present application is a touch screen. A touch sensor 180K may be integrated in the display screen 194. The touch sensor 180K may also be referred to as a "touch panel". In other words, the display screen 194 may include a display panel and a touch panel, and the touch sensor 180K and the display screen 194 form a touch screen, also known as a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. After the touch operation detected by the touch sensor 180K, it can be passed to the upper layer by the driver of the kernel layer (such as the TP driver) to determine the type of touch event. Visual output related to the touch operation can be provided by the display screen 194. In other embodiments, the touch sensor 180K may also be arranged on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0274] The charging management module 140 is used to receive charging input from a charger. The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0275] The mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G etc. applied on the electronic device 100 .

[0276] The camera 193 is used to capture images. It can be triggered to start by application instructions to realize the photo function, such as taking pictures to obtain images of any scene. The camera may include components such as imaging lenses, filters, and image sensors. The light emitted or reflected by the object enters the imaging lens, passes through the filter, and finally converges on the image sensor. The image sensor is mainly used to converge the light emitted or reflected by all objects (also referred to as photographed objects) in the photographing angle of view; the filter is mainly used to filter out the redundant light waves in the light (for example, light waves other than visible light, such as infrared); the image sensor is mainly used to perform photoelectric conversion on the received light signal, convert it into an electrical signal, and input it into the processor 130 for subsequent processing. Among them, the camera 193 can be located in front of the electronic device 100, or on the back of the electronic device 100. The specific number and arrangement of the cameras can be set according to needs, and this application does not impose any restrictions.

[0277] Exemplarily, the electronic device 100 includes a front camera and a rear camera. For example, the front camera or the rear camera may include one or more cameras. Taking the electronic device 100 having one rear camera as an example, when the electronic device 100 starts one rear camera for shooting, the image processing method provided in the embodiment of the present application can be used. Alternatively, the camera is arranged on an external accessory of the electronic device 100, and the external accessory is rotatably connected to the frame of the mobile phone, and the angle formed between the external accessory and the display screen 194 of the electronic device 100 is any angle between 0-360 degrees. For example, when the electronic device 100 takes a selfie, the external accessory drives the camera to rotate to a position facing the user. Of course, when the mobile phone has multiple cameras, only some of the cameras may be arranged on the external accessories, and the remaining cameras may be arranged on the electronic device 100 body, and the embodiment of the present application does not impose any restrictions on this.

[0278] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 may include a program storage area and a data storage area. The internal memory 121 may also store the software code of the image processing method provided in the embodiment of the present application. When the processor 110 runs the software code, the process steps of the image processing method are executed to obtain high-quality captured images. The high-quality captured images are not only high in clarity and rich in details, but also free of smearing, artifacts, pseudo-textures and other problems. The internal memory 121 may also store the captured images.

[0279] Of course, the software code of the image processing method provided in the embodiment of the present application can also be stored in an external memory, and the processor 110 can run the software code through the external memory interface 120 to execute the process steps of the image processing method to obtain a high-quality captured image. The image captured by the electronic device 100 can also be stored in an external memory.

[0280] In some embodiments, the pressure sensor 180A may be provided on the display screen 194. When a touch operation is applied to the display screen 194, the electronic device 100 detects the touch operation strength according to the pressure sensor 180A. The electronic device 100 may also calculate the touched position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation strengths may correspond to different operation instructions.

[0281] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to realize functions such as unlocking, accessing application locks, taking photos, and answering calls.

[0282] The key 190 includes a power key and a volume key. The key 190 can be a mechanical key or a touch key. The electronic device 100 can receive a key input signal and implement a function related to the case input signal.

[0283] In addition, various types of operating systems are running on the above components. For example, Android system, IOS operating system, Symbian operating system, BlackBerry operating system, Linux operating system, Windows operating system, etc. This is only an exemplary description and is not limited to this. Different applications can be installed and run on these operating systems, such as any type of input method application. The image processing method provided in the embodiment of the present application can be implemented in the electronic device 100 having the above hardware structure.

[0284] The above briefly introduces the structure of the electronic device 100 involved in the embodiment of the present application. The following briefly introduces the software structure involved in the embodiment of the present application. Fig.18 , Fig.18 It is a software structure block diagram of an electronic device shown in an exemplary embodiment of the present application.

[0285] The software system may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture or a cloud architecture. The embodiment of the present application takes the layered architecture as an example to exemplarily describe the software system of the electronic device 100.

[0286] like Fig.18 As shown, the layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the electronic device 100 is an Android system as an example for explanation. The Android system is divided into five layers, from top to bottom, namely, application layer 210, application framework layer 220, hardware abstraction layer 230, driver layer 240, and hardware layer 250.

[0287] The application layer 210 may include camera and gallery applications, and may also include calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0288] The application framework layer 220 provides an application access interface and a programming framework for the applications of the application layer 210 .

[0289] For example, the application framework layer 220 includes a camera access interface, which is used to provide camera shooting services through camera management and camera devices.

[0290] The camera management in the application framework layer 220 is used to manage the camera. The camera management can obtain the parameters of the camera, such as determining the working status of the camera.

[0291] The camera device in the application framework layer 220 is used to provide a data access interface between different camera devices and camera management.

[0292] The hardware abstraction layer 230 is used to abstract the hardware. For example, the hardware abstraction layer 230 may include a camera hardware abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer may include camera device 1, camera device 2, etc.; the camera hardware abstraction layer may be connected to the camera algorithm library, and the camera hardware abstraction layer may call the algorithm in the camera algorithm library.

[0293] The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include a camera driver, a digital signal processor driver, and a graphics processor driver.

[0294] The hardware layer 250 may include sensors, image signal processors, digital signal processors, graphics processors, and other hardware devices. The sensors may include sensor 1, sensor 2, etc., and may also include depth sensors (time of flight, TOF) and multi-spectral sensors.

[0295] The following is an illustrative description of the workflow of the software system of the electronic device 100 in conjunction with a shooting scenario.

[0296] When the user performs a single-click operation on the touch sensor 180K, the camera APP is awakened by the single-click operation and calls various camera devices of the camera hardware abstraction layer through the camera access interface. The camera hardware abstraction layer can issue a command to call the camera to the camera device driver, and the camera algorithm library starts to load the algorithm used in the embodiment of the present application.

[0297] When the sensor of the hardware layer is called, for example, after calling the sensor 1 in the camera to obtain the original image, the original image is sent to the image signal processing for preliminary processing such as registration, and then returned to the hardware abstraction layer through the camera device driver, and then processed using the algorithm in the loaded camera algorithm library, for example, using the trained inverse rendering model and the trained face enhancement model, and processing according to the relevant processing steps provided in the embodiment of the present application to obtain the target face image. Among them, the trained inverse rendering model and the trained face enhancement model can be processed by calling the digital signal processor through the digital signal processor driver and calling the graphics processor through the graphics processor driver.

[0298] The obtained face target image and the original image are spliced ​​to obtain a high-definition image corresponding to the original image, which is then sent back to the camera application through the camera hardware abstraction layer and the camera access interface for display and storage.

[0299] Please refer to Fig.19 , Fig.19 FIG. 1 is a schematic diagram of an image processing device shown in an exemplary embodiment of the present application.

[0300] It should be understood that the image processing device 300 can execute the image processing method provided in the present application; the image processing device 300 includes: a display unit 310, an acquisition unit 320, a first processing unit 330 and a second processing unit 340.

[0301] Optionally, the inverse rendering model, the face enhancement model and the light and shadow enhancement model can be deployed in the image processing device 300.

[0302] A display unit 310, configured to display a first interface, where the first interface includes a first control; An acquisition unit 320, configured to acquire a face image in response to a first operation on a first control; A first processing unit 330 is used to process the face image using the trained inverse rendering model to obtain a face normal image, a face albedo image, and an environment image; The second processing unit 340 is used to enhance the face image, face normal image, face albedo image and environment image using the trained face enhancement model to obtain a face target image; the clarity of the face target image is greater than the clarity of the face image.

[0303] It should be noted that the above-mentioned image processing device 300 is embodied in the form of a functional unit. The term "unit" here can be implemented in the form of software and / or hardware, and is not specifically limited to this.

[0304] For example, a "unit" may be a software program, a hardware circuit, or a combination of the two that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group processor, etc.) and a memory for executing one or more software or firmware programs, a combined logic circuit, and / or other suitable components that support the described functions.

[0305] Therefore, the units of each example described in the embodiments of the present application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0306] The embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored; when the computer-readable storage medium is run on an image processing device, the image processing device executes the image processing method of any of the above embodiments. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that can be integrated with one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium, or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0307] The embodiment of the present application also provides a computer program product including computer instructions, which, when executed on an image processing device, enables the image processing device to execute the image processing method of any of the above embodiments.

[0308] The present application also provides a chip. Fig. 20 , Fig. 20 It is a schematic diagram of the structure of a chip provided in an embodiment of the present application. Fig. 20 The chip shown can be a general-purpose processor or a dedicated processor. The chip includes a processor 410. The processor 410 is used to execute the image processing method of any of the above embodiments.

[0309] Optionally, the chip further includes a transceiver 420, which is used to accept control of the processor and to support the image processing device to execute the technical solution shown above.

[0310] Optional, Fig. 20 The chip shown may also include: a storage medium 430 .

[0311] It should be noted that Fig. 20The chip shown can be implemented using the following circuits or devices: one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits that can perform the various functions described throughout this application.

[0312] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0313] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0314] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0315] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0316] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0317] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0318] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be within the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Display a first interface, wherein the first interface includes a first control; In response to a first operation on the first control, acquiring a facial image; Processing the face image using the trained inverse rendering model to obtain a face normal image, a face albedo image, and an environment image; The face image, the face normal image, the face albedo image and the environment image are enhanced using a trained face enhancement model to obtain a face target image; the clarity of the face target image is greater than the clarity of the face image.

2. The method according to claim 1, characterized in that The step of acquiring a facial image in response to a first operation on the first control includes: In response to a first operation on the first control, obtaining an original image: Determining the clarity of the original image; It is detected that the clarity of the original image is less than a preset clarity threshold, and the original image is cropped to obtain the face image.

3. The method according to claim 1, characterized in that The trained inverse rendering model includes a first network, a second network, and a third network. The trained inverse rendering model is used to process the face image to obtain a face normal image, a face albedo image, and an environment image, including: Processing the face image using the first network to obtain the face normal image; Processing the face image and the face normal image using the second network to obtain the face albedo image; The face image, the face normal image and the face albedo image are processed by using the third network to obtain the environment image.

4. The method according to claim 3, characterized in that The first network includes a first vector codebook module, the first vector codebook module includes a first discrete vector set, and the processing of the face image by using the first network to obtain the face normal image includes: Extracting a first latent vector in the face image using the first network; Searching for a vector close to the first potential vector in the first discrete vector set; The face normal image is generated according to a vector close to the first latent vector.

5. The method according to claim 3, characterized in that: The second network includes a second vector codebook module, the second vector codebook module includes a second discrete vector set, and the second network is used to process the face image and the face normal image to obtain the face albedo image, including: Splicing the face image and the face normal image to obtain a first spliced ​​image; extracting a second latent vector in the first spliced ​​image using the second network; Searching for a vector close to the second latent vector in the second discrete vector set; The face albedo image is generated according to a vector close to the second latent vector.

6. The method according to claim 1, characterized in that The trained face enhancement model includes a third vector codebook module, the third vector codebook module includes a third discrete vector set, and the face image, the face normal image, the face albedo image, and the environment image are enhanced by using the trained face enhancement model to obtain a face target image, including: splicing the face image, the face normal image, the face albedo image, and the environment image to obtain a second spliced ​​image; extracting a third latent vector in the second spliced ​​image using the trained face enhancement model; Searching for a vector close to the third potential vector in the third discrete vector set; The face target image is generated according to a vector close to the third latent vector.

7. The method according to claim 1, characterized in that The method of using the trained face enhancement model to enhance the face image, the face normal image, the face albedo image, and the environment image to obtain a face target image includes: Determine a fill-light environment image corresponding to the environment image using a trained light and shadow enhancement model; The face image, the face normal image, the face albedo image and the fill-light environment image are enhanced using a trained face enhancement model to obtain a face target image.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Using a first training set to train an initial inverse rendering network to obtain an inverse rendering model in training; the first training set includes high-definition sample images; The inverse rendering model in training is trained using a second training set to obtain the trained inverse rendering model; the second training set includes low-definition sample images, and the low-definition sample images are obtained by degrading the high-definition sample images.

9. The method according to claim 8, characterized in that The method further comprises: Using a third training set to train the initial face enhancement network to obtain a face enhancement model in training; the third training set includes the high-definition sample image; The face enhancement model in training is trained using the fourth training set to obtain the trained face enhancement model; the fourth training set includes the low-definition sample image, and the normal sample image, albedo sample image, and environment sample image corresponding to the low-definition sample image; the normal sample image and the albedo sample image are obtained by processing the low-definition sample image with the trained inverse rendering model; the environment sample image is simulated by the trained light and shadow enhancement model.

10. The method according to claim 8, characterized in that The method further comprises: In the process of training the face enhancement model, a target loss value is calculated according to a preset loss function; the target loss value is used to represent the loss between the high-definition sample image and the predicted image output by the face enhancement model under training; According to the target loss value, the face enhancement model in training is continuously trained in a back-propagation manner to obtain the trained face enhancement model.

11. The method according to claim 3, characterized in that The first network is VQVAE, and / or the second network is VQVAE.

12. The method according to claim 2, characterized in that: The method further comprises: The original image and the target face image are spliced ​​together to obtain a high-definition image corresponding to the original image.

13. An electronic device, characterized in that: The electronic device comprises: one or more processors, and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the electronic device executes the method as described in any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, which, when executed on an electronic device, enable the electronic device to perform the method as claimed in any one of claims 1 to 12.

15. A computer program product, characterized in that The computer program product comprises a computer program code which, when executed, causes the method according to any one of claims 1 to 12 to be performed.

Images