Image Processing Method, Electronic Device, and Storage Medium

Abstract

The present application provides an image processing method, an electronic device, and a storage medium, relating to the field of image processing. The method includes: displaying a first interface, and in response to a first operation on a first control in the first interface, acquiring a face image; processing the face image by using a trained inverse rendering model to obtain a face normal image, a face albedo image, and an environment image; and performing enhancement processing on the face image, the face normal image, the face albedo image, and the environment image by using a trained face enhancement model to obtain a face target image. In this implementation manner, the processing result of the inverse rendering model on the face image is used as face prior information, and the face enhancement model can effectively utilize the face prior information to accurately restore the face. Even under complex light and shadow conditions, it can avoid generating unnatural textures and output high-quality captured images.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular, to an image processing method, an electronic device, and a storage medium. Background Art

[0002] With the popularization of electronic devices with shooting functions in daily life, it has become a daily behavior for people to use electronic devices to take pictures. However, during the shooting process, due to reasons such as low light, camera shake, and movement of the object being photographed, the quality of the captured images is often poor.

[0003] Although various methods for enhancing image quality have been proposed in current technologies, these methods also bring new problems while improving image quality, especially in face image processing. For example, phenomena such as face blurring and pseudo-textures may appear in the enhanced images, seriously affecting the quality of face images. Therefore, there is an urgent need for a new image processing method to solve these problems. Summary of the Invention

[0004] This application provides an image processing method, an electronic device, and a storage medium. The processing result of the inverse rendering model on the face image is used as face prior information, and the face enhancement model can effectively utilize the face prior information to accurately restore the face. Even under complex light and shadow conditions, it can avoid generating unnatural textures and output high-quality captured images.

[0005] To achieve the above object, this application adopts the following technical solutions:

[0006] In a first aspect, an image processing method is provided. The image processing method includes: displaying a first interface, where the first interface includes a first control; in response to a first operation on the first control, obtaining a face image; using a trained inverse rendering model to process the face image to obtain a face normal image, a face albedo image, and an environment image; using a trained face enhancement model to perform enhancement processing on the face image, the face normal image, the face albedo image, and the environment image to obtain a face target image.

[0007] Optionally, the first operation is used to indicate the start of shooting, and the first operation can be a click operation on the shooting control.

[0008] Optionally, the face normal image is used to represent the normal features corresponding to the portrait in the face image, or in other words, to represent the normal direction of the face surface in the face image.

[0009] Optionally, the face albedo image is used to represent the albedo features corresponding to the portrait in the face image, or in other words, to represent features such as the color and texture of the face surface in the face image.

[0010] Optionally, the environmental image is used to represent environmental content other than the human face, or to represent the influence of the lighting of the shooting environment.

[0011] Optionally, the clarity of the face target image is greater than that of the face image.

[0012] Optionally, the face target image is rich in details and does not have problems such as strong smearing, artifacts, and pseudo-textures.

[0013] In this implementation, the trained inverse rendering model is used to split the face and the environment in the face image to obtain the face normal image, the face albedo image, and the environmental image, that is, accurate prior information is obtained. The trained face enhancement model makes full use of this prior information during the enhancement process, can effectively restore the details of the face image, eliminate the smearing, artifacts, pseudo-textures, etc. existing in the face image, and improve the clarity and quality of the face image.

[0014] Combined with the first aspect, in some implementations of the first aspect, in response to a first operation on the first control, obtaining a face image includes: in response to a first operation on the first control, obtaining an original image: determining the clarity of the original image; detecting that the clarity of the original image is less than a preset clarity threshold, and cropping the original image to obtain a face image.

[0015] In this implementation, the original image is effectively screened by the clarity, so that the determined face images are all face images that need to be enhanced. On the one hand, it is convenient to only process some face images that need to be enhanced subsequently, which can improve the image processing efficiency; on the other hand, it avoids enhancing face images that do not need to be enhanced, saves the resources of the electronic device, reduces power consumption; at the same time, it also avoids enhancing face images that do not need to be enhanced, resulting in artifacts and pseudo-textures.

[0016] Combined with the first aspect, in some implementations of the first aspect, the trained inverse rendering model includes a first network, a second network, and a third network. Using the trained inverse rendering model to process the face image to obtain the face normal image, the face albedo image, and the environmental image includes: using the first network to process the face image to obtain the face normal image; using the second network to process the face image and the face normal image to obtain the face albedo image; using the third network to process the face image, the face normal image, and the face albedo image to obtain the environmental image.

[0017] Optionally, the first network is a VQVAE, and / or the second network is a VQVAE.

[0018] Optionally, the third network can be a VQVAE or a U-Net.

[0019] In this implementation manner, introducing the VQ technology into each network included in the inverse rendering model can extract accurate prior information, which helps the subsequent face enhancement model effectively utilize this prior information to enhance face images, avoiding the generation of pseudo-textures, artifacts, unnatural effects, etc., and greatly improving the quality of the images.

[0020] Combined with the first aspect, in some implementation manners of the first aspect, the first network includes a first vector codebook module, and the first vector codebook module includes a first discrete vector set. Processing a face image by using the first network to obtain a face normal image includes: extracting a first latent vector in the face image by using the first network; searching for a vector similar to the first latent vector in the first discrete vector set; and generating a face normal image according to the vector similar to the first latent vector.

[0021] Optionally, the first latent vector may include 3D structure information and geometric information corresponding to the face.

[0022] In this implementation manner, vector quantization by VQVAE can effectively map the information in the continuous latent space to a discrete space, which helps improve the computing efficiency and generalization ability of the first network. In this way, the first network can extract rich 3D structure information and geometric information from the input face image, laying a foundation for subsequent enhancement processing of the face image, that is, it helps to more accurately restore the 3D structure features and geometric features of the face when enhancing the face image subsequently.

[0023] Combined with the first aspect, in some implementation manners of the first aspect, the second network includes a second vector codebook module, and the second vector codebook module includes a second discrete vector set. Processing a face image and a face normal image by using the second network to obtain a face albedo image includes: splicing the face image and the face normal image to obtain a first spliced image; extracting a second latent vector in the first spliced image by using the second network; searching for a vector similar to the second latent vector in the second discrete vector set; and generating a face albedo image according to the vector similar to the second latent vector.

[0024] Optionally, the second latent vector may include texture information and skin information corresponding to the face.

[0025] In this implementation method, vector quantization is performed through VQVAE, which can effectively map the information in the continuous latent space to a discrete space, helping to improve the computational efficiency and generalization ability of the second network. In this way, the second network can extract accurate texture information and skin information from the input face image and face normal image, laying a foundation for subsequent enhancement processing of the face image, that is, it helps to more accurately restore the texture features and skin features of the face during subsequent enhancement processing of the face image, enhancing the details and realism of the face image and avoiding the generation of pseudo-textures, artifacts, unnatural effects, etc.

[0026] Combined with the first aspect, in some implementation methods of the first aspect, the trained face enhancement model includes a third vector codebook module, and the third vector codebook module includes a third discrete vector set. Using the trained face enhancement model to perform enhancement processing on the face image, face normal image, face albedo image, and environmental image to obtain a face target image, including: splicing the face image, face normal image, face albedo image, and environmental image to obtain a second spliced image; using the trained face enhancement model to extract a third latent vector from the second spliced image; searching for a vector similar to the third latent vector in the third discrete vector set; generating a face target image according to the vector similar to the third latent vector.

[0027] Optionally, the third latent vector may include 3D structure information, geometric information, texture information, skin information, environmental illumination information, light and shadow information, etc. corresponding to the face.

[0028] In this implementation method, the face enhancement model can obtain accurate 3D structure information, geometric information, texture information, skin information, environmental illumination information, light and shadow information, etc. from the prior information. Therefore, even in the case of poor face image quality and complex face light and shadow, it can correctly guide the restoration of the 3D structure, geometry, texture, skin, environmental illumination, light and shadow, etc. of the face image based on this information, effectively avoiding the generation of pseudo-textures and artifacts, improving the quality of the image, and also improving the robustness of the face enhancement model, making the applicability of the face enhancement model more extensive.

[0029] Combined with the first aspect, in some implementation methods of the first aspect, using the trained face enhancement model to perform enhancement processing on the face image, face normal image, face albedo image, and environmental image to obtain a face target image, including: using the trained light and shadow enhancement model to determine a supplementary light environmental image corresponding to the environmental image; using the trained face enhancement model to perform enhancement processing on the face image, face normal image, face albedo image, and supplementary light environmental image to obtain a face target image.

[0030] In this implementation method, the intelligent fill light for the environmental image is performed through the light and shadow enhancement model, and then the face enhancement model is used to finely adjust the face image based on the environmental image after fill light. Such processing not only enhances the face details but also achieves the comprehensive fill light effect for the face and the surrounding environment, ensuring the balance of the face and environmental lighting, which is beneficial to generating a face target image with higher clarity and stronger contrast.

[0031] Combined with the first aspect, in some implementation manners of the first aspect, the image processing method further includes: training the initial inverse rendering network using the first training set to obtain the inverse rendering model during training; the first training set includes high-definition sample images; training the inverse rendering model during training using the second training set to obtain the trained inverse rendering model; the second training set includes low-definition sample images, and the low-definition sample images are obtained by degrading the high-definition sample images.

[0032] In this implementation method, when training the inverse rendering model, it is divided into two stages. In the first stage, the initial inverse rendering network is trained through high-definition sample images, enabling the vector codebook module to capture and save the prior information of the high-definition sample images; in the second stage, further training is performed through low-definition sample images, enabling the inverse rendering model during training to learn the ability to accurately utilize the prior information to restore and output high-quality images. Through the training of these two stages, the trained inverse rendering model has strong robustness, so that when facing face images in various complex light and shadow scenarios, it can also accurately extract and utilize the prior information to generate high-quality images.

[0033] Combined with the first aspect, in some implementation manners of the first aspect, the image processing method further includes: training the initial face enhancement network using the third training set to obtain the face enhancement model during training; the third training set includes high-definition sample images; training the face enhancement model during training using the fourth training set to obtain the trained face enhancement model; the fourth training set includes low-definition sample images, as well as the normal sample images, albedo sample images, and environmental sample images corresponding to the low-definition sample images; the normal sample images and albedo sample images are obtained by processing the low-definition sample images using the trained inverse rendering model; the environmental sample images are simulated by the trained light and shadow enhancement model.

[0034] In this implementation method, when training the face enhancement model, it is divided into two stages. In the first stage, the initial face enhancement network is trained with high-definition sample images, enabling the vector codebook module in the initial face enhancement network to learn how to quantize vectors so as to accurately utilize prior information to restore and output high-quality face images. In the second stage, further training is carried out with low-definition sample images, enabling the face enhancement model being trained to learn the ability to accurately utilize prior information to restore and output high-quality images. Through the training of these two stages, the trained face enhancement model is equipped with strong robustness, so that when facing face images in various complex light and shadow scenarios, it can also accurately utilize prior information to generate high-quality images.

[0035] Combined with the first aspect, in some implementation methods of the first aspect, the image processing method further includes: during the process of training the face enhancement model, calculating a target loss value according to a preset loss function; and continuing to train the face enhancement model being trained in a backpropagation manner according to the target loss value to obtain a trained face enhancement model.

[0036] Optionally, the target loss value is used to represent the loss between the high-definition sample image and the predicted image output by the face enhancement model being trained, and this loss consists of three parts. The first part is the L1 loss, the second part is the perceptual loss, and the third part is the adversarial loss.

[0037] Among them, the L1 loss is used to represent the loss between the high-definition sample image and the predicted image (or called the reconstructed image); the perceptual loss is used to represent the feature loss between the high-definition sample image and the predicted image (or called the reconstructed image); and the adversarial loss is used to represent the loss between the real image and the image generated by the generator in the face enhancement model.

[0038] In this implementation method, using the backpropagation algorithm to adjust the model parameters helps the model converge to the optimal solution faster, improving the training speed of the face enhancement model. Moreover, when adjusting the model parameters, considering the losses of the three parts of the face enhancement model can improve the performance of the face enhancement model in multiple aspects, helping the face enhancement model to handle various low-quality and complex light and shadow images.

[0039] Combined with the first aspect, in some implementation methods of the first aspect, the image processing method further includes: splicing the original image and the face target image to obtain a high-definition image corresponding to the original image.

[0040] In this implementation method, enhancement processing is specifically performed on face images. On the one hand, it can effectively restore the details of the face area, eliminate smear, artifacts, pseudo-textures, etc. existing in the face image, and ensure that the image quality of the face area is significantly improved. On the other hand, by separately enhancing the face image and then splicing the original image and the face target image to obtain a high-definition image, the image processing efficiency can be effectively improved, and the resource consumption of the electronic device can be reduced. In addition, users usually pay more attention to the faces in the captured images. The image processing method provided in this application can significantly improve the image quality of the face area while increasing the image processing speed, which can greatly enhance the user's favorability.

[0041] In a second aspect, the present application provides an image processing device, which includes one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, and the computer program code includes computer instructions. The one or more processors call the computer instructions to enable the image processing device to execute any one of the methods provided in the technical solution of the first aspect.

[0042] In a third aspect, the present application provides an electronic device, which includes: a processor, a memory, and an interface; the processor, the memory, and the interface cooperate with each other to enable the electronic device to execute any one of the methods provided in the technical solution of the first aspect.

[0043] In a fourth aspect, the present application provides a chip, which includes a processor. The processor is used to read and execute the computer program stored in the memory to execute the methods in the first aspect and any possible implementation manners thereof.

[0044] Optionally, the chip further includes a memory, and the memory is connected to the processor through a circuit or a wire.

[0045] Optionally, the chip further includes a communication interface.

[0046] In a fifth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor is enabled to execute any one of the technical solutions in the first aspect.

[0047] In a sixth aspect, the present application provides a computer program product, which includes: computer program code. When the computer program code runs on an electronic device, the electronic device is enabled to execute any one of the technical solutions in the first aspect.

[0048] The technical effects obtained in the above second aspect, third aspect, fourth aspect, fifth aspect, and sixth aspect are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be elaborated here. Description of the Drawings

[0049] Figure 1 is a schematic diagram of a scenario for capturing an image shown in an exemplary embodiment of the present application;

[0050] Figure 2 is a schematic diagram of displaying a captured image shown in an exemplary embodiment of the present application;

[0051] Figure 3 is a schematic diagram of a process flow of an image processing method shown in an exemplary embodiment of the present application;

[0052] Figure 4 is a schematic diagram of a process flow of a method for obtaining a face image shown in an exemplary embodiment of the present application;

[0053] Figure 5 is a schematic diagram of a face image shown in an exemplary embodiment of the present application;

[0054] Figure 6 is a schematic diagram of another process flow of an image processing method shown in an exemplary embodiment of the present application;

[0055] Figure 7 is a schematic diagram of the structure of a trained inverse rendering model shown in an exemplary embodiment of the present application;

[0056] Figure 8 is a schematic diagram of a face image, a face normal image, a face albedo image, and an environment image shown in an exemplary embodiment of the present application;

[0057] Figure 9 is a schematic diagram of a process flow of an inverse rendering model processing shown in an exemplary embodiment of the present application;

[0058] Figure 10 is a schematic diagram of a first network structure shown in an exemplary embodiment of the present application;

[0059] Figure 11 is a schematic diagram of a second network structure shown in an exemplary embodiment of the present application;

[0060] Figure 12 is a schematic diagram of a process flow of a face enhancement model processing shown in an exemplary embodiment of the present application;

[0061] Figure 13 is a schematic diagram of the structure of a face enhancement model shown in an exemplary embodiment of the present application;

[0062] Figure 14 is a schematic diagram of a process flow of training an inverse rendering model shown in an exemplary embodiment of the present application;

[0063] Figure 15It is a schematic diagram showing a light and shadow enhancement model enriching high-definition sample images shown in an exemplary embodiment of the present application;

[0064] Figure 16 It is a schematic flowchart showing the training of a face enhancement model shown in an exemplary embodiment of the present application;

[0065] Figure 17 It is a schematic diagram showing the structure of an electronic device shown in an exemplary embodiment of the present application;

[0066] Figure 18 It is a software structure block diagram of an electronic device shown in an exemplary embodiment of the present application;

[0067] Figure 19 It is a schematic diagram showing an image processing device shown in an exemplary embodiment of the present application;

[0068] Figure 20 It is a schematic diagram showing the structure of a chip provided in an embodiment of the present application. Detailed implementation manners

[0069] In the embodiments of the present application, the following terms "first", "second", "third", and "fourth" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features.

[0070] To facilitate the understanding of the technical solutions in the embodiments of the present application, before introducing the technical solutions in the examples of the present application, some terms involved in the embodiments of the present application are first explained.

[0071] 1. Convolutional Neuron Network (CNN)

[0072] A convolutional neural network is a deep neural network with a convolutional structure. A convolutional neural network includes a feature extractor composed of convolutional layers and subsampling layers, and this feature extractor can be regarded as a filter; a convolutional layer refers to the neuron layer in a convolutional neural network that performs convolutional processing on the input signal. In the convolutional layer of a convolutional neural network, a neuron can only be connected to some neighboring layer neurons. In a convolutional layer, there are usually several feature planes, and each feature plane can be composed of some rectangularly arranged neural units.

[0073] 2. RGB (Red, Green, Blue) color space

[0074] The RGB color space can also be called the RGB domain, which refers to a color model related to the structure of the human visual system. According to the structure of the human eye, all colors are regarded as different combinations of red, green, and blue.

[0075] 3. RGB (Red, Green, Blue) channel values

[0076] The RGB channel values, also known as RGB values, refer to the intensity values of the three color channels of red, green, and blue used to define a specific color in the RGB color model. These intensity values together determine the final displayed color.

[0077] 4. Backlight

[0078] Backlight is a situation where the subject to be photographed (also known as the object to be photographed) is exactly between the light source and the camera. In this state, it is easy to cause the problem of insufficient exposure of the subject to be photographed. Therefore, in general, users should try to avoid photographing objects under backlight conditions.

[0079] 5. Albedo

[0080] Albedo generally refers to the ratio of the reflected radiation flux to the incident radiation flux under the influence of solar radiation on the earth's surface. It is an important variable reflecting many surface parameters, such as reflecting the absorption ability of the earth's surface to solar radiation.

[0081] In the embodiments of the present application, albedo refers to the ratio of the reflected radiation flux to the incident radiation flux when the human head is irradiated by light, and is used to reflect the absorption ability of the surface layer of the human head (such as the face, scalp, etc.) to light radiation.

[0082] 6. Normal direction

[0083] In the embodiments of the present application, the normal direction represents the direction of the normal line.

[0084] 7. Receptive field

[0085] In the deep neural network in the field of machine vision, there is a concept called receptive field, which is used to represent the size of the receptive range of neurons at different positions inside the network for the original image.

[0086] 8. Priori

[0087] Priori refers to the inherent knowledge or preset information about the human face structure and features, which can help the electronic device improve the accuracy and efficiency when processing face images.

[0088] 9. Back Propagation Algorithm (BP)

[0089] The neural network can use the error backpropagation algorithm to correct the magnitudes of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward propagation of the input signal until the output will generate an error loss, and the error loss information is backpropagated to update the parameters in the initial neural network model, thereby converging the error loss.

[0090] The backpropagation algorithm is a backpropagation movement dominated by the error loss, used to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0091] The above is a simple introduction to the terms involved in the embodiments of this application, which will not be elaborated below.

[0092] With the popularization of electronic devices with a shooting function in life, people using electronic devices to take pictures has become a daily behavior. However, during the shooting process, due to factors such as the hardware performance of the shooting device (such as the performance of the photosensitive element), the shooting environment (such as low light), lens shake, telephoto camera shooting, and movement of the object being photographed, the quality of the captured image is not good.

[0093] Although many methods for enhancing image quality have been proposed in current technologies, while these methods improve the image quality, they also cause new problems, especially in the field of face image processing. For example, the composition of the face has strong prior information, and traditional image enhancement methods (such as linear filtering methods, histogram equalization methods, sharpening enhancement methods, etc.) cannot utilize the face prior information, resulting in the inability to obtain a better face contour, enhance face details better when processing face images, and even introduce unnatural and obtrusive pseudo-textures that do not exist on the face when the facial light and shadow are complex. That is to say, using traditional image enhancement methods to process face images will result in problems such as low clarity, insufficient details, strong smear feeling, noise, artifacts, and pseudo-textures in the processed face images.

[0094] Among them, insufficient details means the loss of details. For example, the loss of information such as fine textures (such as skin textures, hair, etc.), edges, and shapes that originally existed in the face image, resulting in the face image looking less clear or lacking a three-dimensional sense.

[0095] Strong smear feeling means that the colors and textures in the face image are over-smoothed, resulting in the face image looking like it has been smeared, lacking sharpness and clarity.

[0096] The existence of artifacts means that non-real existing content is added to the processed face image.

[0097] For example, the facial light and shadow of a face image is complex. There is a shadow above the eyebrows of the face in this face image. When using traditional image enhancement methods for enhancement processing, it is impossible to accurately distinguish real features from light and shadow effects, and wrongly enhance this shadow as an eyebrow. As a result, there are two eyebrows in the enhanced face image. One is the real eyebrow, and the other is an unreal eyebrow. This unreal eyebrow can be called an artifact. The face image processed in this way seriously affects the authenticity of the face image and the user experience.

[0098] The existence of pseudo-texture means that non-existent texture information is added to the processed face image.

[0099] For example, the facial light and shadow of a face image is complex. There is an unnatural light and shadow around the eyes of the face in this face image. When using traditional image enhancement methods for enhancement processing, it is impossible to accurately distinguish real features from light and shadow effects, and wrongly enhance this circle of light and shadow as the outline of glasses. As a result, glasses appear in the enhanced face image, and these glasses are non-existent. The face image processed in this way seriously affects the authenticity of the face image and the user experience.

[0100] In view of this, the embodiments of the present application provide an image processing method, or rather a face image enhancement method. This method can not only eliminate problems such as face blurring, facial smearing, noise, artifacts, and pseudo-textures, but also supplement details and perform super-resolution reconstruction. Even in the case of complex light and shadow, it can process a clear, complete, high-resolution, and high-quality face image from a low-quality image, improving the user's shooting experience.

[0101] Specifically, this image processing method may include displaying a first interface, and in response to a first operation on a first control in the first interface, obtaining a face image; using a trained inverse rendering model to process the face image to obtain a face normal image, a face albedo image, and an environment image; using a trained face enhancement model to perform enhancement processing on the face image, the face normal image, the face albedo image, and the environment image to obtain a face target image; the clarity of the face target image is greater than the clarity of the face image.

[0102] The image processing method provided by the embodiments of the present application uses a trained inverse rendering model to extract the face normal image, face albedo image, and environmental image corresponding to the captured image, thereby obtaining accurate face prior information, laying a solid foundation for subsequent face enhancement processing. In the enhancement stage, when using the trained face enhancement model to perform enhancement processing on the face image, the face prior information can be used to accurately restore the face, effectively improving the face contour and enhancing the face details. Even under complex lighting conditions, unnatural textures can be avoided. Finally, through this image processing method, a high-quality captured image can be output. The high-quality captured image not only has high clarity and rich details, but also does not have problems such as strong smearing, artifacts, and pseudo-textures.

[0103] Before introducing the image processing method provided by the embodiments of the present application, an example of the application scenario of the image processing method provided by the embodiments of the present application will be given in combination with the accompanying drawings.

[0104] Application scenario 1: Photo-taking scenario

[0105] The image processing method provided by the embodiments of the present application can be applied to a photo-taking scenario. For example, it can be applied to captured images in various scenarios. Please refer to Figure 1 , Figure 1 which is a schematic diagram of a scenario of a captured image shown in an exemplary embodiment of the present application.

[0106] In the embodiments of the present application, a mobile phone is used as an example of an electronic device for description. Exemplarily, the display screen of the mobile phone displays the main interface, such as the main interface UI1 shown in (a) of Figure 1 . The main interface UI1 includes icons of multiple application programs, such as the icon of the "Video" application program, the icon of the "Gallery" application program, the icon of the "Camera" application program, etc. When the user wants to capture an image, the user can click on the icon of the "Camera" application program. The mobile phone responds to this click operation and runs the camera application. At the same time, the display screen of the mobile phone jumps from the main interface UI1 to the shooting interface UI2.

[0107] As Figure 1 shown in (b) of

[0108] the shooting interface UI2, the shooting interface UI2 includes a preview area 102, a shooting control 103, a preview thumbnail 104, and multiple shooting mode options. Among them, the preview area 102 is used to display the preview image; the shooting control 103 is used to instruct the mobile phone to capture an image when a shooting operation (such as an operation of clicking on the shooting control 103) is received; the preview thumbnail 104 is used to display the thumbnail corresponding to the currently captured image.

[0108] The multiple shooting mode options may include: aperture mode, night scene mode, portrait mode, photo-taking mode, video recording mode, professional mode, etc. It can be understood that when the camera application is opened, it is generally default in the photo-taking mode.

[0109] Exemplarily, the light source of the shooting environment where the subject 101 is located is sunlight. When the user wants to shoot the subject 101, the user aims the camera of the mobile phone at the subject 101. The camera captures the image corresponding to the subject 101 and displays a preview image in the preview area 102. When the user is satisfied with the effect of the preview image, the user can click the shooting control 103. The mobile phone responds to the operation of the user clicking the shooting control 103, captures the preview image displayed in the current preview area 104, saves it as a photo after processing, and stores it in the photo gallery of the mobile phone.

[0110] When the user wants to view the captured image, the user can click on the icon of the "Photo Gallery" application. The mobile phone responds to this click operation and displays the captured image. Alternatively, the user can click on the preview thumbnail 104. The mobile phone responds to this click operation and quickly displays the currently captured image.

[0111] It should be noted that in this shooting scenario, the subject 101 happens to be between the light source and the camera, making the area where the subject 101 is located a dark area. At this time, the captured images obtained by using related technologies usually result in problems such as low clarity, insufficient details, smeariness, artifacts, and pseudo-textures in the face of the subject 101 in the captured image.

[0112] Please refer to Figure 2 , Figure 2 which is a schematic diagram showing a captured image displayed in an exemplary embodiment of the present application. As shown in (a) of Figure 2 , the display interface UI3 is the interface for displaying images in the photo gallery application. In the embodiment of the present application, the captured image 105 obtained by using related technologies is displayed in the display interface UI3. Obviously, there are problems such as low clarity (which can also be called blurriness), insufficient details, and artifacts in the face of the subject 101 in the captured image 105.

[0113] Through the image processing method provided by the embodiment of the present application, the trained inverse rendering model is used to extract the face normal image, face albedo image, and environment image corresponding to the captured image, thereby obtaining accurate face prior information, which lays a solid foundation for subsequent face enhancement processing. In the enhancement stage, when using the trained face enhancement model to perform enhancement processing on the face image, the face prior information can be used to accurately restore the face, effectively improve the face contour, and enhance the face details. Even under complex light and shadow conditions, unnatural textures can be avoided. Finally, through this image processing method, a high-quality captured image can be output. This high-quality captured image not only has high clarity and rich details, but also has no problems such as smeariness, artifacts, and pseudo-textures.

[0114] As shown in Figure 2 the display interface UI4 shown in (b) of

[0114] . The display interface UI4 is an interface for displaying images in a gallery application. In the embodiment of the present application, a captured image 106 obtained by using the image processing method provided by the embodiment of the present application is displayed in the display interface UI4. Obviously, compared with the captured image 105, the captured image 106 has higher quality. In the captured image 106, the face of the object 101 being photographed is not only clear and rich in details, but also does not have problems such as strong smearing, artifacts, and pseudo-textures.

[0115] Application scenario two: Video recording scenario

[0116] The image processing method provided by the embodiment of the present application can be applied to a video recording scenario. For example, it can be applied to recording videos in various scenarios.

[0117] Exemplarily, after the camera application is run, multiple shooting mode options are displayed in the shooting interface. The mobile phone responds to the user's sliding operation on the multiple shooting mode options and switches the shooting mode from the photo-taking mode to the video recording mode. In the preview state, the preview image collected by the mobile phone can be displayed in real time in the preview area. When the user is satisfied with the effect of the preview image, the user can click the shooting control. The mobile phone responds to the user's operation of clicking the shooting control, captures the preview image displayed in the current preview area in real time, and runs the program corresponding to the image processing method provided by the embodiment of the present application to obtain a captured video and stores it in the gallery of the mobile phone.

[0118] For example, when the object being photographed takes a self-portrait in a backlight environment, due to the strong surrounding light, the area where the object being photographed is located becomes a dark area. At this time, if processed using related technologies, problems such as low clarity, insufficient details, strong smearing, artifacts, and pseudo-textures may occur in the face of the object being photographed in the obtained video. If processed by the image processing method provided by the embodiment of the present application, the images in the captured video can be first split using an inverse rendering model to obtain accurate face prior information. In the enhancement stage, when using the face enhancement model to enhance the face image, the face prior information can be used to accurately restore the face, effectively improving the face contour and enhancing the face details. Even under complex light and shadow conditions, unnatural textures can be avoided. Finally, a high-quality video can be output through this image processing method. The high-quality video is not only clear and rich in details, but also does not have problems such as strong smearing, artifacts, and pseudo-textures.

[0119] Application scenario three: Video call scenario

[0120] The image processing method provided in the embodiments of this application can also be applied to video call scenarios. For example, when a user makes a video call in a backlight environment, due to the strong surrounding light, the area where the user is located becomes a dark area. At this time, if the related technology is used for processing, problems such as low clarity, insufficient details, strong smear feeling, artifacts, and pseudo-textures will occur in the face of the user captured by the electronic device during the video call. If the image processing method provided in the embodiments of this application is used for processing, high-quality video can be output.

[0121] Application Scenario 4: Image Editing / Image Beautification (or Photo Retouching) Scenario

[0122] The image processing method provided in the embodiments of this application can also be applied to image beautification scenarios. For example, the images stored in the gallery have problems such as low clarity, insufficient details, strong smear feeling, artifacts, and pseudo-textures. When the user wants to beautify the stored images (previously captured images, or images downloaded from the Internet, etc.), the user clicks on the controls in the display interface (such as automatic optimization control, intelligent photo retouching control, brightening control, etc.). The electronic device responds to the click operation and processes the image through the image processing method provided in the embodiments of this application, which can improve the image clarity, enrich the image details, and eliminate problems such as strong smear feeling, artifacts, and pseudo-textures in the image, and obtain high-quality images.

[0123] It should be understood that the above application scenarios of the image processing method are only for illustrative purposes and do not limit the actual application scenarios of this application. The image processing method provided in the embodiments of this application can also be applied but is not limited to the following scenarios: scenarios for enhancing images and / or videos in video surveillance, scenarios for enhancing preview images in various application programs, scenarios for enhancing images in video conferencing applications, scenarios for enhancing images in long and short video applications, scenarios for enhancing images in video live broadcast applications, scenarios for enhancing images in intelligent camera movement application scenarios, etc.

[0124] Next, in conjunction with the accompanying drawings of the specification, the image processing method provided in the embodiments of this application will be described in detail.

[0125] Please refer to Figure 3 , Figure 3 which is a schematic flowchart of an image processing method shown in an exemplary embodiment of this application. The method includes:

[0126] S101. Display a first interface, where the first interface includes a first control.

[0127] Exemplarily, the user can indicate the electronic device to run the camera application by clicking on the icon of the "Camera" application. Generally, after running the camera application, the main camera of the electronic device, i.e., the rear camera, is launched by default. Meanwhile, the display interface of the electronic device shows a first interface, such as the shooting interface UI2 shown in (b) of Figure 1 The first interface may include a preview area, a first control, a preview thumbnail, and various shooting mode options, etc.

[0128] Among them, the preview area can be used to display the preview image captured in real time by the rear camera; the first control can be a shooting control, which is used to indicate the electronic device to capture an image when a shooting operation (such as an operation of clicking on the first control) is received.

[0129] Optionally, the embodiments of the present application also provide other ways to run the camera application. For example, when the electronic device is in the locked screen state, the user can double-click the volume down key on the electronic device to indicate the electronic device to run the camera application. Or, when the electronic device is in the locked screen state, the locked screen interface includes an icon of the camera application, and the user can click on the icon of the camera application to indicate the electronic device to run the camera application. Or, when the electronic device is running other applications, other applications have the permission to call the camera application; the user can indicate the electronic device to run the camera application by clicking on the corresponding control in other applications. For example, when the electronic device is running an instant messaging application, the user can indicate the electronic device to run the camera application by selecting the control of the camera function, etc.

[0130] It should be understood that the above is an example description of the operation of running the camera application; the electronic device can also be indicated to run the camera application through voice instructions or other operations; the present application makes no limitation thereto. It should also be understood that running the camera application may refer to starting the camera application.

[0131] S102. In response to a first operation on the first control, obtain a face image.

[0132] Exemplarily, the first operation is used to indicate the start of shooting, and the first operation can be a click operation on the shooting control.

[0133] It can be understood that the above describes the first operation as a click operation by way of example; in the embodiments of the present application, the first operation can also be an operation of indicating shooting through voice instructions; or, the first operation can also be an operation of indicating shooting through face recognition; or, the first operation can also be an operation of indicating shooting through gesture recognition; or, the first operation can also be an operation of pressing a physical button (such as the volume button) to indicate shooting, etc.; the present application makes no limitation thereto.

[0134] The camera application starts shooting in response to various different types of first operations triggered by the user, that is, the electronic device uses the camera to collect the original image. Among them, the camera can be any one of the main camera, the front camera, the telephoto camera, the wide-angle camera, etc. It should be understood that the embodiments of the present application do not impose any restrictions on the types of cameras.

[0135] It should be noted that the main camera has the characteristics of large light input, high resolution, and moderate field of view angle. The main camera is generally used as the default camera of the electronic device. The telephoto camera has a longer focal length and a smaller field of view angle, and is suitable for shooting objects to be photographed that are far from the electronic device, that is, distant objects. The wide-angle camera has a shorter focal length and a larger field of view angle, and is suitable for shooting objects to be photographed that are close to the electronic device, that is, nearby objects.

[0136] In a possible implementation manner, when the original image is an image collected in real time by the camera of the electronic device, the original image can be an RGB domain image, which refers to an image located in the RGB domain color space. The original image can also be a Raw domain image, which refers to an image obtained in the Raw color space, that is, the Raw domain image refers to an image located in the Raw color space. It should be understood that the original image can also be other color images such as multi-spectral images, and the embodiments of the present application do not make any limitations on this.

[0137] In another possible implementation manner, the original image can also be an image pre-stored in the electronic device. For example, an image taken by the user and pre-stored in the gallery of the electronic device, or an image downloaded from the Internet, or an image received from another electronic device. The embodiments of the present application do not impose any restrictions on this.

[0138] The original image in the embodiments of the present application may include a human face. In a possible implementation manner, the original image including the human face can be directly used as the face image. In another possible implementation manner, the original image can be cropped to obtain the face image. For example, the face area in the original image is recognized, and this face area is cropped to retain the image containing the human face part to obtain the face image.

[0139] It is worth noting that the embodiments of the present application do not limit the number of human faces. For example, the original image or the face image may include one or more human faces.

[0140] S103. Process the face image by using the trained inverse rendering model to obtain the face normal image, the face albedo image, and the environment image.

[0141] The face normal image is used to represent the normal features corresponding to the portrait in the face image, or rather, to represent the normal direction of the face surface in the face image. The face normal image can be used to capture the 3D information of the face, and this 3D information is very important for subsequent analysis of the shape and structure of the face.

[0142] The face albedo image is used to represent the albedo features corresponding to the portrait in the face image, or rather, to represent features such as the color and texture of the face surface in the face image. The face albedo image is very important for subsequent accurate analysis of the inherent attributes of the face (such as skin color, spots, wrinkles, etc.).

[0143] The environment image is used to represent the environmental content other than the portrait, or rather, to represent the influence of the lighting in the shooting environment.

[0144] Among them, the normal feature can be understood as the direction of the normal line at each point on the concave and convex surface of the object. For example, in this application, the normal feature is the direction of the normal line at each point on the face surface. The face normal image can mark the direction of the normal line through the RGB color channels. For example, the RGB values of each pixel can be used to encode the three components (X, Y, Z) of the normal vector. Among them, the red channel can represent the X component of the normal vector, the green channel represents the Y component, and the blue channel represents the Z component. In this way, the RGB value of each pixel corresponds to a vector in three-dimensional space, indicating the normal direction of that point.

[0145] The albedo feature refers to the ratio of the light flux scattered in all directions by the fully illuminated part of the object surface to the light flux incident on the object surface. For example, for a face image, the albedo feature refers to the ratio of the light flux scattered in all directions by the fully illuminated part of the face skin to the light flux incident on the portrait skin. It is equivalent to that the albedo feature of the face can tell us how the face skin surface reflects light.

[0146] Exemplarily, a trained inverse rendering model can be used to process the face image, that is, to split the face and the environment in the face image by using the trained inverse rendering model to obtain the face normal image, the face albedo image, and the environment image. In the embodiments of this application, the face normal image, the face albedo image, and the environment image can be understood as the prior information corresponding to the face image. This prior information provides key information about the 3D structure, contour, color, texture, skin, environmental lighting, light and shadow, etc. of the face, and is crucial for subsequent restoration and processing of the face image.

[0147] S104. Use the trained face enhancement model to perform enhancement processing on the face image, the face normal image, the face albedo image, and the environment image to obtain the face target image.

[0148] Exemplarily, after splicing the face image, face normal image, face albedo image, and environmental image together, the spliced image is input into the trained face enhancement model for processing to obtain the face target image. Since the accurate prior information corresponding to the face image has been obtained through the inverse rendering model, such prior information provides the accurate contour of the face, rich texture, true color, clear 3D structure, and accurate environmental lighting conditions of the face, etc. Therefore, the trained face enhancement model can make full use of this prior information during the enhancement process, effectively restore the details of the face image, eliminate the smearing, artifacts, pseudo-textures, etc. existing in the face image, and improve the clarity and quality of the face image.

[0149] Among them, in particular, the face albedo image reflects the face skin information, does not generate unwanted textures, and is not affected by factors such as lighting changes and shadows. When the face enhancement model uses the face albedo image for enhancement processing, it can effectively enhance the details in the face image and improve the quality of the face image. This enhancement not only improves the visual clarity of the face image but also maintains the authenticity and naturalness of the face. Therefore, the enhanced face image not only has high clarity and rich details but also avoids problems such as strong smearing, artifacts, and pseudo-textures, bringing a better shooting experience to users.

[0150] Optionally, the clarity of the face target image is greater than the clarity of the face image.

[0151] Optionally, the face target image is rich in details and does not have problems such as strong smearing, artifacts, and pseudo-textures.

[0152] The image processing method provided by the embodiments of the present application splits the face and the environment in the face image through the trained inverse rendering model to obtain the face normal image, face albedo image, and environmental image, that is, obtain accurate prior information. The trained face enhancement model makes full use of this prior information during the enhancement process, can effectively restore the details of the face image, eliminate the smearing, artifacts, pseudo-textures, etc. existing in the face image, and improve the clarity and quality of the face image.

[0153] Please refer to Figure 4 , Figure 4 which is a schematic flowchart of a method for obtaining a face image shown in an exemplary embodiment of the present application. The method includes:

[0154] S1021. In response to a first operation on a first control, obtain an original image.

[0155] Exemplarily, the first operation may be a click operation on a shooting control, and the camera application responds to various different types of first operations triggered by the user and starts shooting, that is, the electronic device uses the camera to collect the original image.

[0156] S1022. Determine the clarity of the original image.

[0157] The clarity of the original image refers to the clarity of details in the image, which may include the sharpness of edges, the clarity of textures, etc. It should be understood that an original image with high clarity is rich in details and the face edges are obvious.

[0158] In one example, the edge information in the original image can be detected through an edge detection algorithm. The edge information may include the intensity and quantity of edges, and then the clarity of the original image can be evaluated by statistically analyzing the intensity and quantity of edges. Among them, the edge detection algorithm may include the Sobel operator algorithm, the Canny operator algorithm, etc.

[0159] In another example, the original image can be transformed into the frequency domain, for example, by performing a Fourier transform on the original image to transform it into the frequency domain, and then analyzing the high-frequency components in the transformed original image. It should be understood that the more high-frequency components there are, the higher the clarity of the original image; on the contrary, the fewer high-frequency components, the lower the clarity of the original image.

[0160] In yet another example, image quality evaluation metrics (such as sharpness index), edge contrast, etc. can be used to quantify the clarity of the original image.

[0161] In still another example, the value of the Normalized Root Mean Square Similarity (NRSS) corresponding to the original image can be determined.

[0162] NRSS is a reference-free image quality assessment method used to evaluate the quality of the original image. The NRSS value refers to the quality score of the original image calculated using the NRSS method. When the NRSS value is larger, it indicates that the quality of the original image is better, clearer, and less blurred; when the NRSS value is smaller, it indicates that the quality of the original image is worse, less clear, that is, more blurred.

[0163] S1023. When it is detected that the clarity of the original image is less than the preset clarity threshold, crop the original image to obtain a face image.

[0164] Exemplarily, if it is detected that the clarity of the original image is greater than or equal to the preset clarity threshold, it proves that the clarity of the original image is high, the face details in the original image are rich, and there are no problems such as strong smearing, artifacts, and pseudo-textures, so no enhancement processing is required. If it is detected that the clarity of the original image is less than the preset clarity threshold, it proves that the clarity of the original image is low, the face in the original image may lack details, and there may be problems such as strong smearing, artifacts, and pseudo-textures, and enhancement processing is required.

[0165] It can be understood that the size of the preset clarity threshold can be set and modified as needed, and the embodiments of the present application do not impose any restrictions on this.

[0166] Exemplarily, when it is detected that the clarity of the original image is less than the preset clarity threshold, the face area is located (or recognized) in the original image. For example, a face detection algorithm (such as a deep learning model, a face feature classifier, etc.) can be used to detect the face area in the original image. According to the detected face area, the bounding box of the face area is determined; the area within this bounding box is cropped from the original image to obtain a face image.

[0167] Please refer to Figure 5 , Figure 5 which is a schematic diagram of a face image shown in an exemplary embodiment of the present application. As Figure 5 shown in the original image, it is detected that the clarity of the original image is less than the preset clarity threshold, and the left original image is cropped to obtain the right face image. It should be noted that the oblique stripes in the original image and the face image are both used to represent the environment.

[0168] In this implementation manner, the original image is effectively screened by clarity, so that the determined face images are all face images that need to be enhanced. On the one hand, it is convenient to subsequently process only some face images that need to be enhanced, which can improve the image processing efficiency; on the other hand, it avoids enhancing face images that do not need to be enhanced, saving the resources of the electronic device and reducing power consumption; at the same time, it also avoids the situation of artifacts and pseudo-textures caused by enhancing face images that do not need to be enhanced.

[0169] Please refer to Figure 6 , Figure 6 which is a schematic flowchart of another image processing method shown in an exemplary embodiment of the present application. The method includes:

[0170] S201. Obtain a face image.

[0171] Exemplarily, reference can be made to the foregoing description of obtaining a face image, which will not be elaborated here.

[0172] It should be noted that the number of obtained face images can be one frame or multiple frames, and no limitation is imposed on this. In the embodiments of the present application, an example of obtaining one frame of face image is used for illustration.

[0173] S202. Process the face image using a trained inverse rendering model to obtain a face normal image, a face albedo image, and an environment image.

[0174] The trained inverse rendering model is used to split the face and the environment in a face image to obtain a face normal image, a face albedo image, and an environment image. The trained inverse rendering model is obtained by first training an initial inverse rendering network based on a first training set to obtain an inverse rendering model during training, and then training the inverse rendering model during training using a second training set. Among them, the first training set includes high-definition sample images, and the second training set includes low-definition sample images, and the low-definition sample images are obtained by degrading the high-definition sample images. The subsequent training process of the inverse rendering model will be described in detail and is omitted here for the time being.

[0175] The trained inverse rendering model may include a first network, a second network, and a third network. Optionally, the first network may be a Vector Quantized Variational Autoencoder (VQVAE), and / or, the second network may be a VQVAE.

[0176] Optionally, the third network may be a VQVAE or a U-Net.

[0177] Among them, VQVAE is a model that combines Variational Autoencoder (VAE) and Vector Quantized (VQ) technologies. Through the VQ technology, the continuous vector space can be mapped to a discrete codebook to achieve compression and discretization of image data. Specifically, VQ defines a fixed set of vectors as the codebook. For each input vector, VQ will find the closest vector in the codebook for mapping, thereby reducing the dimension of the image data and improving the calculation efficiency.

[0178] In the embodiments of the present application, the VQ technology is introduced into the inverse rendering model, that is, the VQ technology is introduced into each network included in the inverse rendering model. There are many benefits to doing so. First, VQ can extract features in the face image and quantize these features into a set of discrete codes, thereby reducing the dimension of the feature space and improving the calculation efficiency of the inverse rendering model. Second, through vector quantization, VQ helps to enhance the robustness of feature representation, thereby improving the performance of the inverse rendering model. In addition, the application of VQ in the inverse rendering module can improve the reconstruction and enhancement effects of face images, ensuring that the generated images are more natural and retain more details. Therefore, using the inverse rendering module to process face images not only improves the efficiency of image processing, but also can extract accurate prior information, which helps the subsequent face enhancement model to effectively use this prior information to enhance face images, avoiding the generation of pseudo-textures, artifacts, unnatural effects, etc., and greatly improving the quality of the images.

[0179] Taking the first network as VQVAE, the second network as VQVAE, and the third network as U-Net as an example, the inverse rendering model will be described in detail below.

[0180] Please refer to Figure 7 , Figure 7 which is a schematic diagram of the structure of the trained inverse rendering model shown in an exemplary embodiment of the present application. As Figure 7 shown, the inverse rendering model provided by the embodiment of the present application includes three interconnected networks, which are respectively referred to as the first network, the second network, and the third network.

[0181] Exemplarily, the first network is respectively connected to the second network and the third network, and the second network is connected to the third network. Among them, the first network includes an input end and an output end, the second network includes a first input end, a second input end, and an output end, and the third network includes a first input end, a second input end, a third input end, and an output end.

[0182] Exemplarily, the input end of the first network, the first input end of the second network, and the first input end of the third network all serve as the input end of the inverse rendering model; the output end of the first network and the second input end of the second network are respectively connected to the second input end of the third network, and the output end of the second network is connected to the third input end of the third network; the output end of the third network serves as the output end of the inverse rendering model, and the output ends of the first network and the second network also serve as the output ends of the inverse rendering model.

[0183] Based on Figure 7 the structure of the inverse rendering model shown, a face image is input at the input end of the first network, the first network processes the face image, and outputs a face normal image through the output end of the first network; a face image is input at the first input end of the second network, a face normal image is input at the second input end, the second network processes the face image and the face normal image, and outputs a face albedo image through the output end of the second network; a face image is input at the first input end of the third network, a face normal image is input at the second input end, a face albedo image is input at the third input end, the third network processes the face image, the face normal image, and the face albedo image, and outputs an environment image through the output end of the third network.

[0184] Optionally, when inputting a face image into the inverse rendering model, the input end of the first network, the first input end of the second network, and the first input end of the third network respectively receive the face image. The first network is used to determine the face normal image corresponding to the face image. Thus, the output end of the first network can respectively provide the face normal image to the second input end of the second network and the second input end of the third network. The second network is used to determine the face albedo image corresponding to the face image according to the face image and the face normal image. Thus, the output end of the second network can provide the face albedo image to the third input end of the third network. The third network is used to determine the environment image corresponding to the face image according to the face image, the face normal image, and the face albedo image. After that, the face normal image determined by the first network, the face albedo image determined by the second network, and the environment image determined by the third network will all be output as the output of the inverse rendering model.

[0185] It should be understood that the above is only a schematic structural diagram of an inverse rendering model, and the embodiments of the present application do not impose any restrictions on this.

[0186] Please refer to Figure 8 , Figure 8 is a schematic diagram of a face image, a face normal image, a face albedo image, and an environment image shown in an exemplary embodiment of the present application. Exemplarily, the inverse rendering model splits the face and the environment in the face image to obtain the face normal image, the face albedo image, and the environment image as shown in Figure 8 . It is worth noting that the dotted line in the face normal image, the thick oblique line in the face albedo image, and the oblique line in the environment image are only used to distinguish the face normal image, the face albedo image, and the environment image, and do not constitute a limitation on the face normal image, the face albedo image, and the environment image actually output by the inverse rendering model.

[0187] It should be understood that usually, the background in the face normal image actually output by the inverse rendering model is gray, and the face part has color changes to represent the direction of the face surface normal. Among them, red represents the component of the normal in the X-axis direction, green represents the component of the normal in the Y-axis direction, and blue represents the component of the normal in the Z-axis direction.

[0188] The background in the face albedo image actually output by the inverse rendering model is black, and the face part is clearly visible, accurately reflecting the face skin information, without unwanted textures, and not affected by factors such as illumination changes and shadows.

[0189] The environment image actually output by the inverse rendering model shows the environment where the photographed object is located and its illumination conditions at the shooting moment.

[0190] Next, in combination with Figure 8Each of the images shown, as well as the structure of the inverse rendering model, will be described in detail for the processing process of the inverse rendering model.

[0191] Please refer to Figure 9 , Figure 9 which is a schematic diagram of the inverse rendering model processing flow shown in an exemplary embodiment of the present application. The process includes:

[0192] S2021. Process the face image using the first network to obtain a face normal image.

[0193] In one example, extract a first latent vector from the face image using the first network; find a vector similar to the first latent vector in the first discrete vector set; generate a face normal image according to the vector similar to the first latent vector.

[0194] As Figure 9 shown, the first network may include a first encoder, a first vector codebook module, and a first decoder. The first encoder is used to map the input face image to a continuous latent space. It can be understood that the first network is used to extract the latent vector in the face image. For example, use the first encoder to extract features (such as edge features, contour features, geometric features, etc.) in the face image and encode these features into a continuous latent vector, that is, the first latent vector. Among them, the first latent vector may include 3D structure information and geometric information corresponding to the face.

[0195] The first vector codebook module is obtained by the inverse rendering model in training extracting the latent vectors in the face normal image corresponding to the high-definition sample images. The first vector codebook module may include a first discrete vector set. In other words, the first discrete vector set includes the latent vectors in the face normal images corresponding to multiple high-definition sample images. Through the first vector codebook module, the continuous first latent vector extracted by the first encoder can be quantized into a discrete latent vector.

[0196] Exemplarily, according to the first latent vector extracted by the first encoder, find a vector similar to the first latent vector in the first discrete vector set, or find the vector closest to the first latent vector in the first discrete vector set. For example, calculate the distance (such as Euclidean distance) between the first latent vector and each vector in the first discrete vector set, and determine the vector with the smallest distance as the vector similar to the first latent vector.

[0197] The first decoder is used to map the quantized latent vector (i.e., a vector close to the first latent vector found in the first set of discrete vectors) back to the facial image space. The purpose of this process is to use the quantized latent vector to reconstruct a facial image that is as close to the original as possible. For example, a vector close to the first latent vector is used as input to the first decoder. The first decoder maps the vector close to the first latent vector back to the facial image space. During this mapping process, the first decoder generates a facial normal image.

[0198] See also Figure 10 , Figure 10 This is a schematic diagram of a first network structure shown in an exemplary embodiment of the present application. Figure 10 As shown, the first network includes a first encoder, a first vector codebook module, and a first decoder. The first encoder is used to extract the first latent vector in the face image, and the first latent vector is input into the first vector codebook module. The first vector codebook module includes a first discrete vector set, also referred to as a first codebook in the embodiment of the present application. The first discrete vector set is composed of latent vectors in the face normal image corresponding to multiple high-definition sample images. Figure 10 They are represented by rectangular boxes of different forms and labeled with 0, 1...N.

[0199] The first vector codebook module may further include a first matching module (Matching Block), which compares the first potential vector with each vector in the first discrete vector set through Matching Block, and determines the vector in the first discrete vector set that is closest to the first potential vector, such as Figure 10 The first vector codebook module generates quantized latent vectors based on these vectors, and outputs them to the first decoder. The first decoder then maps the quantized latent vectors back to the face image space, generating a normal face image during the mapping process.

[0200] In this implementation, vector quantization via VQVAE effectively maps information from a continuous latent space into a discrete space, improving the computational efficiency and generalization capabilities of the first network. This allows the first network to extract rich 3D structural and geometric information from the input facial image, laying the foundation for subsequent facial image enhancement. This helps more accurately restore the 3D structural and geometric features of the face during subsequent facial image enhancement.

[0201] Optionally, in another example, the first network can be a CNN. Edge features, contour features, geometric features, etc. in the face image are extracted through the CNN; the depth information of each point on the face surface is estimated through the CNN; the normal vector of each point on the face surface is calculated according to the estimated depth information; the calculated normal vector is converted into RGB channel values for visualization in the image; these RGB channel values are combined into a face normal image through the CNN.

[0202] S2022. Process the face image and the face normal image using the second network to obtain a face albedo image.

[0203] In one example, the face image and the face normal image are spliced to obtain a first spliced image; the second latent vector in the first spliced image is extracted using the second network; a vector similar to the second latent vector is searched for in the second discrete vector set; a face albedo image is generated according to the vector similar to the second latent vector.

[0204] As Figure 9 shown, the second network can include a second encoder, a second vector codebook module, and a second decoder. The face image and the face normal image are used as the input of the second network. Specifically, after splicing the face image and the face normal image, the first spliced image is input. For example, the RGB channels of the face image are spliced with the RGB channels of the face normal image to obtain a new six-channel image, that is, the first spliced image.

[0205] After that, the first spliced image is input into the second encoder. The second encoder is used to map the input first spliced image to a continuous latent space. It can be understood that the second network is used to extract the latent vector in the first spliced image. For example, features in the first spliced image (such as texture features, skin features, etc.) are extracted using the second encoder and encoded into a continuous latent vector, that is, the second latent vector. Among them, the second latent vector can include the texture information and skin information corresponding to the face.

[0206] The second vector codebook module is obtained by the inverse rendering model in the training process extracting the latent vectors in the face albedo images corresponding to the high-definition sample images. The second vector codebook module can include a second discrete vector set. In other words, the second discrete vector set includes the latent vectors in the face albedo images corresponding to multiple high-definition sample images. The continuous second latent vector extracted by the second encoder can be quantized into a discrete latent vector through the second vector codebook module.

[0207] Exemplarily, according to the second latent vector extracted by the second encoder, search for a vector similar to the second latent vector in the second discrete vector set, or search for the vector closest to the second latent vector in the second discrete vector set. For example, calculate the distance (such as Euclidean distance) between the second latent vector and each vector in the second discrete vector set, and determine the vector with the minimum distance as the vector similar to the second latent vector.

[0208] The second decoder is used to map the quantized vector (i.e., the vector similar to the second latent vector found in the second discrete vector set) back to the face image space. Exemplarily, the vector similar to the second latent vector is used as the input of the second decoder, and the second decoder maps the vector similar to the second latent vector back to the face image space. During the mapping process, the second decoder generates a face albedo image.

[0209] Please refer to Figure 11 , Figure 11 which is a schematic diagram of the second network structure shown in an exemplary embodiment of the present application. As Figure 11 shown, the second network includes a second encoder, a second vector codebook module, and a second decoder. The second encoder is used to extract the second latent vector in the first spliced image and input the second latent vector into the second vector codebook module. The second vector codebook module includes a second discrete vector set, which is also referred to as the second codebook (Codebook) in the embodiments of the present application. The second discrete vector set is composed of latent vectors in the face albedo images corresponding to multiple high-definition sample images, Figure 11 which are represented by rectangular frames in different forms and are identified by 0, 1... N.

[0210] The second vector codebook module may further include a second matching module (Matching Block). The second Matching Block compares the second latent vector with each vector in the second discrete vector set to determine the vector closest to the second latent vector in the second discrete vector set. For example, the vectors 4, 8, 7, 3, 5, 4, 2, 0, 5, etc. in the nine-square grid shown in Figure 11 are such vectors. The second vector codebook module obtains the quantized latent vector based on these vectors and outputs the quantized latent vector to the second decoder. For example, the vector closest to the second latent vector is used to replace the original second latent vector to obtain the quantized latent vector. Then, the second decoder maps the quantized latent vector back to the face image space, and a face albedo image is generated during the mapping process.

[0211] In this implementation method, vector quantization is performed through VQVAE, which can effectively map the information in the continuous latent space to a discrete space, helping to improve the computational efficiency and generalization ability of the second network. In this way, the second network can extract accurate texture information and skin information from the input face image and face normal image, laying a foundation for subsequent enhancement processing of the face image. That is, when performing subsequent enhancement processing on the face image, it helps to more accurately restore the texture features and skin features of the face, enhance the details and realism of the face image, and avoid generating pseudo-textures, artifacts, unnatural effects, etc.

[0212] Optionally, in another example, the second network can be a U-Net. The U-Net gradually extracts texture features, skin features, etc. in the first spliced image through multiple convolutional layers; in the encoder part of the U-Net, downsampling is performed through the pooling layer to reduce the size of the first spliced image, increase the receptive field, and obtain more local information of the face; in the decoder part of the U-Net, the spatial dimension of the image is gradually restored through upsampling, and in each upsampling step, the features in the encoder are combined with the features in the decoder through skip connections, which can retain more detailed information; in the final stage of the decoder, the multi-channel feature map output by the decoder is converted into the final face albedo image.

[0213] S2023. Process the face image, face normal image, and face albedo image using a third network to obtain an environmental image.

[0214] In one example, the third network can be a U-Net. The face image, face normal image, and face albedo image serve as the input to the third network. Specifically, the face image, face normal image, and face albedo image are spliced to obtain a target spliced image. For example, the RGB channels of the face image, the RGB channels of the face normal image, and the RGB channels of the face albedo image are spliced to obtain a new nine-channel image, which is the target spliced image.

[0215] The U-Net gradually extracts environmental illumination features, light and shadow features, etc. in the target spliced image through multiple convolutional layers. In the encoder part of the U-Net, downsampling is performed through the pooling layer to reduce the size of the target spliced image, increase the receptive field, and obtain more environmental illumination information and light and shadow information; in the decoder part of the U-Net, the spatial dimension of the image is gradually restored through upsampling, and in each upsampling step, the features in the encoder are combined with the features in the decoder through skip connections, which can retain more environmental illumination information and light and shadow information; in the final stage of the decoder, the multi-channel feature map output by the decoder is converted into the final environmental image.

[0216] In another example, the third network can be a VQVAE. The process of processing the face image, face normal image, and face albedo image through the VQVAE to obtain the environmental image can refer to the description in step S2022 above and will not be elaborated here.

[0217] S203. Use the trained face enhancement model to perform enhancement processing on the face image, face normal image, face albedo image, and environmental image to obtain the face target image.

[0218] The trained face enhancement model is obtained by first training the initial face enhancement network based on the third training set to obtain the face enhancement model during training, and then training the face enhancement model during training using the fourth training set. Among them, the third training set includes high-definition sample images, and the fourth training set includes low-definition sample images, as well as the corresponding normal sample images, albedo sample images, and environmental sample images of the low-definition sample images; the normal sample images and albedo sample images are obtained by processing the low-definition sample images through the trained inverse rendering model; the environmental sample images are simulated through the trained light and shadow enhancement model. The subsequent training process of the inverse rendering model will be described in detail and is omitted here for the time being.

[0219] The face enhancement model can be a VQVAE or a U-Net. In this embodiment of the application, the case where the face enhancement model is a VQVAE is taken as an example for illustration.

[0220] It should be understood that introducing the VQ technology into the face enhancement model has many benefits. First, VQ can extract the features in the face image, face normal image, face albedo image, and environmental image, and quantize these features into a set of discrete codes, thereby reducing the dimension of the feature space and improving the computational efficiency of the face enhancement model. Second, through vector quantization, VQ helps to enhance the robustness of the feature representation, thereby improving the performance of the face enhancement model. In addition, the application of VQ in the face enhancement model can improve the reconstruction and enhancement effects of the face image, ensure that the enhanced image is more natural, and retain more details. Therefore, using the face enhancement model to perform enhancement processing on the face image not only improves the efficiency of image processing but also effectively utilizes the accurate prior information extracted by the inverse rendering model, avoiding the generation of pseudo-textures, artifacts, unnatural effects, etc., and greatly improving the quality of the image.

[0221] Please refer to Figure 12 , Figure 12 which is a schematic diagram of the face enhancement model processing flow shown in an exemplary embodiment of the present application. The process includes:

[0222] S2031. Stitch the face image, face normal image, face albedo image, and environmental image to obtain the second stitched image.

[0223] Exemplarily, a face image, a face normal image, a face albedo image, and an environment image are used as inputs to the face enhancement model. Specifically, the face image, the face normal image, the face albedo image, and the environment image are concatenated, and after obtaining the second concatenated image, it is input. For example, the RGB channels of the face image, the RGB channels of the face normal image, the RGB channels of the face albedo image, and the RGB channels of the environment image are concatenated to obtain a new twelve-channel image, that is, the second concatenated image. Then, the second concatenated image is input into the face enhancement model for processing.

[0224] S2032. Use the trained face enhancement model to extract the third latent vector from the second concatenated image.

[0225] Exemplarily, the trained face enhancement model may include a third encoder, a third vector codebook module, and a third decoder. The second concatenated image is input into the third encoder, and the third encoder is used to map the input second concatenated image to a continuous latent space. It can be understood that the third network is used to extract the latent vector in the second concatenated image. For example, use the third encoder to extract features in the second concatenated image (such as edge features, contour features, geometric features, texture features, skin features, environmental lighting features, light and shadow features, etc.), and encode these features into a continuous latent vector, that is, the third latent vector. Among them, the third latent vector may include 3D structure information, geometric information, texture information, skin information, environmental lighting information, light and shadow information, etc. corresponding to the face.

[0226] S2033. Search for a vector similar to the third latent vector in the third discrete vector set.

[0227] The third vector codebook module is obtained by the trained face enhancement model extracting the latent vector corresponding to the face in the high-definition sample image during the training of the face enhancement model. The third vector codebook module may include a third discrete vector set. In other words, the third discrete vector set includes the latent vectors corresponding to the faces in multiple high-definition sample images. Through the third vector codebook module, the continuous third latent vector extracted by the third encoder can be quantized into a discrete latent vector.

[0228] Exemplarily, according to the third latent vector extracted by the third encoder, search for a vector similar to the third latent vector in the third discrete vector set, or search for the vector closest to the third latent vector in the third discrete vector set. For example, calculate the distance (such as Euclidean distance) between the third latent vector and each vector in the third discrete vector set, and determine the vector with the smallest distance as the vector similar to the third latent vector.

[0229] S2034. Generate a face target image based on the vector similar to the third latent vector.

[0230] The third decoder is used to map the quantized vector (i.e., the vector similar to the third latent vector found in the third discrete vector set) back to the face image space. Exemplarily, the vector similar to the third latent vector is used as the input of the third decoder, and the third decoder maps the vector similar to the third latent vector back to the face image space. During the mapping process, the third decoder generates a face target image.

[0231] Please refer to Figure 13 , Figure 13 which is a schematic structural diagram of the face enhancement model shown in an exemplary embodiment of the present application. As Figure 13 shown, the third network includes a third encoder, a third vector codebook module, and a third decoder. The third encoder is used to extract the third latent vector in the second spliced image and input the third latent vector into the third vector codebook module. The third vector codebook module includes a third discrete vector set, which is also referred to as the third codebook (Codebook) in the embodiments of the present application. The third discrete vector set is composed of the latent vectors corresponding to the faces in multiple high-definition sample images, Figure 13 which are represented by rectangular frames in different forms and are labeled with 0, 1... N.

[0232] The third vector codebook module may further include a third matching module (Matching Block). The third latent vector is compared with each vector in the third discrete vector set through the third Matching Block, and the vector closest to the third latent vector is determined in the third discrete vector set. For example, Figure 13 the vectors 2, 6, 5, 1, 3, 8, 0, 7, 3, etc. in the nine-square grid shown; the third vector codebook module obtains the quantized latent vector based on these vectors and outputs the quantized latent vector to the third decoder. Then, the third decoder maps the quantized latent vector back to the face image space, and during the mapping process, a face target image is generated.

[0233] In this implementation, the face enhancement model can obtain accurate 3D structure information, geometric information, texture information, skin information, environmental illumination information, light and shadow information, etc. from the prior information. Therefore, even in the case of poor face image quality and complex face light and shadow, it can correctly guide the restoration of the 3D structure, geometry, texture, skin, environmental illumination, light and shadow, etc. of the face image based on this information, effectively avoiding the generation of pseudo-textures and artifacts, improving the quality of the image, and also improving the robustness of the face enhancement model, making the applicability of the face enhancement model more extensive.

[0234] S204. Stitch the original image and the face target image to obtain the high-definition image corresponding to the original image.

[0235] Exemplarily, a pre-prepared mask can be used to stitch the original image and the face target image. The mask is a binary image used to indicate which areas in the original image should be replaced by the face target image. It should be understood that in the mask, generally the white part represents the area of the face target image, and the black part represents the area that should be retained in the original image. The mask is used to process the face target image. For example, the mask can be used to perform a bitwise AND operation on the face target image to ensure that the pixels in the white part of the mask are retained.

[0236] The processed face target image is fused with the original image, for example, it can be achieved through a bitwise OR operation. Among them, the original image is used as the basis, and the corresponding part of the face target image is combined through the mask. After fusion, the obtained image is the high-definition image corresponding to the original image.

[0237] Optionally, after fusing the processed face target image with the original image, some detailed adjustments can also be made to the stitching area, such as color balance, edge balance, etc., to ensure that the stitching effect is natural.

[0238] In this implementation method, the enhancement process is specifically targeted at face images. On the one hand, it can effectively restore the details of the face area, eliminate the smear feeling, artifacts, pseudo-textures, etc. existing in the face image, and ensure that the image quality of the face area is significantly improved. On the other hand, by separately enhancing the face image and then stitching the original image and the face target image to obtain a high-definition image, the image processing efficiency can be effectively improved, and the resource consumption of the electronic device can be reduced. In addition, users usually pay more attention to the face in the captured image. The image processing method provided in this application can significantly improve the image quality of the face area while increasing the image processing speed, which can greatly enhance the user's favorability.

[0239] Optionally, in a possible implementation method, the image processing method provided in the embodiments of this application can also use a trained light and shadow enhancement model to determine the fill light environment image corresponding to the environmental image; use a trained face enhancement model to enhance the face image, face normal image, face albedo image, and fill light environment image to obtain the face target image.

[0240] Among them, the trained light and shadow enhancement model can be trained based on the U-Net model. This light and shadow enhancement model can simulate the light and shadow conditions of a human face in various lighting environments. Therefore, this light and shadow enhancement model can be used to fill light for the environmental image to generate a filled light environmental image. For example, perform human pose estimation on the original image to determine the human pose. Among them, the human pose refers to the angular information of the face orientation, which can be represented by a rotation matrix, a rotation vector, a quaternion, Euler angles (Yaw, Roll, Pitch), etc. Then, combine the human pose and the position of the light source in the environment shown in the environmental image to determine the light filling position in the environmental image; fill light for the environmental image at the light filling position to obtain a filled light environmental image.

[0241] Optionally, in a possible implementation manner, a new environmental image can be used to replace the original environmental image, and the new environmental image is determined as the filled light environmental image.

[0242] After that, splice the face image, the face normal image, the face albedo image, and the filled light environmental image to obtain a third spliced image; use the trained face enhancement model to extract the fourth latent vector from the third spliced image; search for a vector similar to the fourth latent vector in the third discrete vector set; generate a face target image according to the vector similar to the fourth latent vector. The specific implementation process can refer to the description in the foregoing steps S2031 to S2034, and will not be elaborated here.

[0243] In this implementation manner, the light and shadow enhancement model is used to intelligently fill light for the environmental image, and then the face enhancement model is used to finely adjust the face image based on the filled light environmental image. Such processing not only enhances the face details, but also achieves a comprehensive light filling effect for the face and the surrounding environment, ensuring the balance of the face and environmental lighting, which is beneficial to generating a face target image with higher clarity and stronger contrast.

[0244] Optionally, in a possible implementation manner, after determining the filled light environmental image, the trained light and shadow enhancement model can also be used to render the face normal image, the face albedo image, and the filled light environmental image to generate a filled light face image. It should be understood that according to the light filling situation in the filled light environmental image, the normal in the face normal image, and the face albedo image, the shadow generated by the face in this filled light environment can be calculated and thus shown in the rendered filled light face image. For example, if the light filling is on the right side, the brightness on the right side of the face increases and the left side is relatively darker, then the shadow projected by the left side of the face in the environment can be shown in the filled light face image.

[0245] It is worth noting that the rendering in the embodiments of the present application refers to the process of converting three-dimensional light energy transfer processing into a two-dimensional image.

[0246] After that, the trained face enhancement model is used to enhance the fill-light face image, face normal image, face albedo image, and fill-light environment image to obtain the face target image.

[0247] In this implementation manner, the trained light and shadow enhancement model can achieve fill light, without the need for an additional fill light device of the electronic device, nor does the user need to manually select the fill light area during the shooting process, improving the user experience. Moreover, whether during the shooting process or after shooting, the face image can be filled with light through this method, with high flexibility. In addition, compared with the traditional method of using an external light source or manually selecting the fill light area, in this application, the light and shadow enhancement model is used to render the face normal image, face albedo image, and fill-light environment image to generate the fill-light face image, ensuring that the fill-light face image more conforms to the lighting conditions of the actual environment and making the fill-light effect more natural and real.

[0248] It should be noted that before using the inverse rendering model, the inverse rendering model also needs to be trained. The following will describe in detail the training process of the inverse rendering model provided in the embodiments of the present application with reference to the accompanying drawings.

[0249] Please refer to Figure 14 , Figure 14 which is a schematic flowchart of training the inverse rendering model shown in an exemplary embodiment of the present application, as follows.

[0250] S301. Obtain the first training set.

[0251] The first training set includes high-definition sample images, which contain faces, have high clarity, rich details, and no quality problems such as smearing, artifacts, and pseudo-textures.

[0252] The high-definition sample images can be sourced from various channels, such as pre-collecting high-definition images containing faces through the camera of an electronic device, downloading high-definition images containing faces from the Internet, or receiving high-definition images containing faces transmitted from other electronic devices, etc.

[0253] It should be noted that the number of high-definition sample images will affect the training effect and training speed of the inverse rendering model. In the embodiments of the present application, the number of high-definition sample images can be reasonably set to achieve the best balance between accuracy and training speed, and the specific number is not limited.

[0254] S302. Use the first training set to train the initial inverse rendering network to obtain the inverse rendering model during training.

[0255] Exemplarily, the high-definition sample images in the first training set are cropped to obtain high-definition face images, and the high-definition face images are input into the initial inverse rendering network. The initial inverse rendering network splits the face and the environment in the high-definition face images to obtain a high-definition face normal image, a high-definition face albedo image, and a high-definition environment image.

[0256] The structure of the initial inverse rendering network is the same as that of the trained inverse rendering model. Therefore, the process of the initial inverse rendering network processing the high-definition face images can refer to the process of using the trained inverse rendering model to process the face images in the previous text, which will not be elaborated here.

[0257] During the process of training the initial inverse rendering network using the first training set, the initial inverse rendering network can learn the ability to accurately extract the face normal image, the face albedo image, and the environment image from the high-definition sample images. Moreover, by first training the initial inverse rendering network with high-definition sample images, the vector codebook module in the initial inverse rendering network can learn the prior information corresponding to the high-definition sample images, such as the latent vectors corresponding to the high-definition face normal image, the latent vectors corresponding to the high-definition face albedo image, and the latent vectors corresponding to the high-definition environment image. After that, the vector codebook module in the initial inverse rendering network is fixed to obtain the inverse rendering model during training.

[0258] Optionally, the training data can also be enriched, that is, the lighting scenarios of the high-definition sample images are enriched. Since the lighting scenarios of the high-definition sample images are usually relatively simple and the light and shadow on the human face are not complex enough, the trained inverse rendering model may not be able to accurately process various light and shadow changes when facing real and complex lighting scenarios, resulting in problems such as artifacts and pseudo-textures. Therefore, in the embodiments of the present application, a shadow enhancement technique can also be introduced during the process of retraining the inverse rendering model. For example, the color, brightness, etc. of specific areas of the face can be adjusted to generate more high-definition sample images containing complex light and shadow effects.

[0259] Optionally, the high-definition sample images can also be enriched using a light and shadow enhancement model. For example, a large number of various lighting environment images are collected in advance from multiple channels; the original high-definition face images are processed by the inverse rendering model during training to obtain a high-definition face normal image and a high-definition face albedo image; the high-definition face normal image, the high-definition face albedo image, and various lighting environment images are rendered by the light and shadow enhancement model to simulate the light and shadow conditions of the face under various lighting environments, thereby generating high-definition face images under various lighting environments. After that, the high-definition face images under various lighting environments are used to continue training the inverse rendering model during training, so that the inverse rendering model during training can learn how to accurately process various light and shadow changes when dealing with complex lighting scenarios, and avoid the ability to generate artifacts and pseudo-textures, improving the robustness and adaptability of the inverse rendering model.

[0260] Please refer to Figure 15 , Figure 15 which is a schematic diagram of enriching high-definition sample images by a light and shadow enhancement model shown in an exemplary embodiment of the present application. Exemplarily, by processing the original high-definition face image through an inverse rendering model during training, a high-definition face normal image and a high-definition face albedo image can be obtained; select any light environment map (represented by a grid in Figure 15 ), and render the high-definition face normal image, the high-definition face albedo image, and the any light environment map through the light and shadow enhancement model to generate a high-definition face image under this light environment as shown in

[0261] In this implementation manner, using the light and shadow enhancement model to enrich high-definition sample images greatly ensures the light and shadow richness of the training data, improves the light and shadow complexity of the training data, helps to enhance the robustness and adaptability of the inverse rendering model, facilitates the subsequent trained inverse rendering model to accurately process various light and shadow changes when facing real and complex light scenes, avoids problems such as artifacts and pseudo-textures, and effectively improves the image quality.

[0262] S303. Obtain a second training set.

[0263] The second training set includes low-definition sample images, and the low-definition sample images are obtained by degrading high-definition sample images. Exemplarily, the generation process of these low-definition sample images can be implemented through a pre-trained degradation model, and this degradation model can simulate various degradation effects on high-definition sample images. For example, use the pre-trained degradation model to process high-definition sample images, and add at least one degradation effect such as noise, blur, smear, artifacts, and pseudo-textures to the high-definition sample images to obtain low-definition sample images.

[0264] S304. Train the inverse rendering model under training using the second training set to obtain a trained inverse rendering model.

[0265] Exemplarily, crop the low-definition sample images in the second training set to obtain low-definition face images, input the low-definition face images into the inverse rendering model under training, and the inverse rendering model under training splits the face and the environment in the low-definition face images to obtain a real face normal image, a real face albedo image, and a real environment image.

[0266] The structure of the inverse rendering model under training is the same as that of the trained inverse rendering model. Therefore, the process of the inverse rendering model under training processing the low-definition face images can refer to the process of using the trained inverse rendering model to process face images in the previous text, which will not be elaborated here.

[0267] Since the vector codebook module of the inverse rendering model in training stores the prior information of high-definition sample images, during this stage when processing low-definition sample images, the prior information of the pre-stored high-definition sample images can be effectively utilized to assist the inverse rendering model in training to recover and output high-quality real face normal images, real face albedo images, and real environment images. In this process, the inverse rendering model in training learns the ability to output high-quality images based on the prior information of high-definition sample images when faced with low-definition sample images. When the inverse rendering model in training masters this ability, the vector codebook module at this time is fixed to obtain the trained inverse rendering model.

[0268] Generally speaking, a human face has strong prior information. For example, the facial features of a human face are very regular. The inverse rendering model in training learns these regularities (i.e., the prior information mentioned above) in high-definition face images. When processing low-definition face images, it can infer what the blurred areas in the low-definition face images should look like based on the learned regularities, so as to recover and output high-quality images.

[0269] In this implementation method, when training the inverse rendering model, it is divided into two stages. In the first stage, the initial inverse rendering network is trained with high-definition sample images so that the vector codebook module can capture and store the prior information of high-definition sample images; in the second stage, further training is carried out with low-definition sample images so that the inverse rendering model in training learns the ability to accurately utilize the prior information to recover and output high-quality images. Through the training of these two stages, the trained inverse rendering model has strong robustness, so that when faced with face images under various complex light and shadow scenarios, it can also accurately extract and utilize the prior information to generate high-quality images.

[0270] It is worth noting that before using the face enhancement model, the face enhancement model also needs to be trained. The following will describe in detail the training process of the face enhancement model provided by the embodiments of the present application with reference to the accompanying drawings.

[0271] Please refer to Figure 16 , Figure 16 which is a schematic flowchart of training a face enhancement model shown in an exemplary embodiment of the present application, as follows.

[0272] S401. Obtain a third training set.

[0273] The third training set may include high-definition sample images. It is worth noting that the high-definition sample images included in the third training set may be the same as or different from the high-definition sample images included in the first training set. In the case of being different, multiple high-definition images containing human faces can be collected through various channels to obtain the third training set.

[0274] S402. Use the third training set to train the initial face enhancement network to obtain a face enhancement model during training.

[0275] Exemplarily, crop the high-definition sample images in the third training set to obtain high-definition face images; input the high-definition face images into the trained inverse rendering model, and the inverse rendering model splits the face and the environment in the high-definition face images to obtain high-definition face normal images, high-definition face albedo images, and high-definition environment images. Use the initial face enhancement network to process the high-definition face images, high-definition face normal images, high-definition face albedo images, and high-definition environment images to obtain high-definition face target images.

[0276] The structure of the initial face enhancement network is the same as that of the trained face enhancement model. Therefore, for the specific processing process of the initial face enhancement network, reference can be made to the specific processing process of the previously trained face enhancement model, which will not be elaborated here.

[0277] During the process of training the initial face enhancement network with the third training set, the vector codebook module in the initial face enhancement network can learn how to quantify vectors so as to accurately use prior information to restore and output high-quality face images. After the vector codebook module in the initial face enhancement network has this ability, fix the vector codebook module in the initial face enhancement network to obtain a face enhancement model during training.

[0278] In this implementation, when training the face enhancement model, it is divided into two stages. In the first stage, train the initial face enhancement network with high-definition sample images, and the vector codebook module in the initial face enhancement network can learn how to quantify vectors so as to accurately use prior information to restore and output high-quality face images. In the second stage, further train with low-definition sample images so that the face enhancement model during training learns the ability to accurately use prior information to restore and output high-quality images. Through the training of these two stages, the trained face enhancement model has strong robustness, so that when facing face images in various complex light and shadow scenes, it can also accurately use prior information to generate high-quality images.

[0279] Optionally, the light and shadow scenes of the high-definition sample images in the third training set can also be enriched. For example, adjust the color, brightness, etc. of specific areas of the face through shadow enhancement technology to generate more high-definition sample images with complex light and shadow effects; or use a light and shadow enhancement model to enrich the high-definition sample images.

[0280] In this implementation method, a light and shadow enhancement model is used to enrich high-definition sample images, which greatly ensures the richness of light and shadow in the training data, improves the light and shadow complexity of the training data, helps to enhance the robustness and usability of the face enhancement model, and facilitates the trained face enhancement model to accurately process various light and shadow changes when facing images with poor quality and complex face light and shadow, avoiding problems such as artifacts and pseudo-textures, and effectively improving the image quality.

[0281] S403. Obtain a fourth training set.

[0282] The fourth training set includes low-definition sample images. It should be noted that the low-definition sample images included in the fourth training set can be the same as or different from the low-definition sample images included in the second training set.

[0283] The fourth training set may also include normal sample images, albedo sample images, and environment sample images corresponding to the low-definition sample images. Among them, the normal sample images and albedo sample images are obtained by processing the low-definition sample images through a trained inverse rendering model; the environment sample images can be obtained by processing the low-definition sample images through a trained inverse rendering model, or can be simulated through a trained light and shadow enhancement model. When the environment sample images are simulated through a trained light and shadow enhancement model, it can enrich the light and shadow complexity of the training data of the face enhancement model and help to enhance the robustness and usability of the face enhancement model.

[0284] S404. Use the fourth training set to train the face enhancement model during training to obtain a trained face enhancement model.

[0285] Exemplarily, crop the low-definition sample images in the fourth training set to obtain low-definition face images; splice the low-definition face images, normal sample images, albedo sample images, and environment sample images to obtain a fourth spliced image; use the trained face enhancement model to extract the fifth latent vector in the fourth spliced image; find the vector similar to the fifth latent vector in the vector codebook module of the face enhancement model during training; generate a high-definition face image according to the vector similar to the fifth latent vector. In the embodiments of the present application, the high-definition face image output by the face enhancement model during training is called a predicted image or a reconstructed image.

[0286] The structure of the face enhancement model during training is the same as that of the trained face enhancement model. Therefore, for the specific processing process of the face enhancement model during training, reference can be made to the specific processing process of the trained face enhancement model described above, and details are not repeated here.

[0287] When the target loss value corresponding to the face enhancement model during training converges, the trained face enhancement model is obtained. Exemplarily, during the process of training the face enhancement model, the target loss value is calculated according to a preset loss function; according to the target loss value, the face enhancement model during training is continuously trained in a backpropagation manner to obtain the trained face enhancement model.

[0288] Exemplarily, the target loss value is used to represent the loss between the high-definition sample image and the predicted image output by the face enhancement model during training. This loss consists of three parts. The first part is the L1 loss, the second part is the perceptual loss, and the third part is the adversarial loss.

[0289] Among them, the L1 loss is used to represent the loss between the high-definition sample image and the predicted image (or called the reconstructed image); the perceptual loss is used to represent the feature loss between the high-definition sample image and the predicted image (or called the reconstructed image); the adversarial loss is used to represent the loss between the real image and the image generated by the generator in the face enhancement model.

[0290] The preset loss function is as follows:

[0291] , (1)

[0292] , (2)

[0293] In the above formulas (1) and (2), represents the high-definition sample image, represents the low-definition sample image, represents the reconstructed image generated by passing through the face enhancement network , represents the L1 loss, represents the perceptual loss, represents the adversarial loss, , , represent the weights corresponding to the three losses respectively. Among them, the face enhancement network refers to the network used by the face enhancement model during training.

[0294] The face enhancement network is implemented as follows:

[0295] , (3)

[0296] In the above formula (3), represents the encoder of the face enhancement network, represents the vector codebook module of the face enhancement network, represents the decoder of the face enhancement network, represents the prior information.

[0297] 、 as well as The calculation formula is as follows:

[0298]

[0299]

[0300] , (4)

[0301] In the above formula (4), represents the discriminator network.

[0302] For example, calculate 、 、 The target loss value is obtained by summing the values of the face enhancement model in training and adjusting the target loss value. 、 、 When the target loss value converges, the network parameters of the face enhancement network are fixed to obtain the trained face enhancement model.

[0303] For example, during the training of a face enhancement model, the target loss value gradually decreases and stabilizes. In other words, as the number of training times increases, the target loss value no longer decreases significantly, but instead fluctuates within a smaller range. This indicates that the face enhancement network in training has learned how to accurately use prior information to restore and output high-quality facial images, which means that the face enhancement model has been trained.

[0304] In an embodiment of the present application, when it is detected that the target loss value is greater than a preset loss threshold, the training process of the face enhancement model continues; when it is detected that the target loss value is less than or equal to the preset loss threshold, the training is stopped, and the network parameters of the face enhancement network are saved to obtain a trained face enhancement model.

[0305] In this implementation, the back-propagation algorithm is used to adjust the model parameters, which helps the model converge to the optimal solution faster and improves the training speed of the face enhancement model. In addition, the losses of the three parts of the face enhancement model are taken into account when adjusting the model parameters, which can improve the performance of the face enhancement model in many aspects and help the face enhancement model cope with various low-quality images with complex lighting and shadows.

[0306] The structure of the electronic device involved in the embodiments of the present application is briefly introduced below with reference to the accompanying drawings.

[0307] In the embodiments of the present application, the electronic device may be a device equipped with a touch screen such as a mobile phone, a smart screen, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a handheld or laptop device, a media player, a smart projector, a smart TV, a desktop computer, a vehicle infotainment system, etc. The embodiments of the present application do not impose any restrictions on the specific type and form of the electronic device.

[0308] It should be understood that the software system, hardware system, device, and chip in the embodiments of the present application can all execute the training methods and image processing methods of various models in the foregoing embodiments of the present application. That is, the specific working processes of the following various products can refer to the corresponding processes in the foregoing method embodiments.

[0309] Please refer to Figure 17 , Figure 17 is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0310] It should be noted that Figure 17 the structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more than Figure 17more or fewer components than those shown, or the electronic device 100 may include Figure 17 a combination of some of the components shown, or the electronic device 100 may include Figure 17 sub-components of some of the components shown. Figure 17 The components shown may be implemented by hardware, software, or a combination of hardware and software.

[0311] The processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). Among them, different processing units may be independent devices or integrated devices.

[0312] Among them, the controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0313] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0314] The processor 110 may run the software code of the image processing method provided in the embodiments of the present application, process the face image through the inverse rendering model, and use the processing result of the inverse rendering model as the face prior information. The face enhancement model can effectively utilize the face prior information to accurately restore the face. Even under complex light and shadow conditions, it can avoid generating unnatural textures and output high-quality captured images.

[0315] The electronic device 100 can implement the display function through the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU can also be used to perform mathematical and pose calculations for graphics rendering, etc. The processor 110 may include one or more GPUs, and the execution of program instructions by them can generate or change the display information.

[0316] In the embodiments of the present application, the ability of the electronic device 100 to display various different display interfaces depends on the display functions provided by the above-mentioned GPU, the display screen 194, and the application processor. For example, displaying preview images, captured images, captured videos, processed images, processed videos, etc.

[0317] In some embodiments, the electronic device 100 may include 1 or N display screens 194, and N may be a positive integer greater than 1.

[0318] The display screen 194 in the embodiments of the present application is a touch screen. A touch sensor 180K may be integrated in the display screen 194. The touch sensor 180K may also be referred to as a "touch control panel". That is to say, the display screen 194 may include a display panel and a touch panel, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. After the touch operation detected by the touch sensor 180K, it can be transmitted to the upper layer by the driver of the kernel layer (such as the TP driver) to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 at a different position from that of the display screen 194.

[0319] The charging management module 140 is used to receive a charging input from a charger. The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The wireless communication function of the electronic device 100 can be implemented through antenna 1, antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0320] The mobile communication module 150 can provide solutions for wireless communications such as 2G / 3G / 4G / 5G applied to the electronic device 100.

[0321] The camera 193 is used to capture images. It can be triggered to turn on through application instructions to implement the photographing function, such as taking images of any scene. The camera may include components such as an imaging lens, a filter, and an image sensor. The light emitted or reflected by an object enters the imaging lens, passes through the filter, and finally converges on the image sensor. The image sensor is mainly used to converge and form an image of the light emitted or reflected by all objects (also called the photographed object) in the photographing perspective; the filter is mainly used to filter out the excess light waves in the light (such as light waves other than visible light, such as infrared); the image sensor is mainly used to perform photoelectric conversion on the received optical signal, convert it into an electrical signal, and input it into the processor 130 for subsequent processing. Among them, the camera 193 can be located in front of the electronic device 100 or on the back of the electronic device 100. The specific number and arrangement of the cameras can be set according to requirements, and this application does not make any restrictions.

[0322] Exemplarily, the electronic device 100 includes a front camera and a rear camera. For example, either the front camera or the rear camera can include one or more cameras. Taking the electronic device 100 having one rear camera as an example, in this way, when the electronic device 100 starts one rear camera for shooting, the image processing method provided by the embodiments of this application can be used. Or, the camera is arranged on an external accessory of the electronic device 100, and the external accessory is rotatably connected to the frame of the mobile phone. The angle formed between the external accessory and the display screen 194 of the electronic device 100 is any angle between 0 and 360 degrees. For example, when the electronic device 100 takes a selfie, the external accessory drives the camera to rotate to a position facing the user. Of course, when the mobile phone has multiple cameras, only some of the cameras can be arranged on the external accessory, and the remaining cameras can be arranged on the body of the electronic device 100. The embodiments of this application do not impose any restrictions on this.

[0323] The internal memory 121 can be used to store computer-executable program codes, and the executable program codes include instructions. The internal memory 121 can include a program storage area and a data storage area. The internal memory 121 can also store the software code of the image processing method provided by the embodiments of this application. When the processor 110 runs the software code, it executes the process steps of the image processing method to obtain a high-quality captured image. This high-quality captured image not only has high clarity and rich details, but also has no problems such as smearing, artifacts, and false textures. The internal memory 121 can also store the captured images.

[0324] Of course, the software code of the image processing method provided in the embodiments of the present application can also be stored in an external memory, and the processor 110 can run the software code through the external memory interface 120, execute the process steps of the image processing method, and obtain a high-quality captured image. The image captured by the electronic device 100 can also be stored in the external memory.

[0325] In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0326] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement functions such as unlocking, accessing an application lock, taking pictures, and answering incoming calls.

[0327] The keys 190 include a power-on key and volume keys. The keys 190 can be mechanical keys or touch keys. The electronic device 100 can receive key input signals and implement functions related to the key input signals.

[0328] In addition, various types of operating systems are running on the above components. For example, Android system, IOS operating system, Symbian operating system, BlackBerry operating system, Linux operating system, Windows operating system, etc. This is only an exemplary illustration and is not limited thereto. Different application programs, such as any type of input method application, can be installed and run on these operating systems.

[0329] The image processing method provided in the embodiments of the present application can be implemented in the electronic device 100 having the above hardware structure.

[0330] The structure of the electronic device 100 involved in the embodiments of the present application has been briefly introduced above. Next, the software structure involved in the embodiments of the present application will be briefly introduced. Please refer to Figure 18 , Figure 18 which is a software structure block diagram of an electronic device shown in an exemplary embodiment of the present application.

[0331] The software system can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the embodiments of the present application, the layered architecture is taken as an example to exemplarily describe the software system of the electronic device 100.

[0332] As Figure 18As shown, the layered architecture divides software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, taking the electronic device 100 with an Android system as an example, the Android system is divided into five layers, from top to bottom are the application layer 210, the application framework layer 220, the hardware abstraction layer 230, the driver layer 240, and the hardware layer 250.

[0333] The application layer 210 may include a camera, a gallery application, and may also include applications such as a calendar, a call, a map, a navigation, a WLAN, a Bluetooth, music, a video, a short message, etc.

[0334] The application framework layer 220 provides application access interfaces and programming frameworks for the applications in the application layer 210.

[0335] For example, the application framework layer 220 includes a camera access interface, and this camera access interface is used to provide the shooting service of the camera through camera management and camera devices.

[0336] The camera management in the application framework layer 220 is used to manage the camera. The camera management can obtain the parameters of the camera, such as judging the working state of the camera, etc.

[0337] The camera device in the application framework layer 220 is used to provide a data access interface between different camera devices and camera management.

[0338] The hardware abstraction layer 230 is used to abstract the hardware. For example, the hardware abstraction layer 230 may include a camera hardware abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer may include camera device 1, camera device 2, etc.; the camera hardware abstraction layer can be connected to the camera algorithm library, and the camera hardware abstraction layer can call the algorithms in the camera algorithm library.

[0339] The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include a camera driver; a digital signal processor driver and a graphics processor driver.

[0340] The hardware layer 250 may include sensors, an image signal processor, a digital signal processor, a graphics processor, and other hardware devices. Among them, the sensors may include sensor 1, sensor 2, etc., and may also include a depth sensor (time of flight, TOF) and a multispectral sensor.

[0341] Next, in combination with the shooting scenario, the working process of the software system of the electronic device 100 will be exemplarily described.

[0342] When the user performs a single - click operation on the touch sensor 180K, after the camera APP is awakened by the single - click operation, it calls each camera device of the camera hardware abstraction layer through the camera access interface. The camera hardware abstraction layer can send instructions to call the camera to the camera device driver, and at the same time, the camera algorithm library starts to load the algorithms used in the embodiments of the present application.

[0343] After the sensor in the hardware layer is called, for example, after the sensor 1 in the camera is called to obtain the original image, the original image is sent to the image signal processing for preliminary processing such as registration. After processing, it is returned to the hardware abstraction layer through the camera device driver, and then processed using the algorithms in the loaded camera algorithm library. For example, using the trained inverse rendering model and the trained face enhancement model, it is processed according to the relevant processing steps provided in the embodiments of the present application to obtain the face target image. Among them, the trained inverse rendering model and the trained face enhancement model can be processed by driving the digital signal processor through the digital signal processor driver and driving the graphics processor through the graphics processor driver.

[0344] The obtained face target image and the original image are spliced to obtain the high - definition image corresponding to the original image, and then sent back to the camera application for display and storage through the camera hardware abstraction layer and the camera access interface.

[0345] Please refer to Figure 19 , Figure 19 which is a schematic diagram of an image processing device shown in an exemplary embodiment of the present application.

[0346] It should be understood that the image processing device 300 can execute the image processing method provided by the present application; the image processing device 300 includes: a display unit 310, an acquisition unit 320, a first processing unit 330, and a second processing unit 340.

[0347] Optionally, the inverse rendering model, the face enhancement model, and the light and shadow enhancement model can be deployed in the image processing device 300.

[0348] The display unit 310 is used to display a first interface, and the first interface includes a first control;

[0349] The acquisition unit 320 is used to obtain a face image in response to a first operation on the first control;

[0350] The first processing unit 330 is used to process the face image using the trained inverse rendering model to obtain a face normal image, a face albedo image, and an environment image;

[0351] A second processing unit 340, configured to perform enhancement processing on a face image, a face normal image, a face albedo image, and an environmental image by using a trained face enhancement model to obtain a face target image; the clarity of the face target image is greater than that of the face image.

[0352] It should be noted that the above image processing device 300 is embodied in the form of functional units. The term "unit" here can be implemented in the form of software and / or hardware, and no specific limitation is made thereto.

[0353] For example, the "unit" can be a software program, a hardware circuit, or a combination of the two that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group of processors, etc.) for executing one or more software or firmware programs, and a memory, a merged logic circuit, and / or other suitable components that support the described functions.

[0354] Therefore, the units of the examples described in the embodiments of the present application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0355] The embodiments of the present application further provide a computer-readable storage medium, in which computer instructions are stored; when the computer-readable storage medium runs on an image processing device, the image processing device is caused to execute the image processing method of any of the above embodiments. The computer instructions can be stored in the computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, etc. that includes one or more integrated media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0356] The embodiments of the present application also provide a computer program product containing computer instructions. When it runs on an image processing device, the image processing device can execute the image processing method of any of the foregoing embodiments.

[0357] The embodiments of the present application also provide a chip. Please refer to Figure 20 , Figure 20 which is a schematic structural diagram of a chip provided by the embodiments of the present application. Figure 20 The chip shown can be a general-purpose processor or a dedicated processor. The chip includes a processor 410. Among them, the processor 410 is used to execute the image processing method of any of the foregoing embodiments.

[0358] Optionally, the chip further includes a transceiver 420, which is used to accept the control of the processor and support the image processing device to execute the foregoing technical solutions.

[0359] Optionally, Figure 20 the chip shown may further include: a storage medium 430.

[0360] It should be noted that Figure 20 the chip shown can be implemented using the following circuits or devices: one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout the present application.

[0361] Among them, the electronic device, computer-readable storage medium, computer program product, or chip provided in this embodiment is all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be elaborated here.

[0362] Through the description of the above embodiments, those skilled in the art can understand that for the convenience and simplicity of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0363] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be in electrical, mechanical or other forms.

[0364] The units described as separate components may or may not be physically separated. The components displayed as units may be one physical unit or multiple physical units, that is, they can be located in one place, or they can be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0365] In addition, each functional unit in various embodiments of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0366] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods in various embodiments of this application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read only memory (ROM), random access memory (RAM), magnetic disks or optical discs that can store program codes.

[0367] The above content is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in this application should be covered by the protection scope of this application. Therefore, the protection scope of this application should be within the protection scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: Displaying a first interface, where the first interface includes a first control; In response to a first operation on the first control, acquiring a face image; Using a first network included in a trained inverse rendering model to extract the structural features and contour features of the face in the face image, encoding the structural features and the contour features into a first latent vector; finding a vector similar to the first latent vector in a first discrete vector set included in the first network; generating a face normal image according to the vector similar to the first latent vector; Stitching the face image and the face normal image to obtain a first stitched image; using a second network included in the inverse rendering model to extract the texture features and skin features of the face in the first stitched image, encoding the texture features and the skin features into a second latent vector; finding a vector similar to the second latent vector in a second discrete vector set included in the second network; generating a face albedo image according to the vector similar to the second latent vector; Using a third network included in the inverse rendering model to process the face image, the face normal image, and the face albedo image to obtain an environment image; Determining face prior information according to the face normal image, the face albedo image, and the environment image; the face prior information includes face structure information, face contour information, face light and shadow information, face skin information, and face texture information; Using a trained face enhancement model to perform enhancement processing on the face image, the face normal image, the face albedo image, and the environment image, and using the face prior information to restore the face details in the face image to obtain a face target image; the face details include face structure, face contour, face light and shadow, face skin, and face texture; the clarity of the face target image is greater than the clarity of the face image.

2. The method according to claim 1, wherein The acquiring a face image in response to a first operation on the first control includes: In response to a first operation on the first control, acquiring an original image: Determining the clarity of the original image; Detecting that the clarity of the original image is less than a preset clarity threshold, and cropping the original image to obtain the face image.

3. The method according to claim 1, wherein The trained face enhancement model includes a third vector codebook module, and the third vector codebook module includes a third discrete vector set. The using a trained face enhancement model to perform enhancement processing on the face image, the face normal image, the face albedo image, and the environment image to obtain a face target image includes: Stitching the face image, the face normal image, the face albedo image, and the environment image to obtain a second stitched image; Using the trained face enhancement model to extract a third latent vector from the second stitched image; Finding a vector similar to the third latent vector in the third discrete vector set; Generating the face target image according to the vector similar to the third latent vector.

4. The method according to claim 1, wherein Performing enhancement processing on the face image, the face normal image, the face albedo image, and the environment image by using the trained face enhancement model to obtain a face target image, including: Determining a fill light environment image corresponding to the environment image by using the trained light and shadow enhancement model; Performing enhancement processing on the face image, the face normal image, the face albedo image, and the fill light environment image by using the trained face enhancement model to obtain a face target image.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Training an initial inverse rendering network by using a first training set to obtain an inverse rendering model during training; the first training set includes high-definition sample images; Training the inverse rendering model during training by using a second training set to obtain the trained inverse rendering model; the second training set includes low-definition sample images, and the low-definition sample images are degraded from the high-definition sample images.

6. The method according to claim 5, wherein The method further includes: Training an initial face enhancement network by using a third training set to obtain a face enhancement model during training; the third training set includes the high-definition sample images; Training the face enhancement model during training by using a fourth training set to obtain the trained face enhancement model; the fourth training set includes the low-definition sample images, and the normal sample images, albedo sample images, and environment sample images corresponding to the low-definition sample images; the normal sample images and the albedo sample images are obtained by processing the low-definition sample images by using the trained inverse rendering model; the environment sample images are simulated by using the trained light and shadow enhancement model.

7. The method according to claim 5, characterized in that The method further includes: During the process of training the face enhancement model, calculating a target loss value according to a preset loss function; the target loss value is used to represent the loss between the high-definition sample image and the predicted image output by the face enhancement model during training; Continuing to train the face enhancement model during training in a backpropagation manner according to the target loss value to obtain the trained face enhancement model.

8. The method according to claim 1, wherein The first network is a VQVAE, and / or the second network is a VQVAE.

9. The method according to claim 2, characterized in that, The method further includes: Stitching the original image and the face target image to obtain a high-definition image corresponding to the original image.

10. An electronic device, characterized in that, The electronic device includes: one or more processors, and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions, when the instructions are run on an electronic device, enabling the electronic device to execute the method according to any one of claims 1 to 9.

12. A computer program product, characterized in that, The computer program product includes computer program code, when the computer program code is run, enabling the method according to any one of claims 1 to 9 to be executed.

Images