An image processing method, a model training method and related devices

Abstract

The application provides an image processing method, a model training method and related devices, which are applied to the field of image processing. The method comprises the following steps: obtaining an original image and first parameter information, wherein the original image comprises a first face, and the first parameter information comprises parameters for editing the first face in the original image; updating a first machine learning model based on the original image to obtain an optimized second machine learning model; and editing the first face in the original image by using the first parameter information through the second machine learning model to output a first target image. The application can finely modify the face image and retain the personalized features of the face, so as to improve the accuracy and recognition of the face editing in the original image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, a model training method, and related apparatus. Background Technology

[0002] Facial image editing technology is a crucial field in image processing, widely used in social media, beauty cameras, and film production. It allows for parametric editing and enhancement of facial features, such as adjusting specific areas like the eyes, mouth, and hair, to achieve natural-looking changes. Furthermore, it can address specific needs such as adjusting expressions and poses, style transitions, age changes, and gender transformations.

[0003] However, editing and modifying specific facial features in an image often has uncontrollable effects on other facial features. For example, adjusting the eyes in an image can affect other facial features such as eyebrows and ears. Changes in expression or posture can make it difficult to maintain facial coherence among multiple facial features, resulting in unnatural distortions. Furthermore, each person's facial features are highly individualized, and the editing process needs to preserve the unique details of the face in the image to avoid generating generic or distorted faces.

[0004] Therefore, how to refine facial images while preserving personalized features is an urgent problem to be solved. Summary of the Invention

[0005] This application provides an image processing method, a model training method, and related apparatus for updating a first machine learning model based on an original image to obtain a second machine learning model, so as to better preserve the semantic information of the face in the original image. The second machine learning model uses parameter information to modify the face in the original image and outputs a target image, thereby realizing the parameterized modification of the face in the original image and preserving personalized features, so as to improve the accuracy and recognition of face editing in the original image.

[0006] In a first aspect, this application provides an image processing method, in which an original image and first parameter information are obtained, the original image including a first face, and the first parameter information including parameters for editing the first face in the original image; based on the original image, a first machine learning model is updated to obtain an optimized second machine learning model, the second machine learning model being used to preserve the semantic information of the first face in the original image; the first face in the original image is edited using the first parameter information through the second machine learning model, and a first target image is output.

[0007] The original image includes a first face, which can be a facial image including the first face, or a full-body image or a partial body image including the first face.

[0008] The first parameter information includes parameters for editing the first face in the original image, or it can be understood as the first parameter information that can be used to edit the facial features of the first face. Facial features can be local features of the first face such as the nose, mouth, eyes, ears, skin, and hair, or global features such as face shape and skin color. For example, the first parameter information can include the shape, size, or position of the eyes; the shape, density, and position of the eyebrows; and the overall outline of the face.

[0009] Optionally, the first parameter information includes the first face parameters, the first camera parameters, and the first lighting parameters of the first face. The first face parameters can be parameters that are edited based on the facial features of the first face, such as the first shape parameters, the first pose parameters, and the first expression parameters of the first face.

[0010] Semantic information can be abstract information related to the first face, used to describe its personalized features. For example, semantic information can include the first face's identity information, adornment information, environmental information, gender information, age information, etc. Identity information, also known as identifier information, can be used to uniquely identify the first face's biometric identity; for example, it can be represented using a unique string. Adornment information can be information related to the first face's accessories, such as earrings, glasses, hats, etc. Environmental information can be information related to background elements of the original image, such as buildings, natural landscapes, etc. In subsequent processing steps, machine learning models can combine semantic information to determine the semantic latent variables of the first face.

[0011] The first machine learning model is updated using the original image to obtain a second machine learning model, which better preserves the semantic information of the first face in the original image. Alternatively, the original image can be used to train the first machine learning model, ensuring its generalization performance on the first face. It can also be understood as using the first machine learning model to apply semantic priors to the original image. By updating the first machine learning model with the original image, the updated second machine learning model can understand and utilize the semantic information of the first face without losing or distorting it. This enhances the accuracy and recognizability of the second machine learning model in subsequent editing of the first face. By using the first parameter information to edit the first face through the second machine learning model, the edited target image can be generated using the edited parameter information and the semantic information of the first face.

[0012] In this application, updating the first machine learning model using the original image ensures its generalization performance on faces, allowing the updated second machine learning model to better preserve the semantic information of the face. Different faces possess different semantic information, which can be used to describe the personalized features of a face. This can be understood as the second machine learning model better preserving the personalized features of different faces, thereby improving face recognition accuracy. The second machine learning model uses the first parameter information to edit the facial features. By parameterizing and controlling the facial features, fine-grained modifications to the original image are achieved, improving the accuracy of editing the first face.

[0013] In one possible implementation, the first machine learning model includes a first image generation network and a first face control network. The first face control network is used to determine a first multi-scale conditional feature, which describes the facial feature information of the first face in the original image. The first image generation network is used to generate a second target image, which is obtained by restoring the first face in the original image. The second machine learning model includes a second image generation network and a second face control network. The second face control network is used to determine a second multi-scale conditional feature, which describes the facial feature information of the edited first face. The second image generation network is used to generate a first target image, which is obtained by editing the first face in the original image.

[0014] Understandably, the first machine learning model is used for semantic prior calculations on the original image, or for training on the original image, or for generalizing the representation of the original image based on the first machine learning model. Therefore, the output of the first machine learning model should be as consistent as possible with the original image. The purpose of the first image generation network is to reconstruct the first face, making the face in the output second target image as consistent as possible with the first face, thereby improving face recognition accuracy. The second machine learning model is used to edit the first face in the original image based on the first parameter information. Therefore, the purpose of the second image generation network is to output the first target image after editing the first face.

[0015] In this implementation, a first multi-scale conditional feature is determined by a first face control network to describe the facial feature information of the first face in the original image. This enables the first image generation network to reconstruct the first face based on the first multi-scale conditional feature. In other words, the first image generation network can reconstruct the first face based on the facial feature information and output the second target image. A second multi-scale conditional feature is determined by a second face control network to describe the edited facial feature information of the first face. This enables the second image generation network to generate the edited first target image based on the edited facial feature information.

[0016] In one possible implementation, the first image generation network includes a first semantic encoding network and a first conditional diffusion decoding network. The first semantic encoding network is used to determine a first semantic latent variable, which is used to describe the semantic information of a first face in the original image. The first conditional diffusion decoding network is used to output a second target image. The second image generation network includes a second semantic encoding network and a second conditional diffusion decoding network. The second semantic encoding network is used to determine a second semantic latent variable, which is used to describe the semantic information of the edited first face. The second conditional diffusion decoding network is used to output the first target image.

[0017] Semantic coding networks are a type of neural network structure used to convert the attribute information of input images or text into feature representations with semantic information, such as semantic latent variables. Semantic latent variables can be represented in the form of matrices, vectors, etc. For more information on semantic information, please refer to the previous section, which will not be repeated here.

[0018] Optionally, text attribute information is input into a semantic coding network, which outputs semantic latent variables represented by the text attribute information. The generation effect of the target image is controlled by the text attribute information.

[0019] Conditional diffusion decoding networks are network structures based on diffusion models. Through a back-diffusion process, they use semantic latent variables and / or multi-scale conditional features to output target images under specific conditions. Alternatively, they can be understood as controlling facial feature information and / or semantic information to output target images under specific conditions.

[0020] Alternatively, the conditional diffusion decoding network can generate the target image using a noisy image.

[0021] In this implementation, the first image generation network is used to determine the semantic information of the first face. The first image generation network can output a second target image after restoring the first face based on the semantic information and facial feature information of the first face. The second image generation network is used to determine the semantic information of the first face. The second image generation network can output a first target image after editing the first face based on the semantic information and the edited facial feature information.

[0022] In one possible implementation, the original image is input into a first machine learning model, which outputs a second target image. The model parameters of the first face control network in the first machine learning model are kept unchanged, and the model parameters of the first image generation network in the first machine learning model are updated until the difference between the original image and the second target image satisfies a first condition, thereby obtaining an optimized second machine learning model.

[0023] In this implementation, the original image is used to train the first machine learning model, resulting in a trained model. During training, the model parameters of the first face control network are kept unchanged, while the model parameters of the first image generation network are updated, thus decoupling the first face control network and the first image generation network and improving update efficiency. Furthermore, the first image generation network can control the first semantic latent variable and the first multi-scale conditional feature, thereby better preserving the semantic and facial feature information of the face.

[0024] In one possible implementation, based on the first parameter information, the first face in the original image is edited to obtain an intermediate image of the first face; the intermediate image of the first face is input into the second face control network in the second machine learning model to obtain the second multi-scale conditional features; the mask image and the second multi-scale conditional features are input into the second image generation network in the second machine learning model to output the first target image.

[0025] In this implementation, parametric editing of the first face is achieved using first parameter information to improve the accuracy of face editing. The facial feature information of the first face is controlled by second multi-scale conditional features to control the effect of facial feature editing on the first face.

[0026] In one possible implementation, the mask image is obtained by masking the original image. The masking rate of the mask image is negatively correlated with the number of masking steps and negatively correlated with the amount of semantic information in the original image.

[0027] In this implementation, as the number of mask image generation steps gradually increases, the mask ratio gradually decreases, and the amount of semantic information in the original image gradually increases, thereby gradually improving the accuracy and recognizability of faces in the target image. The mask image can specify the image region to be edited, meaning the model can focus on editing specific parts, thus improving the accuracy of editing and the quality of generation.

[0028] In one possible implementation, the original image is input into the face estimation network of a second machine learning model to obtain second parameter information, which is the parameter information of facial features before editing the first face; based on the first and second parameter information, the first face in the original image is edited to obtain the displacement map, surface normal map, and illumination geometry map of the first face; the displacement map, surface normal map, and illumination geometry map of the first face are superimposed to obtain an intermediate image of the first face.

[0029] In this implementation, a face estimation network is used to obtain relevant parameters (second parameter information) of the facial features of the first face in the original image. Using the first and second parameter information, the facial feature information after editing the first face can be determined to achieve a specific editing effect for that face. Displacement mapping can be used to describe the displacement and height changes of facial features, preserving facial details. Surface normal mapping can simulate the surface details and lighting effects of the face, thereby enhancing visual realism. Lighting geometry mapping can be used to describe the lighting conditions of the face. Combining and rendering the displacement mapping, surface normal mapping, and lighting geometry mapping ensures that the intermediate image retains the detailed information of the edited face.

[0030] In one possible implementation, the first parameter information includes first face parameters, the second parameter information includes second face parameters and a personal feature vector, the displacement map is obtained based on the first face parameters, the second face parameters, and the personal feature vector, the surface normal map is obtained based on the first parameter information, the second parameter information, the face parameterization model, the normal calculation function, and the reflection renderer, and the lighting geometry map is used to describe the lighting conditions of the first face. The personal feature vector can be a vector representation of the facial feature information of the face.

[0031] In one possible implementation, the masked image is input into the second semantic coding network of the second image generation network to obtain the second semantic latent variable; the noisy image, the second semantic latent variable, and the second multi-scale conditional features are input into the second conditional diffusion decoding network of the second image generation network to output the first target image.

[0032] In this implementation, the second image generation network uses a second semantic latent variable and a second multi-scale conditional feature to determine the personalized features and facial features of the face in the output first target image, thereby improving the recognition of the face and achieving specific editing effects on the face.

[0033] In one possible implementation, the noisy image is a random Gaussian noise image with a standard normal distribution.

[0034] In one possible implementation, in response to an operation on the original image, first parameter information is obtained, the first parameter information including first face parameters of a first face, first camera parameters and first illumination parameters, the first face parameters including first shape parameters, first pose parameters and first expression parameters.

[0035] In this implementation, the parameters corresponding to the user-edited control can be obtained to acquire the first parameter information. This control can be a button, slider, text box, etc. For example, the first parameter information can be obtained by dragging the pose parameter button, expression parameter button, lighting parameter button, etc. Another example is obtaining the parameters from the user-input text box to get the first parameter information. Through parametric editing, explicit and refined editing of the facial features of the first face is possible.

[0036] In one possible implementation, in response to user input, textual attribute information of a first face is obtained, which is used to describe the semantic information of the first face, and / or to determine a second semantic latent variable of the first face.

[0037] In this implementation, text attribute information from user input text boxes and other controls can be obtained, such as "get younger" or "get longer hair," allowing the model to generate corresponding effects based on this text attribute information. The text attribute information can be input into a semantic coding network, which outputs semantic latent variables represented by the text attribute information, thus controlling the generation effect of the target image through the text attribute information.

[0038] In one possible implementation, the original image is processed using third parameter information by a second machine learning model to output a third target image. The third parameter information includes the parameter information of the facial features of the second face. The third target image is obtained by replacing the first face in the original image with the second face.

[0039] In this implementation, a second machine learning model can be used to achieve the effect of "replacing the first face in the original image with the second face".

[0040] In one possible implementation, a second machine learning model uses second parameter information to process a reference video and output a target video. The reference video includes a third face, and the target video is obtained by replacing the third face in the reference video with the first face.

[0041] In this implementation, a second machine learning model can be used to achieve the effect of "comparing the first face in the original image to the video-driven comparison".

[0042] In one possible implementation, the face control network is the Exp-FaceNet network, the image generation network is the Diff-AE network, the face estimation network is the EMOCA network, and the face parameterization model is the FLAME model.

[0043] In this implementation, the Exp-FaceNet network can be used with the first parameter information to control explicit editing of the face, enabling more precise adjustment of facial feature details, enhancing the controllability and visualization of the editing effect, and meeting diverse editing needs. Using Diff-AE as the image generation network can better preserve the semantic information of the original image, improving the recognizability and realism of the generated target image. The EMOCA network can capture the parameter information of the first face, better capturing facial expressions.

[0044] Secondly, this application provides a model training method, in which at least one original sample image and at least one first sample target image are obtained, the original sample image includes a fourth face, and the first sample target image is the image that the sample machine learning model expects to output for the original sample image; using at least one original sample image and at least one first sample target image, the sample machine learning model is trained to obtain a first machine learning model, and the sample machine learning model is used to reconstruct the fourth face based on the facial feature information and semantic information of the fourth face in the original sample image.

[0045] The original sample image includes a fourth face. For example, the original sample image can be a facial image including a fourth face, or it can be a full-body image or a partial body image including a fourth face. The specific details are not limited here.

[0046] At least one original sample image and at least one first sample target image can be data used to train a sample machine learning model.

[0047] The first sample target image is the image that the sample machine learning model expects to output based on the original sample image. It can be understood that for any original sample image in the input sample machine learning model, there is a corresponding expected output first sample target image.

[0048] Optionally, the first sample target image and the original sample image can be the same or different; no specific limitation is made here. Specifically, when the first sample target image and the original sample image are the same, the first machine learning model can be used to reconstruct the face. When the first sample target image and the original sample image are different, the first machine learning model can achieve face reconstruction while simultaneously achieving different effects of face editing.

[0049] In this application, at least one original sample image and at least one first sample target image are used to train a sample machine learning model. Since the sample machine learning model is used to reconstruct the fourth face based on the facial feature information and semantic information of the fourth face in the original sample image, the trained first machine learning model can reconstruct the face based on the facial feature information and semantic information of the face in the input image.

[0050] In one possible implementation, the sample machine learning model includes a sample image generation network and a sample face control network. The sample face control network is used to determine the multi-scale conditional features of the samples. The multi-scale conditional features of the samples are used to describe the facial feature information of the fourth face in the original sample image. The sample image generation network is used to generate the second sample target image.

[0051] In this implementation, the multi-scale conditional features of the sample are determined by the sample face control network to describe the facial feature information of the fourth face in the original image, so that the sample image generation network can reconstruct the fourth face based on the facial feature information of the fourth face and output the second sample target image.

[0052] In one possible implementation, the sample image generation network includes a sample semantic encoding network and a sample conditional diffusion decoding network. The sample semantic encoding network is used to determine sample semantic latent variables, which are used to describe the semantic information of the fourth face in the original sample image. The sample conditional diffusion decoding network is used to output the second sample target image.

[0053] In this implementation, the ability of the sample image generation network to determine the semantic information of the fourth face is trained, and the ability of the sample image generation network to output the second sample target image based on the semantic information is also trained, so that the sample image generation network can better reconstruct the fourth face based on the semantic information.

[0054] In one possible implementation, at least one original sample image is input into a sample machine learning model, which outputs at least one second sample target image. The model parameters of the sample image generation network in the sample machine learning model are kept unchanged, and the model parameters of the sample face control network in the sample machine learning model are updated until the difference between at least one second sample target image and at least one first sample target image satisfies the second condition, thereby obtaining a trained first machine learning model.

[0055] In this implementation, when updating the sample machine learning model, the model parameters of the sample image generation network remain unchanged, while the model parameters of the sample face control network are updated. This decouples the sample image generation network and the sample face control network, improving training efficiency. Since the sample face control network is mainly used to train the facial features of the fourth face, updating its model parameters allows it to focus more on learning facial features. The process continues until the difference between at least one second sample target image and at least one first sample target image satisfies the second condition. This indicates that the sample machine learning model has good accuracy in reconstructing the fourth face and good generalization ability. The sample machine learning model at this point is then used as the first machine learning model. This allows for a stronger interpretation of facial feature and semantic information when using the first machine learning model for subsequent face editing, thereby improving the accuracy and recognition of face editing.

[0056] In one possible implementation, at least one intermediate image is determined, which is obtained by processing at least one original sample image using a sample machine learning model; the at least one intermediate image is input into the sample face control network in the sample machine learning model to obtain at least one sample multi-scale conditional feature; the at least one sample mask image and the at least one sample multi-scale conditional feature are input into the sample image generation network in the sample machine learning model to output at least one second sample target image.

[0057] In this implementation, a sample machine learning model is trained on at least one original sample image to output at least one second sample target image. The training of the sample machine learning model determines the learning ability of intermediate images based on the original sample images. The training of the sample face control network enables it to learn the multi-scale conditional features of the output samples based on the intermediate images, thus better learning the facial feature information of the fourth face. Finally, the training of the sample image generation network enables it to output the second sample target image based on the multi-scale conditional features of the samples, allowing it to better reconstruct the fourth face from the facial feature information.

[0058] In one possible implementation, the sample mask image is obtained by masking the original sample image. The mask ratio of the sample mask image is negatively correlated with the number of generation steps and negatively correlated with the amount of semantic information in the original sample image. Alternatively, it can be understood that as the number of generation steps of the sample mask image gradually increases, the mask ratio gradually decreases while the amount of semantic information in the original sample image gradually increases, thereby gradually improving the accuracy and recognizability of the output fourth face during training.

[0059] In this implementation, the sample mask image allows the sample image generation network to focus on learning the mask region during training, enhancing its ability to interpret the facial feature information and semantic information corresponding to the mask region. As the number of generation steps gradually increases, by focusing on different mask regions each time, more facial details are learned, thereby gradually improving the accuracy and recognizability of the output fourth face.

[0060] In one possible implementation, at least one original sample image is input into the face estimation network of the sample machine learning model to obtain at least one sample parameter information, which includes the parameter information of the facial features of the fourth face; based on the at least one sample parameter information, at least one displacement map, at least one surface normal map, and at least one lighting geometry map of the fourth face are obtained; the displacement map, surface normal map, and lighting geometry map of the fourth face are superimposed to obtain an intermediate image of at least one fourth face.

[0061] Among them, displacement mapping can be used to describe the displacement and height changes of facial features, preserving facial details. Surface normal mapping can simulate the surface details and lighting effects of a face, thereby enhancing visual realism. Lighting geometry mapping can be used to describe the lighting conditions of a face.

[0062] In this implementation, the above steps train a sample machine learning model to output corresponding intermediate images based on the original sample images. The face estimation network is trained to output intermediate images based on sample parameter information. Intermediate images are generated through displacement mapping, surface normal mapping, and lighting geometry mapping to improve the model's ability to represent the facial feature details and lighting effects of the fourth face, thereby enhancing the realism of the reconstructed fourth face.

[0063] In one possible implementation, the sample parameter information includes sample face parameters, sample camera parameters, sample lighting parameters, and sample personal feature vectors. The sample face parameters include sample shape parameters, sample pose parameters, and sample expression parameters. The displacement map of the fourth face is obtained based on the face parameters and personal feature vectors. The surface normal map of the fourth face is obtained based on the sample parameter information, the face parameterization model, the normal calculation function, and the reflection renderer. The lighting geometry map is used to describe the lighting conditions of the fourth face.

[0064] In this implementation, the model can be trained to output the displacement map, surface normal map, and lighting geometry map of the fourth face based on the sample parameter information, thereby enhancing the realism of the reconstructed fourth face.

[0065] In one possible implementation, at least one sample mask image is input into the sample semantic encoding network in the sample image generation network to obtain at least one sample semantic latent variable; at least one sample noise image, at least one sample semantic latent variable and at least one sample multi-scale conditional feature are input into the sample conditional diffusion decoding network in the sample image generation network to output at least one second sample target image.

[0066] In this implementation, at least one sample mask image and at least one sample multi-scale conditional feature are input into the sample image generation network to output at least one second sample target image. This trains the sample image generation network to learn the ability to output the second sample target image based on the sample mask image and sample multi-scale conditional features, thereby improving the accuracy and recognizability of face reconstruction based on facial feature information.

[0067] In one possible implementation, the sample noise image is obtained by adding random noise to the original sample image.

[0068] Thirdly, this application provides an image processing apparatus, comprising:

[0069] The acquisition unit is used to acquire an original image and first parameter information. The original image includes a first face, and the first parameter information includes parameters for editing the first face in the original image.

[0070] The update unit is used to update the first machine learning model based on the original image to obtain an optimized second machine learning model. The second machine learning model is used to preserve the semantic information of the first face in the original image.

[0071] The output unit is used to edit the first face in the original image based on the first parameter information and the second machine learning model, and output the first target image.

[0072] The image processing apparatus in the third aspect can perform the operations performed in any possible implementation of the first aspect above and achieve the same technical effect, which will not be elaborated here.

[0073] Fourthly, this application provides a model training apparatus, comprising:

[0074] The acquisition unit is used to acquire at least one original sample image and at least one first sample target image. The original sample image includes a fourth face, and the first sample target image is the image that the sample machine learning model expects to output based on the original sample image.

[0075] The training unit is used to train a sample machine learning model using at least one original sample image and at least one first sample target image to obtain a first machine learning model. The sample machine learning model is used to reconstruct the fourth face based on the facial feature information and semantic information of the fourth face in the original sample image.

[0076] The model training device in the fourth aspect can perform the operations performed in any possible implementation of the second aspect above and achieve the same technical effect, which will not be elaborated here.

[0077] Fifthly, this application provides a computing device including a processor and a memory, wherein the processor is coupled to the memory, and the memory is used to store programs or instructions. When the program or instructions are executed by the processor, the computing device performs the method in the first aspect or any possible implementation thereof, or performs the method in the second aspect or any possible implementation thereof.

[0078] In a sixth aspect, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the method in the first aspect or any possible implementation thereof, or to cause the computing device cluster to perform the method in the second aspect or any possible implementation thereof.

[0079] In a seventh aspect, this application provides a computer program product containing instructions that, when executed by a computing device cluster, allow the computing device cluster to execute the method in the first aspect or any possible implementation thereof, or allow the computing device cluster to execute the method in the second aspect or any possible implementation thereof.

[0080] Eighthly, this application provides a computer-readable storage medium including computer program instructions, which, when executed by a cluster of computing devices, enable the cluster of computing devices to perform the method in the first aspect or any possible implementation thereof, or enable the cluster of computing devices to perform the method in the second aspect or any possible implementation thereof. Attached Figure Description

[0081] Figure 1 A schematic diagram of the structure of an electronic device provided in this application;

[0082] Figure 2 A schematic diagram illustrating one application scenario provided in this application;

[0083] Figure 3A schematic diagram of a system architecture provided for this application;

[0084] Figure 4 A flowchart illustrating an image processing method provided in this application;

[0085] Figure 5 A schematic diagram of the structure of a machine learning model provided in this application;

[0086] Figure 6 A flowchart illustrating a model training method provided in this application;

[0087] Figure 7 A schematic diagram of the structure of an image processing device provided in this application;

[0088] Figure 8 A schematic diagram of the structure of a model training device provided in this application;

[0089] Figure 9 A schematic diagram of the structure of a computer device provided in this application;

[0090] Figure 10 This application provides a schematic diagram of the structure of a computer device cluster;

[0091] Figure 11 This is a schematic diagram of another computer device cluster structure provided in this application. Detailed Implementation

[0092] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. With the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0093] The terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the description of embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to those processes, methods, products, or apparatuses.

[0094] First, the embodiments of this application involve a large number of applications related to neural networks. In order to better understand the solutions of the embodiments of this application, the relevant terms and concepts of neural networks that may be involved in the embodiments of this application will be introduced below.

[0095] (1) Neural Network

[0096] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit that takes input data and an intercept of 1 as input can output the following:

[0097]

[0098] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weight parameters are denoted by b, which represents the bias of the neural unit. f is the activation function of the neural unit, used to introduce non-linear characteristics into the neural network to convert the input signal into the output signal. The output signal of this activation function can be used as the input to the next convolutional layer; the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple individual neural units, meaning the output of one neural unit can be the input of another. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from the local receptive field, which can be a region composed of several neural units.

[0099] (2) Loss Function

[0100] In training deep neural networks, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value and update the weight vector of each layer based on the difference. (Of course, parameters are usually pre-configured for each layer before the first update.) For example, if the prediction is too high, the weight vector is adjusted to predict a lower value, and this process continues until the deep neural network can predict the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so training the deep neural network becomes a process of minimizing this loss. Common loss functions include mean squared error, cross-entropy, logarithmic, and exponential loss functions. For example, mean squared error can be used as the loss function, defined as... The specific loss function can be selected based on the actual application scenario.

[0101] (3) Backpropagation algorithm

[0102] An algorithm for calculating the gradient of model parameters based on a loss function and updating the model parameters. Neural networks can use backpropagation (BP) to correct the initial parameter values ​​during training, thus minimizing the error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial neural network model, thereby converging the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0103] In the embodiments of this application, the BP algorithm can be used to train the model during the training phase to obtain the trained model.

[0104] (4) Attention (also known as attention mechanism)

[0105] Attention mechanisms can quickly extract important features from sparse data. Attention occurs between the encoder and decoder, or more specifically, between the input and generated sentences. In contrast, the self-attention mechanism in a self-attention model occurs within the encoding matrix or the output sequence, extracting connections between distant words within the same sentence, such as syntactic features (phrase structure). Self-attention provides an effective modeling method for capturing global contextual information through QKV (key-value pairs). Assuming the input is Q (query), and the context is stored as key-value pairs (K, V), then the attention mechanism is essentially a mapping function from the query to a series of key-value pairs (key, value). The essence of the attention function can be described as a mapping from a query to a series of (key-value) pairs. Attention essentially assigns a weight coefficient to each element in the sequence, which can also be understood as soft addressing. If each element in the sequence is stored in (K, V) form, then attention performs addressing by calculating the similarity between Q and K. The similarity calculated between Q and K reflects the importance of the extracted V values, i.e., the weights, and then a weighted sum is obtained to obtain the final feature value.

[0106] Attention calculation mainly consists of three steps. The first step is to calculate the similarity between the query and each key to obtain weights. Common similarity functions include dot product, concatenation, and perceptron. The second step typically uses a softmax function (which can normalize the weights, resulting in a probability distribution where the sum of all weight coefficients is 1, and also highlights the weights of important elements) to normalize these weights. Finally, the weights and their corresponding key values ​​are weighted and summed to obtain the final feature value. The specific calculation formula is as follows:

[0107]

[0108] Where d is the dimension of matrix QK.

[0109] Furthermore, attention includes self-attention and cross-attention. Self-attention can be understood as a special type of attention where the inputs to the QKV features are consistent. Cross-attention, on the other hand, involves inconsistent inputs to the QKV features. Attention integrates the queried features as updated values ​​for the current features using the similarity between features (e.g., inner product) as weights. Self-attention is attention extracted based on the attention drawn from the feature map itself.

[0110] For convolutional networks, the kernel size limits the receptive field, often requiring multiple layers to focus on the entire feature map. Self-attention, on the other hand, offers the advantage of global focus; it can acquire global spatial information of the feature map through simple queries and assignments. A unique aspect of self-attention in query-key-value (QKV) models is that the inputs for each QKV value are consistent.

[0111] (5) Raw data: Raw data records the original information of the image sensor. It is an unprocessed and uncompressed format. RAW can be conceptualized as "raw image encoded data" or more figuratively called "digital negative".

[0112] (6)R(red), G(green), B(blue)

[0113] In this image, R represents red, G represents green, and B represents blue. Each image can be represented by the color values ​​of these three channels. For example, an RGB image represents an image with three color channels, and an RGGB image represents an image with four color channels, two of which are G channels.

[0114] This application provides an image processing method to improve the accuracy and recognition of face editing in original images. This application also provides a model training method for training a machine learning model that can reconstruct a face based on facial feature information and semantic information in an image. Furthermore, this application also provides an image processing apparatus, a model training apparatus, a computing device cluster, a computer program product, and a computer-readable storage medium.

[0115] The method provided in this application can include various deployment methods. For example, the method provided in this application can be deployed in an electronic device, allowing users to directly perform image processing or model training using the electronic device. Alternatively, it can be deployed in a cloud platform to provide image processing or model training services to user terminals. The different deployment methods are described below.

[0116] Deployment Method 1: Deployed in electronic devices

[0117] The electronic devices provided in this application embodiment, also referred to as user equipment (UE), terminals, mobile stations (MS), mobile terminals (MT), etc., are devices that include wireless communication functions (providing voice / data connectivity to users), such as handheld devices with wireless connectivity. Specifically, they can include handheld devices, in-vehicle devices, computing devices, and other electronic devices that include or are connected to image sensors. They can also include digital cameras, cellular phones, smartphones, personal digital assistant (PDA) computers, tablet computers, laptop computers, machine-type communication (MTC) terminals, point-of-sale (POS) terminals, in-vehicle computers, head-mounted devices, data processing devices (such as wristbands, smartwatches, etc.), security equipment, virtual reality (VR) devices, augmented reality (AR) devices, and other electronic devices with imaging functions. Applications that need to display images containing faces to be processed can run in this electronic device, such as live streaming applications, news broadcasting applications, conferencing applications, and image creation applications.

[0118] Taking digital cameras as an example, a digital camera is a type of camera that uses photoelectric sensors to convert optical images into digital signals. This includes SLR cameras, industrial cameras, and high-speed cameras. Unlike traditional cameras that rely on changes in photosensitive chemicals on film to record images, digital cameras use photosensitive charge-coupled devices (CCDs) or complementary metal-oxide-semiconductor (CMOS) sensors. Compared to traditional cameras, digital cameras, by directly using photoelectric conversion image sensors, offer advantages such as greater convenience, speed, repeatability, and timeliness. With the development of CMOS processing technology, the functions of digital cameras have become increasingly powerful, almost completely replacing traditional film cameras, and finding extremely wide applications in consumer electronics, human-computer interaction, computer vision, and autonomous driving.

[0119] For example, Figure 1This application provides a schematic diagram of an electronic device, which may include a lens assembly 110, a sensor 120, and an electrical signal processor 130. The electrical signal processor 130 may include an analog-to-digital (A / D) converter 131 and a digital signal processor 132. The A / D converter 131 is an analog-to-digital converter used to convert analog electrical signals into digital electrical signals. The sensor 120 may specifically include an image sensor, a multispectral image (MSI) sensor, etc.

[0120] It should be noted that, Figure 1 The electronic devices shown are not limited to those mentioned above, and may include more or fewer other devices, such as batteries, flashlights, buttons, sensors, etc. This application embodiment only uses an electronic device with sensor 120 installed as an example for illustration, but the components installed on the electronic device are not limited to this.

[0121] The light signal reflected from the object being photographed is converged by the lens assembly 110 and imaged onto the sensor 120. The sensor 120 converts the light signal into an analog electrical signal. The analog electrical signal is converted into a digital electrical signal by the analog-to-digital (A / D) converter 131 in the electrical signal processor 130, and then processed by the digital signal processor 132, for example, by optimizing the digital electrical signal through a series of complex mathematical algorithms, and finally outputting an image. The electrical signal processor 130 may also include an analog signal preprocessor 133, which is used to preprocess the analog electrical signal transmitted from the image sensor before outputting it to the analog-to-digital converter 131.

[0122] The method provided in this application can be specifically deployed in the electrical signal processor 130 of an electronic device, for example, it can be specifically deployed in the digital signal processor 132, or it can be deployed in other processors of the electronic device.

[0123] When editing a face image, sensor 120 acquires the original image and parameter information. The original image can be the face image to be edited, and the parameter information can be used to edit the face in the original image. Electrical signal processor 130 further processes the original image and parameter information, using the original image to update the first machine learning model. This updated second machine learning model retains the semantic information of the face in the original image, including personalized facial features. Then, the second machine learning model uses the parameter information to edit the face in the original image, outputting the target image. By parametrically editing the face—for example, allowing the user to edit facial feature parameters via buttons, sliders, or directly modify facial feature parameters via text boxes—the accuracy of face editing is improved. By using a second machine learning model that retains facial semantic information to edit the face, personalized facial features can be preserved, improving face recognition accuracy.

[0124] When training the model, at least one sample original image and at least one sample target image can be acquired through sensor 120. The sample original image includes a face, and the sample target image corresponds to the sample original image. The sample target image is the expected output image after the sample original image is trained by the sample machine learning model. Electrical signal processor 130 further processes the at least one sample original image and at least one sample target image, and trains the sample machine learning model using the at least one sample original image and at least one sample target image until the difference between the face in the model output image and the face in the sample target image meets the condition, thus obtaining a trained first machine learning model.

[0125] Deployment Method 2: Deployed on a cloud platform

[0126] The method provided in this application can also be deployed on a cloud platform, where one or more terminals can access the platform to provide image processing or model training services to the terminals.

[0127] Please see Figure 2 This is one application scenario of the image processing method in the embodiments of this application. This scenario may include a cloud platform 11 and a terminal 12, which can be connected via wired or wireless means.

[0128] The cloud platform 11 may specifically include a server cluster with storage and processing capabilities. The method provided in this application can be deployed on the cloud platform 11. The server cluster includes storage nodes, scheduling nodes, and worker nodes.

[0129] Storage nodes are used to store data, such as original images and parameter information. Furthermore, storage nodes can store training samples used to train generative models, such as the first machine learning model mentioned earlier. The generative model can then be used to edit faces in the original image based on the parameter information, outputting the edited target image. Storage nodes can send their stored data to the scheduling node.

[0130] The functionality of a scheduling node can be implemented through software or hardware. A scheduling node can retrieve raw images and parameter information from storage nodes, send the raw images and parameter information to worker nodes, and allocate tasks. As an example of a software functional unit, a scheduling node can be responsible for distributing image processing tasks. A scheduling node may include code running on a computing instance. This computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. The scheduling node manages and allocates image processing tasks to multiple worker nodes, such as graphics processing units (GPUs) accelerators or rendering nodes, enabling efficient execution of image processing tasks.

[0131] Worker nodes can be used to process raw images using parameter information through machine learning models to obtain target images. These machine learning models can be stored in storage nodes on cloud platform 11. Worker nodes can execute image processing tasks assigned by scheduling nodes. After obtaining the target image, it can be fed back to the storage node. Worker nodes can be physical machines, virtual machines (VMs), or containers, etc. A worker node can include one or more central processing units (CPUs) and graphics processing units (GPUs), and can be either a CPU or a GPU.

[0132] The above section describes the composition and structure of the cloud platform 11. The following section describes the interaction between the terminal 12 and the cloud platform 11.

[0133] When editing a facial image, the cloud platform 11 can receive the original image and parameter information from the terminal 12. The original image can be the facial image to be edited, and the parameter information can be used to edit the face in the original image. The original image is used to update the first machine learning model, resulting in an updated second machine learning model. This second machine learning model retains the semantic information of the face in the original image, including personalized facial features. Then, the second machine learning model uses the parameter information to edit the face in the original image, outputting the target image. Finally, the target image is fed back to the terminal 12.

[0134] Terminal 12 can perform image processing by interacting with cloud platform 11. For more information about terminal 12, please refer to [link to relevant documentation]. Figure 1 The relevant descriptions will not be repeated here. Terminal 12 can transmit raw images to cloud platform 11. These raw images can be images captured by the terminal itself, images input by the user, images stored locally on the terminal, etc.

[0135] For example, a cloud platform can provide services to users through clients deployed on terminals or web pages on terminals. Taking the deployment of a client on a terminal as an example, a user can send the original image captured by the terminal to the cloud platform through the client deployed on the terminal. The cloud platform 11 uses the method provided in this application to parametrically edit the face in the original image and feeds back the processed target image to the terminal 12.

[0136] In one possible scenario, it can also be applied to a scenario with multiple terminals. For example, a user can use a terminal other than the aforementioned terminal 12 to capture the aforementioned original image, transmit the original image to the aforementioned terminal 12, and the terminal 12 uploads the original image to the cloud platform 11. The cloud platform 11 parametrically edits the face in the original image, and then the cloud platform 11 feeds back the target image to the terminal 12, and the terminal 12 feeds back the processed target image to the aforementioned terminal that captured the original image.

[0137] The preceding sections explained image processing methods in conjunction with application scenarios of cloud platforms and terminals. The following sections, also using cloud platforms and terminals, explain model training methods. For a detailed description of cloud platforms and terminals, please refer to the preceding sections; they will not be repeated here.

[0138] When training the model, the cloud platform 11 can receive at least one sample original image and at least one sample target image from the terminal 12. The sample original image includes a face to be trained, and the sample target image corresponds to the sample original image. The sample target image is the expected output image after the sample original image is trained by the sample machine learning model. Further, using at least one sample original image and at least one sample target image, the sample machine learning model is trained until the difference between the face in the model's output image and the face in the sample target image meets a condition, resulting in a trained first machine learning model. The trained first machine learning model can be deployed on the cloud platform 11 to reduce the local storage space of the terminal 11. Subsequently, the machine learning model deployed on the cloud platform 11 can process the original image. Alternatively, the cloud platform can send the trained first machine learning model, which the terminal 12 receives, and subsequently, the first machine learning model deployed on the terminal 12 can process the original image.

[0139] The above section introduces the deployment method of the image processing method in this application. The following section describes the system architecture on which the image processing method in this application is based.

[0140] Combination Figure 3 As shown, the system architecture based on the embodiments of this application includes a data acquisition module, an image processing module, a visualization module, a model training and inference module, and a decoupled training module.

[0141] (1) Data acquisition module

[0142] Raw images and parameter information can be acquired through data acquisition devices. This parameter information can then be used to edit the raw image. Data acquisition devices can be digital cameras, such as high-speed cameras, SLR cameras, industrial cameras, and ordinary webcams. For a description of digital cameras, please refer to the foregoing. Figure 1 This will not be elaborated upon here. In addition, raw images, parameter information, and other data can be obtained from servers, local storage, and other sources.

[0143] (2) Image processing module

[0144] The image processing module can perform parametric editing on the acquired raw images, such as adjusting the brightness, contrast, and saturation of the raw images; modifying the proportions of the raw images; modifying the shape, expression, and posture of faces in the raw images; in addition, it can also perform style transfer or feature modification on the raw images, such as making faces in the raw images look younger, changing the gender of faces, hairstyles, or accessories.

[0145] (3) Visualization module

[0146] This module provides a user interface for interacting with the system, allowing users to adjust editing parameters (e.g., brightness, contrast, color adjustment). Responding to user editing operations on the original image—for example, users editing facial feature parameters via buttons, sliders, or directly modifying facial feature parameters through text boxes—the module outputs the edited face in the original image and displays intermediate results during processing, allowing users to view the editing effect in real time. Furthermore, the visualization module can record the user's editing history, facilitating undoing or comparing editing results from different historical records.

[0147] (4) Model Training and Inference Module

[0148] In model training, multiple face images can be used as a training set. A suitable sample machine learning model, such as a convolutional neural network or a generative adversarial network, is selected. This sample machine learning model is used to train the face images in the training set, resulting in a first machine learning model. This first machine learning model can reconstruct the face from the input image. See details for further information. Figure 6 The training method for the model shown.

[0149] Before editing a new facial image, the first machine learning model can be used to train the image. Alternatively, it can be understood as using the first machine learning model to perform semantic prior analysis on the facial image, thereby obtaining an updated second machine learning model that preserves the semantic information in the facial image. See details in [link to relevant documentation]. Figure 4 The relevant description of step 402.

[0150] In the case of model inference, the original image and parameter information acquired by the data acquisition module can be input into the trained second machine learning model. This second machine learning model then uses the parameter information to edit the face in the original image and outputs the edited target image. For details, please refer to [link / reference needed]. Figure 4 The relevant description of step 403.

[0151] (5) Decoupling training module

[0152] The decoupled training module allows the face control network and the image generation network to be trained independently, so that when precise editing of specific facial features is achieved, other features (such as semantic information) are not affected.

[0153] Specifically, when training the sample machine learning model, the model parameters of the image generation network can be kept constant while the model parameters of the face control network can be changed to better preserve facial feature information. Similarly, when training the first machine learning model, the model parameters of the face control network can be kept constant while the model parameters of the image generation network can be changed to better preserve semantic information in the face image, such as facial identity information and background information. Decoupling the training modules improves the flexibility and accuracy of face editing, avoiding potential interference between facial features and semantic information when editing facial features.

[0154] Please see Figure 4 This is a flowchart illustrating an image processing method provided in this application.

[0155] 401. Obtain the original image and first parameter information.

[0156] The original image includes a first face. For example, the original image may be a facial image including the first face, or it may be a full-body image or a partial body image including the first face. The specifics are not limited here.

[0157] The original image can be an unprocessed or unedited image of a human face. The color space of the original image can be RGB, CMYK, HSV, YCbCr, etc., and the file format of the original image can be JPEG, PNG, GIF, BMP, RAW, etc.

[0158] The original image can come from various sources, such as local storage, downloading from a server, or acquiring it from an image acquisition device. Specifically, the original image can be an image captured using the aforementioned electronic device, or an image retrieved from storage.

[0159] For example, if the method provided in this application embodiment is deployed in an electronic device, the original image can be captured by the electronic device, which can refer to the foregoing... Figure 1 The corresponding descriptions will not be repeated here.

[0160] For example, if the method provided in this application is deployed in a cloud platform, the original image can be a received image from a terminal. For instance, a user can input an image on the terminal, or take a picture using the terminal and send the image to the cloud platform through the terminal.

[0161] The first parameter information includes parameters for editing the first face in the original image, or it can be understood as the first parameter information that can be used to edit the facial features of the first face. Facial features can be local features of the first face such as the nose, mouth, eyes, ears, skin, and hair, or global features such as face shape and skin color. Optionally, the first parameter information includes first face parameters, first camera parameters, and first lighting parameters, wherein the first face parameters can be parameters for editing the facial features of the first face, such as the first shape parameter, first pose parameter, and first expression parameter of the first face.

[0162] The first shape parameter can be a parameter that is edited for the geometric structure or shape of the first face. Specifically, the first shape parameter can be the edited facial contour, such as face shape, facial feature proportions, etc.; it can also be the edited feature point position, such as the coordinates of feature points such as eyes, nose, mouth, etc.; or it can be the edited facial structure parameters, such as the relative height and width of cheekbones, chin, and forehead, etc.

[0163] The first pose parameter can be a parameter that is editable for the facial orientation or angle of the first face. Specifically, the first pose parameter can be the rotation angle of the first face, such as the pitch angle, yaw angle, and roll angle of the first face, or the outline position of the first face, etc., which is not limited here.

[0164] The first expression parameter can be the muscle changes and emotional state of the edited first face. Specifically, the first expression parameter can be the tension changes of the editor's facial muscles, such as smiling or frowning; it can also be the editor's emotional state, such as happiness, sadness, anger, or fear.

[0165] The first lighting parameter can be the light source characteristics for editing the original image, such as the brightness, contrast, saturation, hue, and highlights of the original image.

[0166] The first camera parameters can be camera parameters that are edited for the original image, such as the image quality, exposure, and resolution of the original image.

[0167] In practical applications, combining the above parameters can achieve different effects. For example, optimizing the first camera parameters and the first lighting parameters can enhance image quality; editing the first face parameters can enable personalized modifications to the first face; and editing the first expression parameters can make the first face more vivid.

[0168] In practical applications, in response to user actions on the original image, the first parameter information can be obtained. Specifically, the parameters corresponding to the controls edited by the user are obtained to set the first parameter. This control can be a button, slider, text box, etc. For example, the first parameter information can be obtained by dragging the pose parameter button, expression parameter button, lighting parameter button, etc. Through parametric editing, explicit and refined editing of the facial features of the first face can be performed.

[0169] It can also respond to user input to obtain textual attribute information of the first face, such as "look younger" or "have longer hair." This textual attribute information can be used to describe the semantic information of the first face. Optionally, the user-inputted textual attribute information can be used as semantic information for editing the first face.

[0170] Semantic information can be abstract information related to the first face, used to describe its personalized features. For example, semantic information can include the first face's identity information, adornment information, environmental information, gender information, age information, etc. Identity information, also known as identifier information, can be used to uniquely identify the first face's biometric identity; for example, it can be represented using a unique string. Adornment information can be information related to the first face's accessories, such as earrings, glasses, hats, etc. Environmental information can be information related to background elements of the original image, such as buildings, natural landscapes, etc. In subsequent processing steps, machine learning models can combine semantic information to determine the semantic latent variables of the first face.

[0171] 402. Based on the original image, update the first machine learning model to obtain the optimized second machine learning model.

[0172] The first and second machine learning models can have the same model structure. The first machine learning model includes a first image generation network and a first face control network, and the second machine learning model includes a second image generation network and a second face control network. The image generation network and the face control network are described below.

[0173] Face control networks can be used to modify face images, achieving precise control over facial features. A first face control network determines first multi-scale conditional features, which describe the facial features, lighting parameters, and camera parameters of the first face in the original image. A second face control network determines second multi-scale conditional features, which describe the facial features, lighting parameters, and camera parameters of the edited first face.

[0174] Different original images possess different multi-scale conditional features. Multi-scale conditional features can be represented as matrices, n-scale numerical vectors, etc. Each scale of the numerical vector corresponds to a feature; thus, the features of the original image can be described using an n-scale numerical vector. For example, a 256-dimensional vector can be used to describe the feature information of the first face in the original image, such as eye shape, eye size, eye position, face shape, and skin color.

[0175] In practical applications, face control networks can be deformable parametric models (3D morphable models), explicit face control networks (Exp-facenet), DeepFace, face control networks based on neural radiance fields (NeRF), etc. Some face control networks are introduced below.

[0176] 3DMM is a technique for 3D face modeling that can generate 3D face models with deformable features. It reconstructs a face using 3D models. Specifically, it uses a set of face models as a basis and combines these base models by incorporating different shape, pose, and expression parameters to output corresponding face models. Furthermore, it can be combined with deep learning and other methods to learn different combinations of face base models, enabling the generation of realistic face images under various angles and lighting conditions.

[0177] Exp-FaceNet is an explicit 3DMM-based model that leverages the detailed control over facial features such as shape, texture, and lighting provided by 3DMM, combined with deep learning techniques to achieve high-quality face synthesis. It uses explicit parameters to represent and control facial features, making the editing process more refined and controllable by adjusting 3D face parameters such as expression, pose, shape, and texture. Through this explicit parameterized control, users can independently adjust specific facial attributes to achieve precise editing results.

[0178] NeRF-based face control networks employ implicit modeling. They generate a series of functions (radiation fields) describing a static object or scene from multiple viewpoints, thus outputting an image of the object or scene at a given camera position. NeRF implicitly provides a 3D representation of the object or scene through these functions, rather than explicit modeling. Each function (model) is strongly correlated with the object or scene, requiring retraining for new objects or scenes. NeRF-based face control networks can handle complex facial structures and details without explicit parameter definitions, automatically learning and extracting facial features to generate high-quality images.

[0179] Image generation networks are characterized by learning from training data and generating output data with a specific distribution based on the characteristics of the training data. A first image generation network can output a second target image that is highly similar to the original image, meaning the second target image is a reconstruction of the first face in the original image. A second image generation network can use first parameter information to edit the first face in the original image, outputting a first target image, meaning the first target image is an edit of the first face in the original image. The faces in the first and second target images belong to the same person; for example, the faces in the first and second target images share the same identity information as the first face.

[0180] In practical applications, image generation networks can be generative adversarial networks (GANs), diffusion autoencoders (Diff-AEs), variational autoencoders (VAEs), etc. Some image generation networks are introduced below.

[0181] Generative Adversarial Networks (GANs) consist of a generator and a discriminator. The generator produces realistic data samples, while the discriminator determines the authenticity of the samples. High-quality generators are obtained through competition between the generator and discriminator. When training a GAN, face images are used as target data. During training, relevant control information (such as facial feature information) guides the generator's generation, allowing for simple editing of the face during the generation process. GANs can capture complex facial details and textures, generating high-quality and realistic face images.

[0182] When generating face images, Diff-AE can implicitly control high-dimensional semantic information. For example, by controlling semantic latent variables, different effects can be achieved when editing faces. This makes the diffusion model highly flexible in handling different types of editing tasks. Moreover, the model training process is relatively simple, without the need for complex network structures and training strategies, and can be trained more quickly.

[0183] Optionally, the image generation network may include a semantic coding network and a conditional diffusion decoding network.

[0184] Semantic coding networks are a type of neural network structure used to convert the attribute information of input images or text into feature representations with semantic information, such as semantic latent variables. Semantic latent variables can be represented in the form of matrices, vectors, etc. For more information on semantic information, please refer to the previous section, which will not be repeated here.

[0185] Optionally, text attribute information is input into a semantic coding network, which outputs semantic latent variables represented by the text attribute information. The generation effect of the target image is controlled by the text attribute information.

[0186] For example, by combining text attribute information, such as user input of text like "get younger", "remove glasses", and "remove freckles", the semantic encoding network can be used to convert the text into semantic latent variables for representation.

[0187] Conditional diffusion decoding networks are network structures based on diffusion models. Through a back-diffusion process, they use semantic latent variables and / or multi-scale conditional features to output target images under specific conditions. Alternatively, they can be understood as controlling facial feature information and / or semantic information to output target images under specific conditions.

[0188] Alternatively, the conditional diffusion decoding network can generate the target image using a noisy image.

[0189] For example, if the user inputs the text attribute information "smiling young woman", and this text attribute information is input into a semantic encoding network, the output semantic latent variable can be represented by features such as gender "female", expression "smiling", and age "20 to 30 years old". This semantic latent variable and the multi-scale conditional features corresponding to the original image are then input into a conditional diffusion decoding network, and the output target image can be "smiling young woman" edited from the face in the original image.

[0190] Therefore, by combining semantic coding networks and conditional diffusion decoding networks, the diversity of generated face images can be improved, enabling the model to generate images of various styles under different conditions, such as different editing parameters and different semantic information.

[0191] It should be understood that parameter information and multi-scale conditional features can represent the meaning of "facial feature information", while semantic latent variables, text attribute information, and personalized features can represent the meaning of "semantic information".

[0192] Optionally, the first image generation network includes a first semantic encoding network and a first conditional diffusion decoding network. The first semantic encoding network is used to determine a first semantic latent variable, which is used to describe the semantic information of the first face in the original image. The first conditional diffusion decoding network is used to output a second target image.

[0193] Optionally, the second image generation network includes a second semantic encoding network and a second conditional diffusion decoding network. The second semantic encoding network is used to determine a second semantic latent variable, which is used to describe the semantic information of the edited first face. The second conditional diffusion decoding network is used to output the first target image.

[0194] Optionally, the machine learning model includes a face estimation network. The face estimation network can be used to acquire 3DMM parameters of a face image, such as camera parameters, lighting parameters, and face parameters (pose parameters, expression parameters, shape parameters), etc. The face estimation network can be an emotion-driven monocular face capture and animation (EMOCA) network, a deep emotion capture and animation (DECA) network, etc. Both EMOCA and DECA can be used to capture 3DMM parameters edited for a first face, and can also be used for facial animation and expression synthesis. By combining emotional factors with 3DMM, various emotional states, such as happiness, anger, and surprise, can be captured and synthesized.

[0195] Combination Figure 5 The diagram shown is a structural schematic of a machine learning model provided in this application. This machine learning model includes an EMOCA network, a face control network (the aforementioned Exp-FaceNet), and an image generation network (…). Figure 5 The Diff-AE network shown includes a semantic coding network. Figure 5 The semantic encoder and conditional diffusion decoding network shown Figure 5 (The conditional diffusion decoder is shown). This machine learning model can serve as the first machine learning model, the second machine learning model, and subsequent sample machine learning models in this application.

[0196] The above section describes the structure of the first and second machine learning models. The following section describes the process of updating the first machine learning model to obtain the second machine learning model.

[0197] The first machine learning model is trained using the original image to obtain the second machine learning model. Specifically, the original image is input into the first machine learning model to update its model parameters, and the second target image is output. The expected output of the first machine learning model is the original image. Whether to continue updating the model parameters of the first machine learning model is determined by judging whether the difference between the original image and the second target image satisfies a first condition. When the difference between the original image and the second target image satisfies the first condition, the first machine learning model is used as the second machine learning model.

[0198] When updating the model parameters of the first machine learning model, the model parameters of the first face control network within the first machine learning model are kept unchanged, and only the model parameters of the first image generation network are updated. For example, the model parameters of Exp-FaceNet are kept unchanged, while the model parameters of Diff-AE are changed. The first image generation network can control the first semantic latent variables and the first multi-scale conditional features. Updating only the model parameters of the first image generation network allows the first machine learning model to focus on learning the semantic and facial feature information of the first face during training. This enables the trained second machine learning model to better preserve the semantic information of the first face. For example, it can reduce problems such as identity shifts or background changes that occur during subsequent editing of the first face. In addition, updating only the model parameters of the first image generation network can decouple the first face control network and the first image generation network, improving update efficiency.

[0199] It should be understood that the first machine learning model is updated using the original image to obtain the second machine learning model, so that the second machine learning model can better preserve the semantic information of the first face in the original image; or it can be understood as using the original image to train the first machine learning model to ensure the generalization performance of the first machine learning model on the first face; it can also be understood as using the first machine learning model to perform semantic prior on the original image, and by using the original image to update the first machine learning model, the updated second machine learning model can understand and utilize the semantic information of the first face without losing or distorting the semantic information, resulting in identity shift or background change, and in the subsequent editing of the first face, it can enhance the accuracy of the second machine learning model in generating the first face.

[0200] The following section details the process of training the first machine learning model using the original images to obtain the second machine learning model.

[0201] S1. Determine the intermediate image corresponding to the original image.

[0202] The original image is input into the face estimation network (such as the EMOCA network) of the first machine learning model to obtain the parameter information of the first face in the original image. This parameter information is the facial feature information of the first face itself, including face parameters, camera parameters, illumination parameters and personal feature vectors. Among them, face parameters include shape parameters, pose parameters and expression parameters, and personal feature vectors can be a vector representation of the facial feature information of the first face.

[0203] Using the parameter information of the first face, a surface normal map of the first face is generated through a face parametric model (such as the FLAME model), a normal calculation function, and a reflection renderer (such as the Lambertian reflection renderer). Using the face parameters and personal feature vectors of the first face, a displacement map of the first face is generated. The grayscale values ​​of the color map of the first face are set to fixed values ​​to generate a lighting geometry map of the first face, which describes the lighting conditions of the first face. Finally, the combined effect of the surface normal map, displacement map, and lighting geometry map is rendered to obtain an intermediate image corresponding to the original image.

[0204] S2. Input the intermediate image into the first face control network of the first machine learning model to obtain the first multi-scale conditional features.

[0205] The first multi-scale conditional feature is used to describe the facial feature information of the first face in the original image. It can be understood as a feature representation of the parameter information of the first face in the original image; it can be a feature vector, a matrix, etc. The first face control network, such as Exp-FaceNet, processes intermediate images and outputs the first multi-scale conditional feature.

[0206] S3. Input the first multi-scale conditional features into the first image generation network, keep the model parameters of the first face control network unchanged, update the model parameters of the first image generation network, and output the second target image until the difference between the original image and the second target image satisfies the first condition, and obtain the trained second machine learning model.

[0207] The first multi-scale conditional features, the mask image corresponding to the original image, and the first noise image corresponding to the original image are input into the first image generation network to obtain the second target image. Specifically, the original image is divided into regular, non-overlapping image blocks, and different parts of the image blocks are randomly masked to obtain a mask image at different time steps. This mask image is input into the first semantic encoding network of the first image generation network to obtain the first semantic latent variable. The first semantic latent variable is used to describe the semantic information of the first face, and this semantic information contains fragmented and incomplete content information in the original image at different time steps. The first noise image, the first semantic latent variable, and the first multi-scale conditional features are input into the first conditional diffusion decoding network of the first image generation network to output the second target image. Optionally, the first noise image is obtained by adding random noise to the original image.

[0208] The first multi-scale conditional feature describes fine-grained facial feature information, camera parameters, and illumination parameters, used to recover the facial region of the occluded first face in the current mask image at each generation step. The semantic information described by the first semantic latent variable contains fragmented and incomplete content information from the original image; as the number of generation steps increases, the semantic information described by the first semantic latent variable gradually increases. During the output of the second target image, the first face in the second target image is generated gradually. As the number of generation steps increases, the facial feature information and semantic information of the first face in the second target image gradually increase, and the more personalized information the first face has, the more recognizable the face becomes. Specifically, at each generation, the masking rate of the first mask image is negatively correlated with the number of generation steps of the first mask image, and the masking rate of the first mask image is negatively correlated with the amount of semantic information in the original image. This can be understood as follows: as the number of generation steps of the first mask image gradually increases, the masking rate gradually decreases, and the amount of semantic information in the original image gradually increases.

[0209] The system determines whether the difference between the original image and the second target image meets the first condition. Specifically, it determines whether the difference value between the original image and the second target image meets a threshold. If the difference value does not exceed the threshold, it indicates that the first face in the input original image and the first face in the output second target image are highly similar. In other words, the first machine learning model reconstructs or restores the first face in the original image with good accuracy, thus meeting the first condition, and the first machine learning model is used as the second machine learning model. If the difference value exceeds the threshold, it indicates that the first face in the input original image and the first face in the output second target image are less similar. In other words, the first machine learning model reconstructs or restores the first face in the original image with poor accuracy, thus not meeting the first condition. In this case, the model parameters of the first image generation network in the first machine learning model are updated until the difference between the output image and the original image meets the first condition, and the updated first machine learning model is used as the second machine learning model. Optionally, a loss function is used to measure the difference value between the second target image and the original image. This loss function can be the mean squared error, cross-entropy, logarithmic, exponential, etc. For details, please refer to the aforementioned terminology section, which will not be elaborated here.

[0210] Optionally, the parameters of the first machine learning model can be updated using the backpropagation algorithm. For details, please refer to the aforementioned terminology section, which will not be repeated here.

[0211] It should be noted that restoring the first face can be understood as reconstructing the first face. The process of training the first machine learning model using the original image is the process of continuously reducing the difference between the first face in the input original image and the first face in the output second target image. In other words, the purpose of training the first machine learning model using the original image is to better learn the facial feature information and semantic information of the first face, that is, to be able to better reconstruct or restore the first face.

[0212] Step 402 can be understood as the process of training the first machine learning model using the original image. By using the original image to fine-tune the first machine learning model, the model's generalization performance and adaptability to new inputs are improved. This allows the generated image to better retain personalized features, better reflect individual differences, enhance the model's expressiveness when processing specified face images, and reduce identity shifts or background changes that occur during subsequent editing of the first face. Next, step 403 will explain the process of editing the original image.

[0213] 403. Using the first parameter information through the second machine learning model, edit the first face in the original image and output the first target image.

[0214] By using the first parameter information to edit the first face through the second machine learning model, a target image after editing the first face can be generated using the parameter information and semantic information of the first face. The process of outputting the first target image is explained in detail below.

[0215] S1. Using the first parameter information, edit the first face in the original image to obtain an intermediate image of the first face.

[0216] The original image is input into the face estimation network of the second machine learning model to obtain second parameter information. This second parameter information can be the facial feature parameters of the first face before editing; it can be understood as the facial feature information inherent in the first face itself. For example, the original image is input into the EMOCA network to obtain second parameter information, which includes second face parameters of the first face, second camera parameters, second illumination parameters, and a personal feature vector. The personal feature vector can be a vector representation of the facial feature information of the first face.

[0217] The second face parameter can be a parameter related to the facial features of the first face, such as the shape parameter, pose parameter, and expression parameter of the second face.

[0218] The second shape parameter can be a parameter related to the geometric structure or shape of the first face. Specifically, the first shape parameter can be a parameter related to the facial contour of the first face, such as face shape, facial feature proportions, etc.; it can also be the location of feature points, such as the coordinates of feature points such as eyes, nose, mouth, etc.; it can also be a parameter related to facial structure, such as the relative height and width of cheekbones, chin, and forehead, etc., which is not limited here.

[0219] The second pose parameter can be a parameter related to the facial orientation or angle of the first face. For example, the first face in the original image is offset by 20 degrees from the reference direction.

[0220] The second facial expression parameter can be related to the muscle changes and emotional state of the first face. For example, it can be the emotional state, such as happiness, sadness, anger, or fear.

[0221] The second lighting parameter can be used to describe the light source characteristics of the original image, such as the brightness, contrast, saturation, hue, and highlights of the original image.

[0222] The second camera parameters can be used to describe the camera parameters of the original image, such as the image quality, exposure, and resolution of the original image.

[0223] By using the first and second parameter information for calculation, the edited parameter information for the first face can be obtained. For example, the coordinates of the nose feature points in the second shape parameter are added to or subtracted from the edited values ​​of the nose feature points in the first shape parameter to obtain the edited nose feature point coordinates. Similarly, the brightness value in the second lighting parameter is calculated and processed with the adjusted brightness value in the first lighting parameter to obtain the edited lighting parameters. Furthermore, if the second expression parameter indicates a sad state and the first expression parameter indicates a happy state, the edited expression of the first face will change from sad to happy.

[0224] Based on the parameter information after editing the first face, generate the displacement map, surface normal map, and lighting geometry map of the first face.

[0225] A displacement map of the first face is generated using first face parameters, second face parameters, and personal feature vectors. Specifically, the first face parameters can be used as a reference point. Then, displacement vectors for each facial feature are calculated using the first and second face parameters. Finally, the personal feature vectors are applied to the displacement vectors to obtain the displacement map of the first face. During rendering, the displacement map can describe the displacement and height changes of the first face's facial features, and also reflect changes in the shape, posture, and expression of the first face. For example, when the face turns to the left, the displacement map corresponds to this process. Another example is when the user edits out wrinkles on the first face; the displacement map can be used to reduce wrinkle details.

[0226] Using first and second parameter information, a surface normal map of the first face is generated through a parametric face model (such as the FLAME model), a normal calculation function, and a reflection renderer (such as the Lambertian reflection renderer). The surface normal map can simulate the surface details and lighting effects of the first face by encoding surface normals, thereby enhancing visual realism. Specifically, the parametric face model uses the first and second parameter information to generate the 3D vertex coordinates of the first face. The normal calculation function is used to calculate the normals of the 3D vertex coordinates, for example, the vertex normals and face normals of the 3D vertex coordinates. Determining the surface normal map based on the normals can be understood as mapping the normals to the RGB range and converting them into an image format. The Lambertian reflection renderer is used to adjust the light source effects; the Lambertian reflection renderer can calculate the light intensity based on the normals and the light source direction. Thus, the surface normal map of the first face is obtained. Set the grayscale value of the edited first face to a fixed value. For example, set the grayscale value of the color map of the first face to a fixed value, and generate the lighting geometry map of the first face. This lighting geometry map is used to describe the lighting conditions of the first face.

[0227] Next, the displacement map, surface normal map, and lighting geometry map of the first face are overlaid to obtain an intermediate image of the first face. Alternatively, it can be understood as rendering a combination of the displacement map, surface normal map, and lighting geometry map of the first face to obtain an intermediate image of the first face.

[0228] S2. Input the intermediate image of the first face into the second face control network in the second machine learning model to obtain the second multi-scale conditional features.

[0229] A second multi-scale conditional feature is determined by a second face control network to describe the facial feature information after editing the first face. For example, Exp-FaceNet processes the intermediate image to obtain the second multi-scale conditional feature. This enables the second image generation network to generate a second target image after editing the first face based on the second multi-scale conditional feature.

[0230] Optionally, the machine learning models involved in this application, such as the first machine learning model, the second machine learning model, and the sample machine learning model, may include the functional modules involved in this application, such as face parameterization models, normal calculation functions, reflection renderers, etc., or include modules that can achieve the same function.

[0231] S3. Input the mask image and the second multi-scale conditional features into the second image generation network in the second machine learning model, and output the first target image.

[0232] First, the mask image is input into the second semantic coding network within the second image generation network to obtain the second semantic latent variable, which describes the semantic information of the edited first face. The mask image is used to specify the region of interest in the original image, identifying the areas where operations will be performed. The mask image can be a binary image, a grayscale image, or a multi-channel image; no specific limitation is made here. When the mask image is a binary image, each pixel in the mask image can correspond to two values: one indicating that an operation will be performed on that pixel, and the other indicating that no operation will be performed on that pixel. For example, "1" represents an operation on that pixel, and "0" represents no operation on that pixel. Operations are performed between the resulting mask image and the original image, such as bitwise AND, bitwise OR, and bitwise XOR operations. Operations can be performed on pixel regions with a value of 1. When the mask image is a grayscale image, different grayscale values ​​are set for each pixel to identify the region of interest and to quantify the importance of the pixel region. For example, each pixel can take different values ​​from 0 to 255. A pixel region with a value of 0 can represent a region of no interest, while a pixel region with a value of 255 can represent a region of complete interest, and other values ​​represent some degree of interest.

[0233] The mask image can be obtained by specifying a mask region (region of interest) or by randomly masking the original image; the specific method is not limited here. For example, the original image can be divided into regular, non-overlapping image blocks, and different parts of the image blocks can be randomly masked in each generation step to obtain the mask image.

[0234] Then, the noisy image, the second semantic latent variable, and the second multi-scale conditional features are input into the second conditional diffusion decoding network in the second image generation network to output the first target image.

[0235] The noisy image is an image containing random interference or disordered information. In step 403, the noisy image is a random Gaussian noise image with a standard normal distribution.

[0236] During the output of the first target image, the first face in the first target image is generated gradually. It can be understood that the output target image is obtained through continuous optimization. As the number of generation times increases, the facial feature information and semantic information of the first face in the target image gradually increase. The more personalized information the first face has, the more recognizable the face becomes.

[0237] During each generation, the mask image can be randomly generated. The mask ratio of the mask image is negatively correlated with the number of generation steps, and it is also negatively correlated with the amount of semantic information in the original image. This can be understood as follows: as the number of generation steps gradually increases, the mask ratio gradually decreases, while the amount of semantic information in the original image gradually increases. The mask ratio refers to the proportion of the image covered by the mask, which can be expressed as the ratio of the number of masked pixels to the total number of pixels. For example, a mask ratio of 20% means that 20% of the pixels in the original image are masked.

[0238] For example, for each denoising step t, it can be calculated by ρ t The original image is randomly masked using a ratio of 0.75-0.5(Tt) / T to generate a masked image. Here, T is the total number of generation steps, for example, it can be set to 20; the initial value of the denoising step t is T, and t gradually decreases as the number of generation steps increases. In the initial generation stage, the mask ratio ρ... t The masking rate ρ is relatively high, meaning the original image has a large amount of masking. During the generation process, the denoising step t gradually decreases, and the facial feature information of the first face is gradually restored using the second multi-scale conditional features, resulting in a masking rate ρ. t As the mask image is gradually reduced, it provides more semantic information about the original image.

[0239] As described above, during the editing of the first face using the first parameter information, a mask image is used through the second image generation network. Since the mask image can specify a specific region, it effectively separates the editing of different parts during face image editing, reducing interference and dependencies between different editing tasks. For example, adjusting the eyes in an image avoids affecting other facial features such as eyebrows and ears. This further enhances the flexibility and independence of editing, allowing each editing operation to achieve the desired effect more precisely. Using a second image generation network (such as Diff-AE) to generate the first target image can better preserve the semantic information of the original image, improving the recognizability and realism of the face in the generated first target image. Using a second face control network (such as Exp-FaceNet) to process the first parameter information allows for control over parameters to achieve explicit editing of the face, enabling more precise adjustments to specific facial details, enhancing the controllability and visualization of the editing effect, and meeting diverse editing needs.

[0240] Step 403 describes the process of editing the first face using the first parameter information. The first face is edited using the first parameter information to obtain an intermediate image. This intermediate image is then input into a second face control network, which outputs a second multi-scale conditional feature. Finally, the mask image and the second multi-scale conditional feature are input into a second image generation network, which outputs a first target image.

[0241] In this embodiment, through a new sample training phase (step 402) and an image editing phase (step 403), parameterized modifications to the face in the original image are achieved while preserving personalized features, thereby improving the accuracy and recognizability of face editing in the original image. In the new sample training phase, the original image is used to update the first machine learning model, ensuring its generalization performance on the first face. This allows the updated second machine learning model to better preserve the semantic information of the first face. Semantic information can be used to describe the personalized features of the face; in other words, the second machine learning model can better preserve the personalized features of different faces, thus improving the recognizability of the first face. In the image editing phase, the second machine learning model uses the first parameter information to edit the first face. By parameterizing the facial features, refined modifications to the first face in the original image are achieved, improving the accuracy of the editing.

[0242] The following section explains the extended application scenarios of the second machine learning model.

[0243] In practical applications, a second machine learning model can be used to achieve a "face-swapping" effect. For example, replacing the first face in the original image with a second face in the comparison image. Specifically, the user uploads an additional face image, and the second machine learning model can replicate the pose, expression, or lighting information of that face image onto the first face in the original image.

[0244] The specific process is as follows: A comparison image uploaded by the user is obtained, which includes a second face. The comparison image is input into the face estimation network of the second machine learning model, such as the EMOCA network. The EMOCA network captures the third parameter information of the second face in the comparison image. The third parameter information includes facial feature information of the second face, such as shape parameters, pose parameters, and expression parameters. For a description of the relevant parameter information, please refer to the preceding sections; it will not be repeated here. The second machine learning model uses the third parameter information to process the original image including the first face. This can be understood as replacing the previously edited parameter information for the first face with the third parameter information, i.e., using the third parameter information to reconstruct the first face and outputting a third target image. The third target image is obtained by replacing the first face in the original image with the second face, similar to step 403 above; it will not be repeated here.

[0245] In practical applications, a second machine learning model can be used to achieve the effect of an "AI digital human." For example, a user uploads another video, which drives the first face in the original image. The process is as follows: obtain a reference video, which includes a third face; use the second machine learning model with second parameter information to process the reference video and output a target video, which is obtained by replacing the third face in the reference video with the first face.

[0246] Specifically, the video includes at least one frame of image. The video is input into a second machine learning model, which processes each frame of the video and outputs a corresponding target image. The output target images are then stitched together in chronological order to output the processed target video. The processing of one frame is as follows: The facial feature information of the first face, i.e., the second parameter information, is used to replace the facial feature information of the third face. Based on the second parameter information, the displacement map, surface normal map, and lighting geometry map of the first face are determined. The displacement map, surface normal map, and lighting geometry map are superimposed to obtain an intermediate image of the first face. The intermediate image is input into a second face control network, which outputs a second multi-scale conditional feature. The mask image of the first face and the second multi-scale conditional feature are input into a second image generation network, which outputs the target image of that frame.

[0247] Please see Figure 6This is a flowchart illustrating a model training method provided in this application. Using this model training method, a sample machine learning model can be trained to obtain the aforementioned first machine learning model.

[0248] 601. Obtain at least one original sample image and at least one first sample target image.

[0249] The original sample image includes a fourth face. For example, the original sample image can be a facial image including a fourth face, or it can be a full-body image or a partial body image including a fourth face. The specific details are not limited here.

[0250] At least one original sample image and at least one first sample target image can be data used to train a machine learning model, such as open-source portrait datasets like the WFLW dataset, SCUT-FBP5500 dataset, and EmotioNet facial expression dataset. They can also be face images acquired through various means, such as from local storage, downloaded from a server, or acquired from an image acquisition device. Specifically, at least one original sample image and at least one first sample target image can be obtained using the aforementioned... Figure 1 Images can be captured by electronic devices or retrieved from storage.

[0251] The sample machine learning model is used to reconstruct the fourth face based on the facial feature information and semantic information of the fourth face in the original sample image.

[0252] The first sample target image is the image that the sample machine learning model expects to output based on the original sample image. It can be understood that for any original sample image in the input sample machine learning model, there is a corresponding expected output first sample target image.

[0253] Optionally, the first sample target image and the original sample image can be the same or different; no specific limitation is made here. Specifically, when the first sample target image and the original sample image are the same, the purpose of training the sample machine learning model is to reconstruct the face. When the first sample target image and the original sample image are different, different face editing effects can be achieved by controlling the sample parameter information and / or semantic information of the face during training. For example, if the effect of training the sample machine learning model is to "change the facial expression from frowning to smiling," at least one original sample image can be a face image with a "frowning expression," and at least one first sample target image can be a face image with a "smiling expression." Different expression parameters can be controlled during training to enable the trained model to achieve this effect.

[0254] It should be understood that the process of training a sample machine learning model can be viewed as "reconstructing a fourth face based on facial feature information and semantic information." When the trained sample machine learning model is used to achieve different effects in face editing, it is achieved by controlling the sample parameter information and / or semantic information of the face. Therefore, the foundation of face editing is face reconstruction. Alternatively, the basic function of the sample machine learning model can be understood as performing face reconstruction.

[0255] Optionally, the sample machine learning model includes a sample image generation network and a sample face control network. The sample face control network is used to determine the multi-scale conditional features of the samples. These multi-scale conditional features are used to describe the facial feature information of the fourth face in the original sample image. The sample image generation network is used to generate the second sample target image. For details on multi-scale conditional features, please refer to the preceding sections; they will not be repeated here. The sample face control network can be the aforementioned Exp-FaceNet, a NeRF-based face control network, etc., and the sample image generation network can be the aforementioned GAN, Diff-AE, etc., which will not be repeated here.

[0256] Optionally, the sample image generation network includes a sample semantic encoding network and a sample conditional diffusion decoding network. The sample semantic encoding network is used to determine sample semantic latent variables, which describe the semantic information of the fourth face in the original sample image. The sample conditional diffusion decoding network is used to output the second sample target image. Training with the sample semantic encoding network and the sample conditional diffusion decoding network can improve the diversity of generated face images, enabling the model to generate images of various styles under different conditions, such as different sample parameter information and different semantic information. For a description of the semantic encoding network and the conditional diffusion decoding network, please refer to the preceding sections, which will not be repeated here.

[0257] Optionally, by training the sample machine learning model using different text attribute information, a first machine learning model is obtained, which allows the first machine learning model to control the generation effect of the original image using the text attribute information.

[0258] Optionally, a machine learning model can be trained using noisy images to obtain a first trained machine learning model.

[0259] Optionally, the sample machine learning model includes a face estimation network, such as an EMOCA network or a DECA network, which is used to capture sample parameter information of a fourth face.

[0260] After obtaining at least one original sample image and at least one first sample target image, a sample machine learning model can be trained using the at least one original sample image and at least one first sample target image to obtain the first machine learning model. The following provides a detailed explanation in conjunction with steps 602 to 605.

[0261] 602. Determine at least one intermediate image, which is obtained by processing at least one sample original image using a sample machine learning model.

[0262] Using a sample-based machine learning model, an intermediate image is obtained by processing one of the original sample images. This process is repeated to obtain at least one intermediate image from at least one original sample image. The specific process is as follows:

[0263] At least one original sample image is input into the face estimation network of the sample machine learning model, such as the EMOCA network, to obtain at least one sample parameter information. This sample parameter information includes facial feature parameters of the fourth face. The sample parameter information includes sample face parameters, sample camera parameters, sample illumination parameters, and sample personal feature vectors. Sample face parameters include sample shape parameters, sample pose parameters, and sample expression parameters. The personal feature vector can be a vector representation of the facial feature information of the fourth face. For a more detailed description of the sample parameter information, please refer to the preceding sections, such as the introduction to the second parameter information, which will not be repeated here.

[0264] Using at least one sample parameter, at least one displacement map, at least one surface normal map, and at least one lighting geometry map of the fourth face are obtained. The displacement map of the fourth face is generated using the sample face parameters and individual feature vectors. The surface normal map of the fourth face is generated using the sample parameter information, a face parameterization model (such as the FLAME model), a normal calculation function, and a reflection renderer (such as the Lambertian reflection renderer). The grayscale values ​​of the fourth face are set to fixed values; for example, the grayscale values ​​of the color map of the fourth face are set to fixed values ​​to generate the lighting geometry map of the fourth face, which describes the lighting conditions of the fourth face.

[0265] Next, at least one displacement map, at least one surface normal map, and at least one lighting geometry map of the fourth face are overlaid to obtain an intermediate image of at least one fourth face. Alternatively, it can be understood as rendering a combination of the displacement map, surface normal map, and lighting geometry map of at least one fourth face to obtain an intermediate image of the fourth face.

[0266] 603. Input at least one intermediate image into the sample face control network in the sample machine learning model to obtain at least one sample multi-scale conditional feature.

[0267] Multi-scale conditional features are used to describe the facial feature information of a fourth face in the original sample image. They can be understood as a feature representation of sample parameter information, which can be a feature vector, a matrix, etc. Sample face control networks, such as Exp-FaceNet, process at least one intermediate image and output at least one multi-scale conditional feature.

[0268] 604. Input at least one sample mask image and at least one sample multi-scale conditional feature into the sample image generation network in the sample machine learning model, and output at least one second sample target image.

[0269] At least one sample mask image is input into the sample semantic encoding network within the sample image generation network to obtain at least one sample semantic latent variable. This sample semantic latent variable describes the semantic information of the fourth face. The original sample image is divided into regular, non-overlapping image patches, and different parts of these patches are randomly occluded to obtain a mask image at different time steps. The sample mask image is input into the sample semantic encoding network, and the semantic information described by the output sample semantic latent variable includes fragmented and incomplete content information from the original sample image. The sample multi-scale conditional features output by the sample face control network describe fine-grained facial feature information, camera parameters, and illumination parameters. These multi-scale conditional features are used to recover the occluded facial regions in the current sample mask image during each generation step.

[0270] At least one sample noisy image, at least one sample semantic latent variable, and at least one sample multi-scale conditional feature are input into the sample conditional diffusion decoding network in the sample image generation network, and at least one second sample target image is output.

[0271] Optionally, the sample noise image is obtained by adding random noise to the original sample image.

[0272] During the output of the second sample target image, the fourth face in the second sample target image is generated gradually. It can be understood that the output second sample target image is obtained through continuous optimization. As the number of generation times increases, the facial feature information and semantic information of the fourth face in the second sample target image gradually increase. The more personalized information the fourth face has, the more recognizable the face becomes.

[0273] In each generation, the mask ratio of the sample mask image is negatively correlated with the number of generation steps, and the mask ratio is also negatively correlated with the amount of semantic information in the original sample image. This can be understood as follows: as the number of generation steps gradually increases, the mask ratio gradually decreases, while the amount of semantic information in the original image gradually increases.

[0274] Steps 602 to 604 above describe the process of “inputting at least one original sample image into a sample machine learning model and outputting at least one second sample target image”.

[0275] 605. Keep the model parameters of the sample image generation network unchanged, update the model parameters of the sample face control network until the difference between at least one second sample target image and at least one first sample target image satisfies the second condition, and obtain the trained first machine learning model.

[0276] The system determines whether the difference between the second sample target image and the first sample target image satisfies the second condition. Specifically, it determines whether the difference value between the second sample target image and the first sample target image meets a threshold. If the difference value does not exceed the threshold, it indicates that the similarity between the fourth face in the second sample target image (trained output) and the fourth face in the expected first sample target image is high. In other words, the sample machine learning model reconstructs or restores the fourth face with good accuracy, thus satisfying the second condition. The sample machine learning model is then used as the first trained machine learning model. If the difference value exceeds the threshold, it indicates that the similarity between the fourth face in the second sample target image (trained output) and the fourth face in the expected first sample target image is low. In other words, the sample machine learning model reconstructs or restores the fourth face with poor accuracy, thus not satisfying the second condition. The sample machine learning model is then updated until the difference between the output second sample target image and the first sample target image satisfies the second condition. The updated sample machine learning model is then used as the first trained machine learning model. Optionally, a loss function can be used to measure the difference between the second sample target image and the first sample target image. This loss function can be the mean square error, cross-entropy, logarithmic, exponential, or other loss functions. For details, please refer to the aforementioned terminology section, which will not be elaborated here.

[0277] Optionally, the parameters of the sample machine learning model can be updated using the backpropagation algorithm. For details, please refer to the aforementioned terminology section, which will not be repeated here.

[0278] It should be noted that the purpose of using the original image to train the first machine learning model in step 402 is to generalize the representation ability of the first face in the original image. Figure 6 The purpose of the illustrated embodiment is to train a sample machine learning model to obtain a first machine learning model. Figure 3The decoupled training module can decouple the two training processes. For example, when training the first machine learning model, the model parameters of the first face control network are kept unchanged, while the model parameters of the first image generation network are updated. This can be understood as training the first machine learning model mainly training the first image generation network, which can control semantic latent variables and multi-scale conditional features to better preserve the semantic and facial feature information of the face. Similarly, when training the sample machine learning model, the model parameters of the sample image generation network are kept unchanged, while the model parameters of the sample face control network are updated. This can be understood as training the sample machine learning model mainly training the sample face control network to better preserve the facial feature information of the face.

[0279] In summary, at least one original sample image and at least one first sample target image are obtained. The at least one original sample image is input into the sample machine learning model, which outputs at least one second sample target image. The model parameters of the sample image generation network are kept unchanged, and the model parameters of the sample face control network are updated until the difference between at least one second sample target image and at least one first sample target image satisfies the second condition, thus obtaining the trained first machine learning model.

[0280] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. Figure 7 As shown, the image processing apparatus 700 includes an acquisition unit 701, an update unit 702, and an output unit 703.

[0281] The acquisition unit 701 is used to acquire an original image and first parameter information. The original image includes a first face, and the first parameter information includes parameters for editing the first face in the original image.

[0282] The update unit 702 is used to update the first machine learning model based on the original image to obtain an optimized second machine learning model. The second machine learning model is used to preserve the semantic information of the first face in the original image.

[0283] The output unit 703 is used to edit the first face in the original image based on the first parameter information and the second machine learning model, and output the first target image.

[0284] In one possible implementation, the first machine learning model includes a first image generation network and a first face control network. The first face control network is used to determine a first multi-scale conditional feature, which is used to describe the facial feature information of the first face in the original image. The first image generation network is used to generate a second target image. The second machine learning model includes a second image generation network and a second face control network. The second face control network is used to determine a second multi-scale conditional feature, which is used to describe the facial feature information of the edited first face. The second image generation network is used to generate a first target image, which is obtained by editing the first face in the original image.

[0285] In one possible implementation, the first image generation network includes a first semantic encoding network and a first conditional diffusion decoding network. The first semantic encoding network is used to determine a first semantic latent variable, which is used to describe the semantic information of a first face in the original image. The first conditional diffusion decoding network is used to output a second target image. The second image generation network includes a second semantic encoding network and a second conditional diffusion decoding network. The second semantic encoding network is used to determine a second semantic latent variable, which is used to describe the semantic information of the edited first face. The second conditional diffusion decoding network is used to output the first target image.

[0286] In one possible implementation, the update unit 702 is specifically used to input the original image into the first machine learning model and output the second target image; keep the model parameters of the first face control network in the first machine learning model unchanged, and update the model parameters of the first image generation network in the first machine learning model until the difference between the original image and the second target image satisfies the first condition, thereby obtaining the optimized second machine learning model.

[0287] In one possible implementation, the output unit 703 is specifically used to edit the first face in the original image based on the first parameter information to obtain an intermediate image of the first face; input the intermediate image of the first face into the second face control network in the second machine learning model to obtain the second multi-scale conditional features; input the mask image and the second multi-scale conditional features into the second image generation network in the second machine learning model to output the first target image.

[0288] In one possible implementation, the mask image is obtained by masking the original image. The masking rate of the mask image is negatively correlated with the number of masking steps and negatively correlated with the amount of semantic information in the original image.

[0289] In one possible implementation, the output unit 703 is specifically used to input the original image into the face estimation network of the second machine learning model to obtain second parameter information, which is the parameter information before editing the facial features of the first face; based on the first parameter information and the second parameter information, the first face in the original image is edited to obtain the displacement map, the surface normal map, and the lighting geometry map of the first face; the displacement map, the surface normal map, and the lighting geometry map of the first face are superimposed to obtain the intermediate image of the first face.

[0290] In one possible implementation, the first parameter information includes first face parameters, the second parameter information includes second face parameters and personal feature vectors, the displacement map is obtained based on the first face parameters, the second face parameters and personal feature vectors, the surface normal map is obtained based on the first parameter information, the second parameter information, the face parameterization model, the normal calculation function and the reflection renderer, and the lighting geometry map is used to describe the lighting conditions of the first face.

[0291] In one possible implementation, the output unit 703 is specifically used to input the mask image into the second semantic coding network in the second image generation network to obtain the second semantic latent variable; and to input the noisy image, the second semantic latent variable, and the second multi-scale conditional features into the second conditional diffusion decoding network in the second image generation network to output the first target image.

[0292] In one possible implementation, the noisy image is a random Gaussian noise image with a standard normal distribution.

[0293] In one possible implementation, the acquisition unit 701 is specifically used to acquire first parameter information in response to an operation on the original image. The first parameter information includes first face parameters of the first face, first camera parameters, and first illumination parameters. The first face parameters include first shape parameters, first pose parameters, and first expression parameters.

[0294] In one possible implementation, the acquisition unit 701 is specifically used to acquire text attribute information of the first face in response to the user's input operation. The text attribute information is used to describe the semantic information of the first face and / or determine the second semantic latent variable of the first face.

[0295] In one possible implementation, the output unit 703 is used to process the original image using the third parameter information through the second machine learning model to output a third target image. The third parameter information includes the parameter information of the facial features of the second face. The third target image is obtained by replacing the first face in the original image with the second face.

[0296] In one possible implementation, the output unit 703 is used to process the reference video using the second parameter information through a second machine learning model and output a target video, wherein the reference video includes a third face and the target video is obtained by replacing the third face in the reference video with a first face.

[0297] In one possible implementation, the face control network is the Exp-FaceNet network, the image generation network is the Diff-AE network, the face estimation network is the EMOCA network, and the face parameterization model is the FLAME model.

[0298] The acquisition unit 701, update unit 702, and output unit 703 can all be implemented in software or in hardware. For example, the implementation of the acquisition unit 701 will be described below. Similarly, the implementation of the update unit 702 and output unit 703 can refer to the implementation of the acquisition unit 701.

[0299] As an example of a software functional unit, the acquisition unit 701 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the acquisition unit 701 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0300] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0301] As an example of a hardware functional unit, the acquisition unit 701 may include at least one computing device, such as a server. Alternatively, the acquisition unit 701 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0302] The multiple computing devices included in the acquisition unit 701 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the acquisition unit 701 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the acquisition unit 701 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0303] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of the model training device provided in an embodiment of this application. Figure 8 As shown, the model training device 800 includes an acquisition unit 801 and a training unit 802.

[0304] The acquisition unit 801 is used to acquire at least one original sample image and at least one first sample target image. The original sample image includes a fourth face, and the first sample target image is the image that the sample machine learning model expects to output based on the original sample image.

[0305] Training unit 802 is used to train a sample machine learning model using at least one original sample image and at least one first sample target image to obtain a first machine learning model. The sample machine learning model is used to reconstruct the fourth face based on the facial feature information and semantic information of the fourth face in the original sample image.

[0306] In one possible implementation, the sample machine learning model includes a sample image generation network and a sample face control network. The sample face control network is used to determine the multi-scale conditional features of the samples. The multi-scale conditional features of the samples are used to describe the facial feature information of the fourth face in the original sample image. The sample image generation network is used to generate the second sample target image.

[0307] In one possible implementation, the sample image generation network includes a sample semantic encoding network and a sample conditional diffusion decoding network. The sample semantic encoding network is used to determine sample semantic latent variables, which are used to describe the semantic information of the fourth face in the original sample image. The sample conditional diffusion decoding network is used to output the second sample target image.

[0308] In one possible implementation, the training unit 802 is specifically used to input at least one original sample image into the sample machine learning model and output at least one second sample target image; keep the model parameters of the sample image generation network in the sample machine learning model unchanged, update the model parameters of the sample face control network in the sample machine learning model until the difference between at least one second sample target image and at least one first sample target image satisfies the second condition, and obtain the trained first machine learning model.

[0309] In one possible implementation, the training unit 802 is specifically used to determine at least one intermediate image, which is obtained by processing at least one sample original image using a sample machine learning model; inputting the at least one intermediate image into the sample face control network in the sample machine learning model to obtain at least one sample multi-scale conditional feature; inputting at least one sample mask image and at least one sample multi-scale conditional feature into the sample image generation network in the sample machine learning model to output at least one second sample target image.

[0310] In one possible implementation, the sample mask image is obtained by masking the original sample image. The masking rate of the sample mask image is negatively correlated with the number of generation steps of the sample mask image, and the masking rate of the sample mask image is negatively correlated with the amount of semantic information in the original sample image.

[0311] In one possible implementation, the training unit 802 is specifically used to input at least one original sample image into the face estimation network of the sample machine learning model to obtain at least one sample parameter information, which includes parameter information of the facial features of the fourth face; based on the at least one sample parameter information, to obtain at least one displacement map, at least one surface normal map, and at least one lighting geometry map of the fourth face; and to superimpose the at least one displacement map, at least one surface normal map, and at least one lighting geometry map of the fourth face to obtain an intermediate image of at least one fourth face.

[0312] In one possible implementation, the sample parameter information includes sample face parameters, sample camera parameters, sample lighting parameters, and sample personal feature vectors. The sample face parameters include sample shape parameters, sample pose parameters, and sample expression parameters. The displacement map of the fourth face is obtained based on the sample face parameters and sample personal feature vectors. The surface normal map of the fourth face is obtained based on the sample parameter information, the face parameterization model, the normal calculation function, and the reflection renderer. The lighting geometry map is used to describe the lighting conditions of the fourth face.

[0313] In one possible implementation, the training unit 802 is specifically used to input at least one sample mask image into the sample semantic encoding network in the sample image generation network to obtain at least one sample semantic latent variable; and to input at least one sample noise image, at least one sample semantic latent variable and at least one sample multi-scale conditional feature into the sample conditional diffusion decoding network in the sample image generation network to output at least one second sample target image.

[0314] In one possible implementation, the sample noise image is obtained by adding random noise to the original sample image.

[0315] Both the acquisition unit 801 and the training unit 802 can be implemented in software or in hardware. The implementation methods of the acquisition unit 801 and the training unit 802 can be referred to the foregoing. Figure 7 The implementation of the acquisition unit 701 will not be described in detail here.

[0316] Please see Figure 9 , Figure 9 This is a schematic diagram of a computing device provided in an embodiment of this application. The computing device 900 includes a processor 901, a communication interface 902, a bus 903, and a memory 904. The processor 901, the communication interface 902, and the memory 904 communicate with each other via the bus 903. In practical applications, communication can also be achieved through other means such as wireless transmission; the specific method is not limited here.

[0317] The computing device 900 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memory in the computing device 900.

[0318] Processor 901 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0319] The communication interface 902 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 900 and other devices or communication networks.

[0320] The 903 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 9 The bus 903 is represented by only one line, but this does not mean that there is only one bus or one type of bus. The bus 903 may include a path for transmitting information between various components of the computing device 900 (e.g., memory 904, processor 901, communication interface 902).

[0321] Memory 904 may include volatile memory, such as random access memory (RAM). Memory 904 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0322] The memory 904 stores executable program code. The processor 901 executes this executable program code to implement the functions of the acquisition unit 701, update unit 702, and output unit 703 in the aforementioned image processing apparatus, thereby implementing the image processing method. Alternatively, the processor 901 executes the executable program code to implement the functions of the acquisition unit 801 and training unit 802 in the aforementioned model training apparatus, thereby implementing the model training method. That is, the memory 904 stores instructions for executing the image processing method or the model training method.

[0323] This application also provides a computing device cluster, which includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some optional embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0324] Please see Figure 10 and Figure 11 , Figure 10 and Figure 11 These are all schematic diagrams of the computing device clusters provided in the embodiments of this application.

[0325] like Figure 10 As shown, the computing device cluster includes at least one computing device 900. The memory 904 in one or more computing devices 900 within the computing device cluster may store the same instructions for executing the image processing methods and / or model training methods provided in the embodiments of this application.

[0326] In some possible implementations, the memory 904 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing image processing methods and / or model training methods. In other words, the combination of the memory 904 of one or more computing devices can jointly execute instructions for image processing methods, or jointly execute instructions for model training methods, or jointly execute instructions for both image processing methods and model training methods.

[0327] It should be noted that the memory 904 in different computing devices 900 within the computing device cluster can store different instructions, which are used to execute parts of the functions of the image processing device and / or the model training device, respectively. That is, the instructions stored in the memory 904 of different computing devices 900 can implement the functions of one or more units among the aforementioned image processing device's acquisition unit 701, update unit 702, and output unit 703, and / or the functions of one or more units among the model training device's acquisition unit 801 and training unit 802.

[0328] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN). Figure 11 This illustrates one possible implementation of a computing device cluster-based model training method. For example... Figure 11 As shown, the two computing devices 900A and 900B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 904 in computing device 900A stores instructions for executing the functions of the acquisition unit 801. Simultaneously, the memory 904 in computing device 900B stores instructions for executing the functions of the training unit 802.

[0329] Figure 11 The connection method between the computing device clusters shown can be as follows: considering that a large amount of data needs to be stored in the model training method provided in this application, the function of the acquisition unit 801 is to be performed by the computing device 900A, and the function of the training unit 802 is to be performed by the computing device 900B.

[0330] It should be understood that Figure 11 The functions of the computing device 900A shown can also be performed by multiple computing devices 900. Similarly, the functions of the computing device 900B can also be performed by multiple computing devices 900.

[0331] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computer device, it causes the at least one computer device to perform the aforementioned image processing method and / or model training method.

[0332] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the aforementioned image processing method and / or model training method.

[0333] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0334] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized in that, include: Obtain an original image and first parameter information, wherein the original image includes a first face and the first parameter information includes parameters for editing the first face in the original image; Based on the original image, the first machine learning model is updated to obtain an optimized second machine learning model; Based on the first parameter information and the second machine learning model, the first face in the original image is edited to output the first target image.

2. The method according to claim 1, characterized in that, The first machine learning model includes a first image generation network and a first face control network. The first face control network is used to determine a first multi-scale conditional feature, which is used to describe the facial feature information of a first face in the original image. The first image generation network is used to generate a second target image. The second machine learning model includes a second image generation network and a second face control network. The second face control network is used to determine a second multi-scale conditional feature, which is used to describe the facial feature information of an edited first face. The second image generation network is used to generate a first target image, which is obtained by editing the first face in the original image.

3. The method according to claim 2, characterized in that, The first image generation network includes a first semantic encoding network and a first conditional diffusion decoding network. The first semantic encoding network is used to determine a first semantic latent variable, which is used to describe the semantic information of the first face in the original image. The first conditional diffusion decoding network is used to output the second target image. The second image generation network includes a second semantic encoding network and a second conditional diffusion decoding network. The second semantic encoding network is used to determine a second semantic latent variable, which is used to describe the semantic information of the edited first face. The second conditional diffusion decoding network is used to output the first target image.

4. The method according to any one of claims 1 to 3, characterized in that, The step of updating the first machine learning model based on the original image to obtain an optimized second machine learning model includes: The original image is input into the first machine learning model, which outputs the second target image. Keeping the model parameters of the first face control network in the first machine learning model unchanged, the model parameters of the first image generation network in the first machine learning model are updated until the difference between the original image and the second target image satisfies the first condition, thereby obtaining an optimized second machine learning model.

5. The method according to any one of claims 1 to 4, characterized in that, The step of editing the first face in the original image based on the first parameter information and the second machine learning model to output the first target image includes: Based on the first parameter information, the first face in the original image is edited to obtain an intermediate image of the first face; The intermediate image of the first face is input into the second face control network in the second machine learning model to obtain the second multi-scale conditional features; The mask image and the second multi-scale conditional features are input into the second image generation network in the second machine learning model to output the first target image.

6. The method according to claim 5, characterized in that, The mask image is obtained by masking the original image. The masking rate of the mask image is negatively correlated with the number of generation steps of the mask image, and the masking rate of the mask image is negatively correlated with the amount of semantic information in the original image.

7. The method according to claim 5 or 6, characterized in that, The step of editing the first face in the original image based on the first parameter information to obtain an intermediate image of the first face includes: The original image is input into the face estimation network of the second machine learning model to obtain the second parameter information, which is the parameter information before the facial features of the first face are edited; Based on the first parameter information and the second parameter information, the first face in the original image is edited to obtain the displacement map of the first face, the surface normal map of the first face, and the lighting geometry map of the first face. By overlaying the displacement map, surface normal map, and lighting geometry map of the first face, an intermediate image of the first face is obtained.

8. The method according to claim 7, characterized in that, The first parameter information includes a first face parameter, the second parameter information includes a second face parameter and a personal feature vector, the displacement map is obtained based on the first face parameter, the second face parameter and the personal feature vector, the surface normal map is obtained based on the first parameter information, the second parameter information, the face parameterization model, the normal calculation function and the reflection renderer, and the lighting geometry map is used to describe the lighting conditions of the first face.

9. The method according to any one of claims 5 to 8, characterized in that, The step of inputting the mask image and the second multi-scale conditional features into the second image generation network in the second machine learning model to output the first target image includes: The masked image is input into the second semantic coding network in the second image generation network to obtain the second semantic latent variable; The noisy image, the second semantic latent variable, and the second multi-scale conditional feature are input into the second conditional diffusion decoding network in the second image generation network to output the first target image.

10. The method according to claim 9, characterized in that, The noise image is a random Gaussian noise image with a standard normal distribution.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: In response to an operation on the original image, first parameter information is obtained, the first parameter information including first face parameters of the first face, first camera parameters and first illumination parameters, the first face parameters including first shape parameters, first pose parameters and first expression parameters.

12. The method according to any one of claims 1 to 11, characterized in that, The method further includes: In response to user input, the text attribute information of the first face is obtained, the text attribute information is used to describe the semantic information of the first face, and / or to determine the second semantic latent variable of the first face.

13. The method according to any one of claims 1 to 12, characterized in that, The method further includes: The original image is processed using the third parameter information through the second machine learning model to output a third target image. The third parameter information includes the facial feature parameter information of the second face. The third target image is obtained by replacing the first face in the original image with the second face.

14. The method according to any one of claims 1 to 13, characterized in that, The method further includes: The second machine learning model uses the second parameter information to process the reference video and output a target video. The reference video includes a third face, and the target video is obtained by replacing the third face in the reference video with the first face.

15. The method according to any one of claims 1 to 14, characterized in that, The face control network is the Exp-FaceNet network, the image generation network is the Diff-AE network, the face estimation network is the EMOCA network, and the face parameterization model is the FLAME model.

16. An image processing apparatus, characterized in that, include: An acquisition unit is used to acquire an original image and first parameter information, wherein the original image includes a first face and the first parameter information includes parameters for editing the first face in the original image; An update unit is used to update the first machine learning model based on the original image to obtain an optimized second machine learning model; The output unit is used to edit the first face in the original image based on the first parameter information and the second machine learning model, and output the first target image.

17. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 15.

18. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 15.

19. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 15.

Images