A method, system, electronic device and storage medium for generating multi-pose faces

Abstract

This application belongs to the field of computer vision image generation technology and discloses a method, system, electronic device, and storage medium for generating multi-pose facial images. The method first obtains a frontal image of a source face, a reference image of a target pose, and personalized editing prompts. Then, the IP-Adapter module is used to extract and inject identity features of the frontal image into a preset diffusion model. Simultaneously, the ControlNet module is used to generate and inject posture control conditions based on the target pose information into the diffusion model. Finally, the diffusion model combines the identity features, the posture control conditions, and text features corresponding to the optional text prompts to generate a multi-pose facial image that is consistent with the target pose and maintains the identity features of the source face through a step-by-step denoising process. By combining the precise posture control capabilities of ControlNet with the identity preservation and editing capabilities of IP-Adapter, and based on the stable generation characteristics of the diffusion model, the present invention achieves high-quality and highly controllable multi-pose facial generation.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision image generation, and in particular to a method, system, electronic device and storage medium for generating multi-poses of a human face using a diffusion model, a ControlNet module and an IP-Adapter module. Background Art

[0002] Face pose steering, which generates face images from different viewing angles from a given face image (usually a frontal image), is an important research direction in computer vision, with broad application prospects in virtual reality, augmented reality, video games, and identity authentication. Traditionally, methods based on generative adversarial networks (GANs) have been widely used for this task. GANs learn data distributions and generate new samples through adversarial training of a generator and a discriminator.

[0003] However, GAN-based methods have some inherent defects and deficiencies in the task of multi-pose face generation:

[0004] First, the training process of GAN is often unstable and prone to mode collapse, which causes the generator to only produce a limited number of samples with insufficient diversity, or the quality of the generated images fluctuates greatly, making it difficult to ensure the realism and details of the generated side face images.

[0005] Secondly, the GAN model has poor adaptability to complex scenes or situations with large changes in posture. When the face posture deflects significantly or the background is complex, the images generated by the GAN may show obvious artifacts, distortion, or loss of details.

[0006] Thirdly, traditional GAN ​​models have limited ability to fine-tune the attributes of generated faces, making it difficult to implement customized editing operations such as "putting on glasses", "changing hairstyle" or adjusting specific facial expressions, which limits their potential in interactive applications.

[0007] In recent years, the diffusion model, as an emerging generative model, has shown great potential in the field of image generation due to its advantages such as stable training, high-quality generated images, and rich details. The diffusion model generates samples by simulating the gradual addition and removal of noise from data, effectively avoiding the mode collapse problem commonly seen in GANs.

[0008] Therefore, how to utilize the advantages of the diffusion model and combine it with an effective control mechanism to develop a multi-pose face generation method that can generate high-quality, accurate poses, maintain identity, and support flexible editing is a technical problem that needs to be solved urgently. Summary of the Invention

[0009] The present invention aims to overcome the problems of unstable training, low generated image quality, imprecise posture control, easy loss of identity information, and limited editing capabilities in existing GAN-based multi-pose face generation methods, which make it difficult to achieve high-quality, high-control accuracy, and high-editing freedom multi-pose face conversion. A method, system, electronic device, and storage medium for multi-pose face generation based on ControlNet and IP-Adapter are provided.

[0010] In a first aspect, the present invention provides a method for removing moiré from a high-resolution image, comprising the following steps:

[0011] S1: Obtain the source frontal face image, target posture reference image and personalized editing prompt words;

[0012] S2: extracting profile key points from the target pose reference image, wherein the profile key points are used to represent the target face pose;

[0013] S3: Inputting the source frontal face image into an image encoder to extract identity features, wherein the identity features are used to maintain identity information consistent with the source frontal face image in the generated face image;

[0014] S4: inputting the personalized editing prompt words into a text encoder to extract text features, wherein the text features are used to perform personalized editing on the generated facial image;

[0015] S5: Inputting the profile key points into a ControlNet module to generate a posture control condition, wherein the posture control condition is used to control the posture of the generated face image;

[0016] S6: The identity features, the text features and the posture control conditions are input into a pre-trained diffusion model, and a target face image is generated through a denoising process. The target face image has an identity consistent with the source frontal face image, a target face posture represented by the side face key points, and integrates the editing effect of the personalized editing prompt words.

[0017] As an optional implementation of the first aspect of the present application, in step S6, the identity feature and the text feature are input into the diffusion model through a decoupled cross-attention mechanism, and the decoupled cross-attention mechanism sets independent cross-attention layers for the identity feature and the text feature respectively.

[0018] As an optional implementation of the first aspect of the present application, the ControlNet module includes a trainable copy and a locked copy corresponding to the network block of the diffusion model, and the parameters of the locked copy are frozen during the training process to retain the original generation capability of the diffusion model.

[0019] As an optional implementation of the first aspect of the present application, the diffusion model is a Stable Diffusion model.

[0020] As an optional implementation of the first aspect of the present application, the side face key points include 83 key points that characterize the facial features and facial contours.

[0021] As an optional implementation of the first aspect of the present application, the method is implemented through a pre-trained model, and the training process of the model includes the following steps: constructing training sample pairs, each training sample pair including: a source target front face image, a source target side face image, side face key points extracted from the source target side face image, and personalized face editing prompts related to the source target front face image; inputting the training sample pairs into the model to be trained, wherein the source target front face image is used to extract identity features, the side face key points are used to generate posture control conditions, and the personalized face editing prompts are used to extract text features; inputting the identity features, posture control conditions and text features into a diffusion model, and optimizing the parameters of the model to be trained based on the loss between the image generated by the diffusion model and the source target side face image.

[0022] As an optional implementation of the first aspect of the present application, the training sample pairs are derived from the Multi-Pie dataset and the LMHQ dataset.

[0023] In a second aspect, an embodiment of the present application provides a face multi-pose generation system, comprising:

[0024] An acquisition module is used to obtain a source frontal face image, a target posture reference image, and personalized editing prompt words;

[0025] A key point extraction module is used to extract side face key points from the target posture reference image, wherein the side face key points are used to represent the target face posture;

[0026] A feature extraction module includes an image encoder and a text encoder; the source frontal face image is input into the image encoder to extract identity features, the identity features being used to maintain identity information consistent with the source frontal face image in the generated face image; the personalized editing prompt words are input into a text encoder to extract text features, the text features being used to perform personalized editing on the generated face image;

[0027] a posture control module, configured to input the profile key points into a ControlNet module to generate posture control conditions, wherein the posture control conditions are used to control the posture of the generated facial image;

[0028] An image generation module is used to input the identity features, the text features and the posture control conditions into a pre-trained diffusion model, and generate a target facial image through a denoising process. The target facial image has an identity consistent with the source frontal facial image, a target facial posture represented by the side face key points, and incorporates the editing effect of the personalized editing prompt words.

[0029] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein when the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented.

[0030] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0031] Compared with the existing technology, the present invention proposes a method for generating multi-pose faces, which has the following significant advantages:

[0032] (1) High Generation Quality and Precise Control: This invention combines ControlNet and IP-Adapter. ControlNet uses facial key points to precisely control the pose and angle of the generated face, while IP-Adapter extracts identity features from the source frontal face image, ensuring the identity of the generated face is consistent with that of the source face. This multi-condition control mechanism ensures that the generated face image not only has accurate pose, but also maintains high identity information and realistic details.

[0033] (2) Stable and efficient training: Based on the diffusion model, the training process of this invention is more stable than that of GAN, avoiding the problem of mode collapse. At the same time, the design of ControlNet locks the main parameters of the pre-trained diffusion model and only trains a small number of new parameters, which greatly reduces training cost and time and improves training efficiency.

[0034] (3) Flexible personalized editing capabilities: This invention introduces text prompts as one of the control conditions, allowing users to implement personalized editing such as "put on glasses" or "change hairstyle" while performing facial posture transformation. This greatly enhances the flexibility of the method and the breadth of its application scenarios.

[0035] (4) Innovative training data construction method: The present invention proposes a method for constructing training sample pairs consisting of four parts: the source target front face, the source target side face, the source target side face key points, and personalized editing prompts. This method can effectively train the model to simultaneously learn the three capabilities of posture control, identity preservation, and text editing, laying the foundation for high-quality multi-pose face generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of a method for generating multiple facial poses according to an embodiment of the present invention;

[0037] Figure 2 Schematic diagram of the forward noise addition process of the diffusion model in an embodiment of the present invention;

[0038] Figure 3 Schematic diagram of the reverse denoising process of the diffusion model in an embodiment of the present invention;

[0039] Figure 4 Schematic diagram of the structure of the multi-pose generation model of the face provided by an embodiment of the present invention;

[0040] Figure 5 This is a diagram showing the effect of generating multiple facial postures in an embodiment of the present invention;

[0041] Figure 6 1 is a structural diagram of a face multi-pose generation system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0043] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the application can be implemented in a sequence other than those illustrated or described here. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated before and after are in a kind of "or" relationship. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically limited.

[0044] The present invention provides a method for generating multiple facial postures based on ControlNet, IP-Adapter and Stable Diffusion model (such as Figure 1 The core idea of ​​this method is to leverage multi-source information to finely control the generation process of the diffusion model to achieve high-quality, high-fidelity, and editable face pose transfer.

[0045] First, the core technical components involved in the present invention are described:

[0046] 1. Diffusion Model. The Diffusion Model is a type of generative model that generates samples by simulating the gradual perturbation process of data and gradually removing noise. Unlike the Generative Adversarial Network (GAN), the generation process of the diffusion model is more clear. It first "destroys" the data by gradually adding noise to the training data, and then gradually restores the noise through a denoising process, ultimately generating high-quality, structurally complete images. The advantage of the diffusion model lies in the stability of its training and generation process, which can effectively avoid the common mode collapse and instability problems in GAN training. In practical applications, the diffusion model exhibits better image quality and diversity than GAN, especially in high-resolution image generation tasks. The diffusion model is divided into a forward process and a reverse process. In the forward process, images from the training set are T noises will be added, so that Conforms to the standard normal distribution. To be precise, "adding noise" does not mean adding noise to the image at the previous moment, but sampling a new image from a normal distribution whose mean is related to the image at the previous moment. As shown in the following formula, It is the image of the previous moment, is the image generated at this moment, which is generated from a mean and are sampled from the relevant normal distribution.

[0047] ,

[0048] in, represents a multivariate Gaussian distribution, represents the mean, represents the variance, Represents the identity matrix.

[0049] Most diffusion models set the normal distribution to this form: ( ) , we can get by deduction:

[0050] = ,

[0051] in , , . from The above inference can be simply described as follows: the noise addition formula can change the original image from slow to fast, making the image's final mean 0 and variance 1, turning it into a blurred image.

[0052] In the forward process, a T-step noise addition process is artificially set. In the reverse process, in order to reversely cancel each step of noise addition and turn a pure noise image back into the image in the dataset, the reverse process uses a UNet network. From a macro perspective, after a real image is noised in the above step, the noisy image is sent to the UNet network, and the noise is predicted by the network. Then, a comparison is made between the predicted noise and the real noise. or If we look at each step, we see Figure 2 As shown. Predict noise for the UNet network. After training a UNet network that predicts noise, we can predict the noise added at each step. Specifically speaking, the noise addition formula obtained from the above is = It can be inferred that , here is actually to get the feeling of the original image in one step, use the noise prediction network to predict how much noise the current noise image has added to the original image, and then directly reverse it to the original image in one step. However, the result is actually blurry, so a better idea is to start from the image with the most noise and push forward step by step, such as Figure 3 shown.

[0053] Starting from the noise map, the UNet network is used to predict , and then use this Become a , then This is then thrown into the UNet network to predict the new , and then deduced The previous step, and so on, until we get . It can be organized into a formula as + Z .

[0054] 2. Stable Diffusion. Stable Diffusion is an advanced diffusion model specifically designed for high-quality image generation tasks. Its core idea is to gradually transform an image from its initial state (noise image) into a target image. This process is learned through back diffusion during training. The Stable Diffusion model effectively reduces computational complexity and resource consumption by introducing the concept of "latent space". In Stable Diffusion, the original image is first mapped to a latent space, and then a diffusion operation is performed in this space. The trained model can gradually transform the noisy image into a clear image with a small amount of computation, thereby achieving image generation. The biggest advantage of this process is that it can generate high-quality, high-resolution images with low computing resources, and the generated images are rich in details and consistent in style. In the application of the present invention, the Stable Diffusion model is used to generate side face images at different angles (such as left and right) from frontal face images, greatly improving the accuracy and quality of face posture steering tasks.

[0055] 3. ControlNet. ControlNet is a control mechanism for the diffusion model that allows users to influence specific properties of the generated image by providing additional conditional inputs. For example, ControlNet can control the content of the generated image based on user-provided edge maps, keypoints, depth information, or pose information. In this invention, the introduction of ControlNet enables precise control of the generated profile face images by specifying target pose angles (e.g., 30° to the left, 60° to the right) for frontal face images. Through ControlNet, the diffusion model not only generates random images, but instead produces specific outputs based on specified control conditions. For example, in a face pose steering task, users can input a frontal face image and a simple text description (e.g., "30° to the left" or "60° to the right") to guide the generation of profile face images. ControlNet effectively embeds this conditional information into the generation process of the diffusion model, enabling the model to generate images of the desired facial poses based on different control signals. ControlNet is a neural network architecture that controls the diffusion model by adding additional conditions. The network structure is divided into two types: trainable and locked. The trainable portion learns based on the controllable portion. The locked part retains the original data of the Stable Diffusion model, so using a small amount of data to guide it can ensure that the pre-constraints can be fully learned while retaining the learning ability of the original diffusion model. ControlNet injects additional conditions into the neural network block. is a trainable counterpart whose parameters are , its function is to transform a feature map x Convert to another feature map y .

[0056] ,

[0057] Here x , y It is usually a two-dimensional feature map, usually with height and width and the corresponding number of channels.

[0058] ControlNet freezes the parameters of the original large model , clone the trainable model parameters , input features x , the input condition of the trainable part is c The zero convolution between the trainable part and the frozen part is represented by Z(·;·). The zero convolution is a 1*1 convolution layer with weights and biases initialized to 0; the entire ControlNet structure contains two zero convolutions with parameters of and The complete operation mode after adding ControlNet is: .

[0059] 4. IP-Adapter. IP-Adapter can embed the same image features through the newly added cross-attention layer. It mainly consists of two modules: an image encoder for extracting image features, and an adapter module with decoupled cross-attention for embedding image features into the pre-trained text-to-diffusion model. The cross-attention corresponding to the text is:

[0060] ,

[0061] in, represents the text attention weighted output feature, Q is the query vector (Query), is the transposed key vector, V is the value vector, is the dimensionality scaling factor (d is the feature dimension).

[0062] In the image encoder component, IP-Adapter uses the pre-trained CLIP image encoder to extract image features. Because CLIP maintains high consistency between image and text embedding spaces, it effectively expresses image content and style information. Throughout training, the parameters of the CLIP image encoder remain unchanged (frozen). Subsequently, a lightweight, trainable projection network is used to convert the image embedding features into a feature sequence of length N, whose dimensions match the text features used in the diffusion model. The cross-attention corresponding to the image is:

[0063] ，

[0064] in, represents the image attention weighted output feature, Q is the query vector (Query), is the transposed key of the image key vector, is the image value vector (Value), is the dimensionality scaling factor (d is the feature dimension).

[0065] In the original Stable Diffusion model, text information is injected into the UNet through the cross-attention mechanism. If image features and text features are directly concatenated and input into these attention layers, although image information can be introduced to a certain extent, the actual effect is not ideal. To this end, IP-Adapter proposes an improved method - decoupled cross-attention mechanism. Specifically, it adds a cross-attention layer for image features next to each original cross-attention layer in UNet, so that text features and image features are input into the model through their own independent attention paths. This structure avoids information interference and helps to make more effective use of the complementary information between image and text. The final decoupled cross-attention is:

[0066] ,

[0067] in, Represents the decoupled attention-weighted output features of image and text.

[0068] During training, only the IP-Adapter needs to be optimized, while the parameters of the pre-trained diffusion model and CLIP are kept fixed. The training set is also image-text pairs, and the training objective is the same as the original Stable Diffusion:

[0069] ,

[0070] in Represents a text condition, Represents image condition.

[0071] Since text cross-attention and image cross-attention are decoupled, the weight of the image condition can be adjusted during the inference phase to maintain a balance between text editability and image realism:

[0072] ，

[0073] in is a weighting factor, if =0, the model becomes the original text-to-image diffusion model.

[0074] In summary, the present invention uses ControlNet to control the direction and angle of the generated face through key points, uses IP-Adapter to ensure that the generated face after turning is consistent with the source face, and uses Stable Diffusion to perform personalized editing on the generated face. Ultimately, the generated face is turned, the direction is correct, the identity is consistent with the original face, and some personalized editing can be performed on the turned face.

[0075] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0076] Example 1

[0077] This embodiment provides a complete process of a method for generating multiple poses of a face. Figure 4 As shown in Figure 1, this method is based on a Stable Diffusion model that integrates ControlNet and IP-Adapter. This method is mainly divided into two stages: model training and model inference (generation).

[0078] Phase 1: Model Training

[0079] During the model training phase, the goal is to train a model that can understand three control signals: posture, identity, and text.

[0080] First, construct the training dataset.

[0081] The method uses facial images from two public datasets, Multi-Pie and LMHQ, and extracts 83 key points from the profile of these facial images. These key points correspond to key locations such as facial features and facial contours, accurately representing the facial features and contours of a face. After extracting all the key points, an original sample pair construction method for multi-pose face conversion is used to construct a sample pair consisting of a source target frontal face image, a source target profile face image, the source target profile face key points extracted from the source target profile face image, and a personalized face editing prompt.

[0082] For example, in the Multi-Pie dataset, each image object contains 13 pictures in different directions and angles, including 15°, 30°, 45°, 60°, 75°, 90° and the front face. The present invention selects the pictures with 30°, 60°, 90° and the front face for model training. These three angles can cover the training needs of small and large posture transformations of the face, which enhances the generalization ability of the model of the present invention. Moreover, since the entire dataset of Multi-Pie is not used, the training time and training cost are reduced, and excellent results are achieved with less cost. Six training sample pairs can be constructed for each face object in the training set. Taking the first image object in the training set as an example, the construction formula is as follows:

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089] In the above formula, Represents the first image object in the training set. The entire training set contains more than 200 image objects, including people of different genders, skin colors, and ages from different countries, which further enhances the generalization ability of the model. A frontal face photo representing the first image object, A 30° photo of the left side of the face representing the first image subject, It represents the key point image extracted from the 30° photo of the left side of the face of the first image object. The cue word representing the first image object. During training, the cue word only specifies the object's gender and whether they are wearing glasses. This cue word allows for personalized face editing. Refer to the explanations above for the other symbols in the formula.

[0090] Secondly, model construction and training.

[0091] like Figure 4 As shown, the model structure to be trained is as follows:

[0092] Input: a training sample pair.

[0093] IP-Adapter part: source target front face image ( Figure 4 The Ref image in is fed into an image encoder, which extracts high-dimensional identity features (such as the pre-trained CLIP image encoder).

[0094] Text encoder part: personalized face editing prompt words ( Figure 4 The text in the text is sent to a text encoder (such as CLIP text encoder) to extract text features.

[0095] ControlNet part: Source target side face key point map ( Figure 4 The Keypoints in the graph are sent to the ControlNet module to generate posture control conditions.

[0096] Diffusion model UNet: Identity features and text features are injected into each layer of the StableDiffusion UNet network through a decoupled cross-attention mechanism. At the same time, posture control conditions are also input into the UNet layer by layer.

[0097] Training Optimization: Based on these three input conditions, the model generates a facial image through a back-diffusion process. A loss (e.g., L1 or L2 loss) is calculated between the generated image and the source and target profile images from the training examples, and this loss is back-propagated to optimize the trainable parameters in the IP-Adapter and ControlNet. Notably, the parameters of the main part of Stable Diffusion and the CLIP encoder remain frozen during this process, significantly accelerating training and reducing computational resource requirements.

[0098] By performing multiple rounds (e.g., 200 rounds) of training on the constructed dataset, we eventually obtain a face multi-pose generation model that can collaboratively process three types of input signals.

[0099] Phase 2: Model Reasoning (Multi-pose Face Generation)

[0100] Once the model training is completed, it can be used to generate multiple facial poses.

[0101] First, input preparation. The user provides three inputs: (1) a source frontal face image, which is any frontal photo of the person whose pose is desired to be converted; (2) a target pose reference image, which is any face photo with the desired target pose. This photo is only used to extract the pose; the person's identity is irrelevant; and (3) a personalized editing prompt: text describing the additional attributes that the generated image wants to see, such as "smiling", "wearing sunglasses", "cyberpunk style", etc.

[0102] Second, the generation process:

[0103] (a) Extract facial key points from the target pose reference image provided by the user to obtain a target pose key point map;

[0104] (b) Input the source frontal face image into the IP-Adapter part (image encoder) of the trained model to extract identity features;

[0105] (c) Inputting personalized editing prompt words into the text encoder to extract text features;

[0106] (d) Input the target posture key point graph obtained in step (a) into the ControlNet module to generate posture control conditions;

[0107] (e) The above identity features, text features and posture control conditions are fed into the UNet of the model;

[0108] (f) The model starts from a random noise map and gradually generates the final target face image in a multi-step denoising process.

[0109] like Figure 5 As shown, the final generated target face image ( Figure 5 Right) successfully transforms the source front face image ( Figure 5 Left) and the pose represented by the target pose key point ( Figure 5 The generated faces maintain identity consistency while displaying the correct target pose, with high image quality and natural details.

[0110] It should be noted that this invention provides a powerful and easy-to-use tool. Without complex 3D modeling, users only need to provide a frontal photo, a pose reference photo, and a simple text description to generate high-quality, pose-free, customizable facial images. This tool has high application value in fields such as avatar creation, film and television production, and human-computer interaction.

[0111] In summary, in the embodiment of the present invention: (1) a training sample pair consisting of an original source target front face, source target side face, source target side face key points and personalized face editing prompt words is used, wherein the source target side face key points are extracted from the source target side face for the present invention. The model trained by the training sample pair can perform face multi-pose conversion by using the side face key points of any given other targets, and can also perform some customized face editing. (2) An original model for face multi-pose conversion is constructed. In addition to the original ability of the Stable Diffusion diffusion model to generate images based on text, the model uses the side face key points extracted from the face image and the front face image for maintaining identity information as control conditions to guide the image to generate a face image that maintains the source face identity information and has the correct turning direction and turning angle, thereby achieving high-quality face multi-pose conversion effects. In addition, due to the characteristics of ControlNet, the pre-trained network block of the source Stable Diffusion is frozen, so that the parameters therein do not participate in recalculation and updating, which improves the calculation speed of the model of the present invention and reduces the training cost of the model. (3) Based on the constructed diffusion model, the common mode collapse and instability problems in GAN training are avoided, the training process is smoother, and it can effectively generate diverse high-quality images.

[0112] Example 2

[0113] See also Figure 6 , shown is a schematic diagram of the structure of a face multi-pose generation system proposed in the second embodiment of the present application, which includes the following key modules:

[0114] An acquisition module 100 is used to acquire a source frontal face image, a target posture reference image, and personalized editing prompt words;

[0115] A key point extraction module 200 is used to extract side face key points from the target posture reference image, wherein the side face key points are used to represent the target face posture;

[0116] The feature extraction module 300 includes an image encoder and a text encoder. The source frontal face image is input into the image encoder to extract identity features, which are used to maintain identity information consistent with the source frontal face image in the generated face image. The personalized editing prompt is input into a text encoder to extract text features, which are used to perform personalized editing on the generated face image.

[0117] a posture control module 400 for inputting the profile key points into a ControlNet module to generate posture control conditions, wherein the posture control conditions are used to control the posture of the generated face image;

[0118] The image generation module 500 is used to input the identity features, the text features and the posture control conditions into a pre-trained diffusion model, and generate a target facial image through a denoising process. The target facial image has an identity consistent with the source frontal facial image, a target facial posture represented by the side face key points, and incorporates the editing effect of the personalized editing prompt words.

[0119] In the embodiments of the present application, a multi-pose facial generation system can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), while the non-mobile electronic device can be a server, network attached storage (NAS), personal computer (PC), etc., which are not specifically limited in the embodiments of the present application.

[0120] In the embodiments of the present application, a face multi-pose generation system can be a device having an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

[0121] The multi-pose generation system of a face provided by the embodiment of the present application can realize Figure 1 To avoid repetition, each process of implementing a method for generating multi-pose faces in a method embodiment will not be described here.

[0122] Optionally, an embodiment of the present application further provides an electronic device, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, the various processes of the above-mentioned embodiment of the method for generating multi-poses of a face are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be described here.

[0123] An embodiment of the present application further provides a readable storage medium having a program or instruction stored thereon. When the program or instruction is executed by a processor, the various processes of the embodiment of the above-mentioned method for generating multi-pose faces are implemented, and the same technical effects can be achieved. To avoid repetition, they are not described here.

[0124] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0125] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0126] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a more preferred embodiment. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of this application.

[0127] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for generating multi-pose faces, characterized in that: The following steps are involved: S1: Obtain the source frontal face image, target posture reference image and personalized editing prompt words; S2: extracting profile key points from the target pose reference image, wherein the profile key points are used to represent the target face pose; S3: Inputting the source frontal face image into an image encoder to extract identity features, wherein the identity features are used to maintain identity information consistent with the source frontal face image in the generated face image; S4: inputting the personalized editing prompt words into a text encoder to extract text features, wherein the text features are used to perform personalized editing on the generated facial image; S5: Inputting the profile key points into a ControlNet module to generate a posture control condition, wherein the posture control condition is used to control the posture of the generated face image; S6: The identity features, the text features and the posture control conditions are input into a pre-trained diffusion model, and a target face image is generated through a denoising process. The target face image has an identity consistent with the source frontal face image, a target face posture represented by the side face key points, and integrates the editing effect of the personalized editing prompt words.

2. The method according to claim 1, characterized in that In step S6, the identity feature and the text feature are input into the diffusion model through a decoupled cross-attention mechanism, and the decoupled cross-attention mechanism sets independent cross-attention layers for the identity feature and the text feature respectively.

3. The method according to claim 1, characterized in that The ControlNet module includes a trainable copy and a locked copy corresponding to the network block of the diffusion model, and the parameters of the locked copy are frozen during the training process to preserve the original generation capability of the diffusion model.

4. The method according to claim 1, wherein The diffusion model is a Stable Diffusion model.

5. The method according to claim 1, wherein The side face key points include 83 key points that characterize the facial features and facial contours.

6. The method according to any one of claims 1 to 5, characterized in that The method is implemented by a pre-trained model, and the training process of the model includes the following steps: Constructing training sample pairs, each training sample pair including: a source target front face image, a source target side face image, side face key points extracted from the source target side face image, and personalized face editing prompt words related to the source target front face image; Inputting the training sample pairs into the model to be trained, wherein the source and target frontal face images are used to extract identity features, the profile face key points are used to generate posture control conditions, and the personalized face editing prompt words are used to extract text features; The identity features, posture control conditions and text features are input into a diffusion model, and the parameters of the model to be trained are optimized according to the loss between the image generated by the diffusion model and the source target side face image.

7. The method according to claim 6, characterized in that The training sample pairs are derived from the Multi-Pie dataset and the LMHQ dataset.

8. A face multi-pose generation system, characterized in that: include: An acquisition module is used to obtain a source frontal face image, a target posture reference image, and personalized editing prompt words; A key point extraction module is used to extract side face key points from the target posture reference image, wherein the side face key points are used to represent the target face posture; A feature extraction module includes an image encoder and a text encoder; the source frontal face image is input into the image encoder to extract identity features, the identity features being used to maintain identity information consistent with the source frontal face image in the generated face image; the personalized editing prompt words are input into a text encoder to extract text features, the text features being used to perform personalized editing on the generated face image; a posture control module, configured to input the profile key points into a ControlNet module to generate posture control conditions, wherein the posture control conditions are used to control the posture of the generated facial image; An image generation module is used to input the identity features, the text features and the posture control conditions into a pre-trained diffusion model, and generate a target facial image through a denoising process. The target facial image has an identity consistent with the source frontal facial image, a target facial posture represented by the side face key points, and incorporates the editing effect of the personalized editing prompt words.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein when the program or instruction is executed by the processor, the steps of the method for generating multi-poses of a face are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the method for generating multi-poses of a face according to any one of claims 1 to 7 are implemented.

Images