Image generation method, electronic equipment and storage medium

Abstract

The invention discloses an image generation method. The method comprises the steps of obtaining a reference face image, a text cue word and source noisy data ZT; the text cue word is used for describing features of the customized image, and ZT is noisy data obtained after Gaussian noise of T time steps is gradually added to the reference face image through a diffusion model; generating a lexical element embedding vector based on the reference face image and the text cue word, and inputting the lexical element embedding vector into a de-noising device of the diffusion model to obtain a de-noising device taking the lexical element embedding vector as a condition; inputting the Zi into a de-noising device under the condition of lexical element embedding vectors for noise estimation to obtain first noise data belonging to i; according to the set i, the Zi and the reference face image, second noise data # imgabs0 # are generated, noisy data Zi-1 of the (i-1) th time step are generated according to # imgabs1 # and Zi, the initial value of i is T, the step is executed circularly after i is subtracted by 1 accumulatively until i is equal to 0, and noise-free data Z0 are obtained; and generating a customized image corresponding to the reference face image based on the Z0.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image generation method, an electronic device and a storage medium. Background Art

[0002] Customized character generation technology is a technology that uses artificial intelligence algorithms to generate realistic customized portrait images. At present, customized character generation technology is widely used, for example, in e-commerce advertising, artificial intelligence portraits, image animation, and virtual fitting, which can not only improve user experience but also meet personalized needs. However, since human facial identity involves more subtle semantics, it requires higher details and fidelity standards compared to general styles or objects. Therefore, it becomes crucial to generate customized images that can accurately retain the complex identity details of the face.

[0003] In the related art, a generative model based on a diffusion algorithm is mainly used. The user provides at least three reference face images and a unique prompt (special token) as the input of the generative model, and the generative model is fine-tuned to finally output a customized image. However, in actual usage scenarios, if the user can only provide a single reference face image, a customized image cannot be generated. Summary of the invention

[0004] The embodiments of the present application provide an image generation method, an electronic device, and a storage medium, which can generate a customized image based on a single reference face image provided by a user.

[0005] According to a first aspect of the present application, a method for generating an image is disclosed, the method comprising:

[0006] Get a reference face image, text prompt word and source noisy data Z T ; wherein the text prompt word is used to describe the characteristics of the custom image corresponding to the reference face image, the Z T The noisy data is obtained by gradually adding T time steps of Gaussian noise to the reference face image through the forward diffusion algorithm of the diffusion model, where T is an integer greater than 1;

[0007] Generate a word unit embedding vector based on the reference face image and the text prompt word, and input the word unit embedding vector into the denoiser of the diffusion model to obtain a denoiser conditioned on the word unit embedding vector;

[0008] Z i The denoiser conditioned on the word element embedding vector is input to perform noise estimation to obtain the first noise data ∈ i ; According to the ∈ i , the Z iand the reference face image to generate second noise data According to the and the Z i , generate the noisy data Z at the i-1th time step i-1 , the initial value of i is T, i is reduced by 1, and this step is repeated until i = 0, and the noise-free data Z is obtained. 0 ;

[0009] Based on the Z 0 , generating a customized image corresponding to the reference face image.

[0010] According to a second aspect of the present application, an image generating device is disclosed, the device comprising:

[0011] The acquisition module is used to obtain a reference face image, text prompt words and source noisy data Z T ; wherein the text prompt word is used to describe the characteristics of the custom image corresponding to the reference face image, the Z T The noisy data is obtained by gradually adding T time steps of Gaussian noise to the reference face image through the forward diffusion algorithm of the diffusion model, where T is an integer greater than 1;

[0012] A first generating module, configured to generate a word unit embedding vector based on the reference face image and the text prompt word;

[0013] A first processing module, configured to input the word unit embedding vector into a denoiser of the diffusion model to obtain a denoiser conditioned on the word unit embedding vector;

[0014] The second processing module is used to convert Z i The denoiser conditioned on the word element embedding vector is input to perform noise estimation to obtain the first noise data ∈ i ; According to the ∈ i , the Z i and the reference face image to generate second noise data According to the and the Z i , generate the noisy data Z at the i-1th time step i-1 , the initial value of i is T, i is reduced by 1, and this step is repeated until i = 0, and the noise-free data Z is obtained. 0 ;

[0015] The second generation module is used to generate a 0 , generating a customized image corresponding to the reference face image.

[0016] According to a third aspect of the present application, an electronic device is disclosed, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the image generation method in the first aspect.

[0017] According to a fourth aspect of the present application, a computer-readable storage medium is disclosed, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the image generation method in the first aspect is implemented.

[0018] According to a fifth aspect of the present application, a computer program product is disclosed, comprising a computer program / instruction, which implements the image generation method in the first aspect when executed by a processor.

[0019] In the embodiment of the present application, a reference face image, a text prompt word and a source noise data Z are obtained. T ; Among them, the text prompt word is used to describe the characteristics of the custom image corresponding to the reference face image, Z T The noisy data is obtained by gradually adding T time steps of Gaussian noise to the reference face image through the forward diffusion algorithm of the diffusion model, where T is an integer greater than 1; based on the reference face image and the text prompt word, a word unit embedding vector is generated, and the word unit embedding vector is input into the denoiser of the diffusion model to obtain a denoiser conditioned on the word unit embedding vector; Z i Input the denoiser conditioned on the word embedding vector to perform noise estimation and obtain the first noise data ∈ i ; According to ∈ i , Z i and the reference face image to generate the second noise data according to and Z i , generate the noisy data Z at the i-1th time step i-1 , the initial value of i is T, i is reduced by 1, and this step is repeated until i = 0, and the noise-free data Z is obtained. 0 ; Based on Z 0 , generate a customized image corresponding to the reference face image.

[0020] It can be seen that in the embodiment of the present application, when the user can only provide a single reference face image, a word-meta embedding vector is generated based on the single reference face image and the text prompt word, so as to fuse the facial identity features of the reference face image with the text features of the text prompt word, so as to guide the denoiser of the diffusion model to denoise the source noisy data generated based on the reference face image on a time-step basis. In the process of the denoiser denoising the source noisy data on a time-step basis, the reference face image is used as a guide to explicitly align the facial features of the reference face image with the facial features of the image denoised on a time-step basis, thereby ensuring the consistency between the identity of the denoised image and the identity of the reference face image, thereby achieving the generation of fine facial details and high-fidelity customized images based on the single reference face image provided by the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is an example diagram of the working principle of a diffusion model in the related art;

[0022] Figure 2 is a flow chart of an image generation method provided in an embodiment of the present application;

[0023] Figure 3 is a flowchart of an implementation of step 202 provided in an embodiment of the present application;

[0024] Figure 4 is a flowchart of a training process of an image encoder provided in an embodiment of the present application;

[0025] Figure 5 is an example diagram of an image generation method provided in an embodiment of the present application;

[0026] Figure 6 This is one of the flowcharts of an implementation of step 203 provided in the embodiment of the present application;

[0027] Figure 7 This is a second flowchart of an implementation method of step 203 provided in an embodiment of the present application;

[0028] Figure 8 This is a flowchart of an implementation method of step 203 provided in an embodiment of the present application;

[0029] Fig. 9 is an example diagram of a classifier guidance module provided in an embodiment of the present application;

[0030] Fig.10 This is a fourth flowchart of an implementation method of step 203 provided in an embodiment of the present application;

[0031] Fig.11 is a structural schematic diagram of an image generating device provided in an embodiment of the present application;

[0032] Fig.12 It is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0034] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0035] To facilitate understanding, some relevant concepts and application scenarios involved in the embodiments of the present application are first introduced below.

[0036] 1. Related concepts

[0037] Diffusion Probabilistic Models (DPMs) are a class of generative models that define a Markov chain of diffusion steps to slowly add random noise to the data, and then learn a reverse diffusion process to gradually transform the Gaussian noise distribution into the target data. Figure 1 As shown in Figure 2, the application of the diffusion model is usually divided into two stages: the forward diffusion process (indicated by the dotted line) and the reverse denoising process (indicated by the solid line). The input image is x 0 The forward diffusion process of the diffusion model is implemented by the forward diffusion algorithm q(x t |x t-1 ), and calculate x at each time step {1,2,…,t-1,t,…,T} 0 Add Gaussian noise and get the noisy data x in turn 1 ,x 2 ,…,x t-1 ,x t ,…,x T ; The reverse denoising process of the diffusion model is to denoise x at each time step {T,T-1,T-2,…,2,1} by the denoiser T Perform denoising to obtain the final output image.

[0038] Among them, the input image x 0 Equivalent to a single reference face image in the embodiment of the present application; x TThe final noisy data is equivalent to the source noisy data in the embodiment of the present application; the final output image is equivalent to the customized image in the embodiment of the present application.

[0039] The denoiser, also known as a denoising autoencoder, is a key component in the diffusion model that takes corrupted data as input and is trained to predict the uncorrupted original data as output. This process is similar to the inverse process of the diffusion model, that is, recovering the original data from the data with added noise. It recovers the original data from the noisy data through learning, which is one of the key mechanisms for the diffusion model to successfully generate clear images from noise.

[0040] Image encoder is a deep learning model for image processing. Its main purpose is to compress image data into low-dimensional code while retaining the key features of the image. Its working principle involves multiple layers, including convolutional layers, pooling layers, and fully connected layers. The convolutional layer extracts features by sliding the convolution kernel on the input data, the pooling layer reduces the amount of calculation by downsampling the feature map, and the fully connected layer maps the extracted features to the final output category. The characteristics of this model are its hierarchical structure, local perception, and weight sharing, which make it particularly suitable for processing image data. Image encoders can be applied to tasks such as data dimensionality reduction and feature extraction. Through training, the reconstructed data is made as close as possible to the original input data, thereby learning the important feature representation of the input data.

[0041] A text encoder is a tool that converts text into digital representations, which is used to process text data in machine learning models. The main functions of a text encoder include splitting text into subwords and converting each subword into digital representations so that machine learning models can understand and process these text data. Text encoders play an important role in the field of machine learning. For example, in the diffusion model, text encoders are used to convert text into embeddings, which are used to control the image generation process.

[0042] The face recognition model is a biometric technology that identifies a person based on their facial features. By inputting two face images into the face recognition model, the similarity between the two face images can be obtained.

[0043] Multilayer Perceptron (MLP) is a type of artificial neural network. It is a feedforward neural network model consisting of one or more intermediate layers, including an input layer and an output layer. MLP is usually used to solve classification and regression problems. It can handle nonlinear data and optimize weights and biases through the back propagation algorithm to reduce prediction errors. This network model has a wide range of applications, such as image recognition, speech recognition, natural language processing, and financial forecasting.

[0044] Variational Auto Encoder (VAE) is a deep generative model that combines the techniques of autoencoders and variational inference. VAE is able to generate new samples similar to the input data by learning the latent space distribution of the input data. Its working principle mainly involves two parts: encoder and decoder, where the encoder is responsible for converting the input data into a latent form, and the decoder tries to reconstruct the original data based on this latent representation. The advantage of this model is that it can generate new data samples similar to the training data, and because the latent space is continuous, the decoder can generate new data points that are seamlessly interpolated between the training data points.

[0045] Prompt is the content source of the image generated by the diffusion model. In the diffusion model, Prompt plays a vital role. It is not only the starting point or guide for the model to generate images, but also the key for the model to understand the input intent. By providing a clear and specific Prompt, the user can guide the model to generate images that meet expectations. This Prompt can be a question, a text description, a conversation, or any form of text input. The model will generate corresponding outputs based on the provided context and semantic information. For example, for a language model, a Prompt may be "The cat sat on the", and the model will predict the next words based on this Prompt and generate different outputs such as "mat", "chair", and "sofa".

[0046] 2. Application Scenarios

[0047] Customized character generation technology generally uses one or more images with consistent identities provided by users as conditional references for generating models, which is of great significance for tasks such as e-commerce advertising, AI portraits, image animation, and virtual fitting. However, human facial identity involves more subtle semantics and requires higher standards of detail and fidelity compared to general styles or objects. Therefore, there is still a lack of mature solutions for how to generate customized images that can accurately retain the complex identity details of the face.

[0048] The generation methods for customized images in related technologies can be divided into two types, depending on whether the model needs to be fine-tuned during the inference process. The first type of method usually uses 3-5 reference face images as input, and injects a specific facial identity into the generation model by optimizing a unique hint (special token) embedding. However, this type of method is difficult to deal with the situation of a single reference face image, and requires a long time of optimization, making it difficult to apply in actual scenarios.

[0049] The second type of method generally designs a face identity encoder to bypass fine-tuning during inference. However, due to the inherent defects of using pre-trained models, it can only achieve usable results for concepts that already exist in pre-trained models such as celebrities, but it is insufficient in creating high-fidelity, customized images of arbitrary identities.

[0050] In order to solve the above technical problems, the embodiments of the present application provide an image generating method, an electronic device and a storage medium.

[0051] Next, the image generation method provided in the embodiment of the present application is introduced in conjunction with the accompanying drawings.

[0052] It should be noted that the image generation method provided in the embodiment of the present application is applicable to electronic devices. In practical applications, the electronic devices may include: mobile terminals such as smart phones, tablet computers, personal digital assistants, attendance machines, etc., or the electronic devices may include: servers, laptops, desktops and other computer devices, which are not limited in the embodiment of the present application.

[0053] Figure 2 is a flow chart of an image generation method provided in an embodiment of the present application, such as Figure 2 As shown, the method may include the following steps: step 201, step 202, step 203 and step 204;

[0054] In step 201, a reference face image, a text prompt word and a source noisy data Z are obtained. T ; Among them, the text prompt word is used to describe the characteristics of the custom image corresponding to the reference face image, Z T It is the noisy data obtained by gradually adding T time steps of Gaussian noise to the reference face image through the forward diffusion algorithm of the diffusion model, where T is an integer greater than 1.

[0055] In the embodiment of the present application, the reference facial image can be a facial image of any person, not limited to a specific person or a specific group of people.

[0056] In the embodiment of the present application, the text prompt includes the user's expected information for the customized image. For example, the text prompt is "a man in a suit", which expresses that the user expects the portrait in the customized image to be the portrait of the reference face image and that the portrait is wearing a suit.

[0057] In the embodiment of the present application, noisy data refers to data with Gaussian noise. In the forward noise adding process of the diffusion model, the model continuously adds Gaussian noise to the reference face image. Each time step of Gaussian noise is added, a noisy data is obtained. As the time step increases, it is gradually transformed into completely random noise. This process simulates the diffusion process of data from order to disorder. The noisy data finally obtained in the forward noise adding process is the source noisy data Z T .

[0058] In the embodiment of the present application, in the reverse denoising process of the diffusion model, the model starts from random noise, gradually removes the noise, and finally generates data. The specific process is shown in the following steps 203 and 204. These two processes together constitute the learning mechanism of the diffusion model, so that the model can recover the original data from random noise.

[0059] In step 202, a word unit embedding vector is generated based on the reference face image and the text prompt word, and the word unit embedding vector is input into the denoiser of the diffusion model to obtain a denoiser conditioned on the word unit embedding vector.

[0060] Word Embeddings is a technology that maps words in a vocabulary to vector representations in the real number domain. Each word is assigned a real-valued vector so that semantically similar words are close in distance in the vector space. This representation method enables computers to better understand and process semantic relationships in language.

[0061] A denoiser conditioned on word embedding vectors refers to a technique that uses the concept of word embedding to map words in the vocabulary to vector representations in the real domain, and uses these vectors as conditions for denoising. By mapping words to vector space, word embedding technology makes semantically similar words close in distance in the vector space. This representation provides the denoiser with rich semantic information, enabling it to more accurately identify and correct noise in noisy data.

[0062] In an embodiment of the present application, a word-meta embedding vector is generated based on a single reference face image and a text prompt word so as to fuse the face identity features of the reference face image with the text features of the text prompt word, so as to guide the denoiser of the diffusion model to more accurately denoise the source noisy data generated based on the reference face image on a time-step basis.

[0063] In some embodiments, considering that the reference face image and the text prompt word are two types of information and cannot be directly fused, in order to facilitate fusion, the reference face image and the text prompt word can be converted into a vector space and fused in the vector space. Accordingly, Figure 3As shown, the above step 202 may include the following steps: step 301, step 302 and step 303;

[0064] In step 301, a reference face image is input into an image encoder for encoding to obtain a visual feature vector.

[0065] In the embodiment of the present application, the visual feature vector includes portrait feature information of the reference face image.

[0066] In the embodiment of the present application, the above-mentioned image encoder is a pre-trained image encoder.

[0067] Considering that training the image encoder alone cannot generate high-quality facial features, in some embodiments, the image encoder is trained in a specific application scenario. Specifically, the image encoder and the denoiser are connected together for training. During the training process, the network parameters of the denoiser are kept unchanged (i.e., the denoiser is fixed), and only the network parameters of the image encoder are updated. Accordingly, Figure 4 As shown, the training process of the above-mentioned image encoder includes the following steps: step 401, step 402, step 403 and step 404;

[0068] In step 401, a data set is obtained, wherein the data set includes a plurality of sample face images.

[0069] In the embodiment of the present application, in order to ensure the encoding capability of the facial features of the trained image encoder, the data set may include a large number of sample facial images of different persons.

[0070] In step 402, an initial backbone network corresponding to the image encoder is constructed.

[0071] In the embodiment of the present application, the initial backbone network is an encoder whose network parameters are not updated.

[0072] In step 403, the denoiser is connected to the initial backbone network to obtain a network to be trained.

[0073] In an embodiment of the present application, the initial backbone network and the denoiser of the image encoder are directly connected together, that is, the initial backbone network is in front and the denoiser is directly connected behind the initial backbone network, so that the denoiser can constrain the image encoder during the training process and improve the image encoder's ability to generate facial features.

[0074] In step 404, the network parameters of the denoiser in the network to be trained are fixed, and the network to be trained is trained using the data set until the model converges to obtain an image encoder.

[0075] In an embodiment of the present application, during the training process, sample face images in the data set are input into the initial backbone network in batches, the initial backbone network outputs face features, and the face features are further input into the denoiser. The denoiser outputs the final image, and the loss value between the image output by the denoiser and the sample face image is calculated. The network parameter value of the initial backbone network is adjusted in reverse according to the loss value, and the above process is repeated multiple times until the model converges to obtain an image encoder. At this time, the image encoder finally trained is more suitable for the denoising scenario of the denoiser.

[0076] It can be seen that in the embodiment of the present application, the image encoder and the denoiser are connected together for training. During the training process, the network parameters of the denoiser are kept unchanged (i.e., the denoiser is fixed), and only the network parameters of the image encoder are updated. The image sensor finally trained can provide a pivot for the generation of customized character images, thereby improving the fidelity of the customized image finally generated by the denoiser.

[0077] In step 302, the text prompt word is input into a text encoder for encoding to obtain a text feature vector.

[0078] In the embodiment of the present application, the text feature vector contains semantic feature information of the text prompt word.

[0079] In an embodiment of the present application, a reference face image is input into an image encoder for encoding processing, and a text prompt word is input into a text encoder for encoding processing, thereby ensuring that the encoded visual feature vector and text feature vector are in the same embedding representation space, thereby facilitating feature fusion.

[0080] In step 303, the visual feature vector and the text feature vector are concatenated in the channel dimension, and the vector concatenation result is input into a feature alignment network based on a multi-layer perceptron for fusion processing to obtain a word unit embedding vector.

[0081] The concatenation of features in the channel dimension, also known as the Concatenate operation, is a method of merging multiple feature maps in the channel dimension, which can make full use of features from different sources.

[0082] In an embodiment of the present application, the visual feature vector and the text feature vector are spliced ​​in the channel dimension to obtain a vector splicing result, which can fully utilize the features from the reference face image and the text prompt word.

[0083] In an embodiment of the present application, the vector concatenation result is input into a feature alignment network based on a multi-layer perceptron for fusion processing, which can further bridge the difference between visual encoding and text embedding, and finally obtain a fused and aligned word embedding vector.

[0084] It can be seen that in the embodiments of the present application, considering that the reference facial image and the text prompt words are two types of information and cannot be directly fused, in order to facilitate the fusion, the reference facial image and the text prompt words can be converted into a vector space, and spliced ​​and aligned in the vector space to achieve full fusion of the features from the reference facial image and the features of the text prompt words.

[0085] In step 203, Z i Input the denoiser conditioned on the word embedding vector to perform noise estimation and obtain the first noise data ∈ i ; According to ∈ i , Z i and the reference face image to generate the second noise data according to and Z i , generate the noisy data Z at the i-1th time step i-1 , the initial value of i is T, i is reduced by 1, and this step is repeated until i = 0, and the noise-free data Z is obtained. 0 .

[0086] In the embodiment of the present application, the noise-free data Z 0 is the final denoising result.

[0087] For ease of description, the denoiser conditioned on the word embedding vector will be referred to as the “denoiser” in the following.

[0088] In the embodiment of the present application, since the quality of the image obtained by one denoising is poor, the denoiser performs a denoising on the source noisy data Z. T When denoising, you cannot get a customized image after just one denoising. The denoiser needs to denoise the source noisy data Z step by step through multiple steps (each step is called a time step). T The noise in the image is removed, and finally a high-quality customized image is obtained.

[0089] In the embodiment of the present application, for ease of understanding, the single-step denoising of the denoiser is taken as an example, for example, i=T, the source noisy data Z T Input the denoiser conditioned on the word embedding vector to perform noise estimation and obtain the first noise data ∈ T , it can be seen that the first noise data ∈ T is the source noisy data Z estimated by the denoiser T Since the denoiser is a universal model, its ability to estimate noise for different types of noisy data is different. Therefore, the first noise data ∈ T May not be accurate enough. To address this problem, according to the first noise data ∈ T , source noisy data Z T and the reference face image to generate the second noise data By referring to the face image, the noise data is corrected. is the corrected source noise data Z T The noise in the image is corrected according to the reference face image, and the reference face image contains individual portrait features. Therefore, compared with the first noise data ∈ T , the second noise data More accurate. According to the second noise data and source noisy data Z T , generate the noisy data Z at the T-1 time step T-1 , to complete accurate denoising for one time step. Noisy data Z T-1 is the source noisy data Z T The data obtained after denoising for one time step is the source noisy data Z T After denoising for multiple time steps, noise-free data can be obtained, so Z T-1 It is still noisy data.

[0090] It should be noted that the above only takes the case of i=T as an example. The case of i<T is similar to the case of i=T and will not be described in detail here.

[0091] In the embodiment of the present application, for the source noisy data Z at the i-th time step i , the first noise data ∈ is obtained through the denoiser conditioned on the above word embedding vector i , the calculation process can be expressed as the following formula (1):

[0092] ∈ i＝ ∈ θ (Z i ,T,y) (1)

[0093] Among them, ∈ θ is the noise estimation formula of the denoiser, and y is the word embedding vector.

[0094] Through the DDIM sampler, the following formula (2), the second noise data and source noisy data Z i , generate the noisy data Z at the i-1th time step i-1 :

[0095]

[0096] Among them, α i is a predefined value associated with i in the sampler, The calculation method is:

[0097]

[0098] For example, T=50, the source noisy data is Z 50 , the denoiser needs to be 50 Perform denoising for 50 time steps and obtain the noisy data Z 49 , Z 48 ,……,Z 1 After denoising the last time step, we get the noise-free data Z 0 .

[0099] First, Z 50 Input the denoiser conditioned on the word embedding vector to perform noise estimation and obtain the first noise data ∈ 50 ; According to ∈ 50 , Z 50 and the reference face image to generate the second noise data according to and Z 50 , generate the noisy data Z at the 49th time step 49 At this point, the denoising of one time step is completed.

[0100] Afterwards, Z 49 Input the denoiser conditioned on the word embedding vector to perform noise estimation and obtain the first noise data ∈ 49 ; According to ∈ 49 , Z 49 and the reference face image to generate the second noise data according to and Z 49 , generate the noisy data Z at the 48th time step 48 At this point, the denoising of one time step is completed.

[0101] Then, Z 48 Input the denoiser conditioned on the word embedding vector to perform noise estimation and obtain the first noise data ∈ 48 ; According to ∈ 48 , Z 48 and the reference face image to generate the second noise data according to and Z 48 , generate the noisy data Z at the 47th time step 47 At this point, the denoising of one time step is completed.

[0102] And so on, finally Z 1 Input the denoiser conditioned on the word embedding vector to perform noise estimation and obtain the first noise data ∈ 1 ; According to ∈ 1 , Z 1 and the reference face image to generate the second noise data according to and Z1 , generate noise-free data Z at time step 0 0 . At this point, denoising of all time steps is completed.

[0103] In step 204, based on Z 0 , generate a customized image corresponding to the reference face image.

[0104] For example, the reference facial image is a portrait image of "Xiao Ming" (containing only the face, not the body), the text prompt is "a man in a suit", and the final generated customized image is an image of "Xiao Ming wearing a suit". In addition to Xiao Ming's face, the image also contains a body part, and the clothing on the body part is a suit.

[0105] In the embodiment of the present application, Z can be 0 Decoding is performed to obtain a customized image corresponding to the reference face image.

[0106] For example, Figure 5 As shown, various modules involved in the generation process of customized images are shown, namely, image encoder, text encoder, feature alignment network, denoiser and classifier guidance module, wherein the image encoder is used to encode the input reference face image to obtain a visual feature vector; the text encoder is used to encode the input text prompt word to obtain a text feature vector; the feature alignment network is used to fuse the splicing results of the visual feature vector and the text feature vector to obtain a word unit embedding vector; the denoiser is used to denoise the source noisy data time step by time step based on the word unit embedding vector; the classifier guidance module is used to correct the noise data output by the denoiser with the reference face image as a guide during the denoising process of each time step of the denoiser; the denoiser is also used to estimate the denoised image of the time step based on the corrected denoised data and noisy data of the same time step.

[0107] Compared with the customized image generation method in the related art, in the embodiment of the present application, the method for generating a customized image through the cooperation of the above modules has the following advantages:

[0108] Advantage 1: In actual use scenarios, users can only provide a single reference face image, and customized generation is difficult when computing resources are limited. In the embodiment of the present application, only the reference face image needs to be input, and the time-consuming and resource-consuming fine-tuning is avoided through the guidance of the zero-sample reasoning process.

[0109] Advantage 2: For the problem of insufficient control conditions, in the embodiment of the present application, there is no need to fine-tune the denoiser of the pre-trained diffusion model, so it can support plug-and-play spatial structure control models such as Control Net, while retaining the text control capability;

[0110] Advantage three, for the problem of poor fidelity and inconsistent identity in customized character generation, in the embodiments of the present application, by applying a reference facial image for guidance during inference, the facial features of the reference facial image can be explicitly aligned with the facial features of the generated image, ensuring the consistency of the identity of the customized image with the identity of the reference facial image.

[0111] It should be noted that zero-sample means that there is no data about the sample during training. The image encoder will be trained with a large number of face images, but during inference, it is impossible to have seen all the face images of everyone. For the reference face images that the image encoder has not seen, the denoiser can still generate a customized image that is consistent with the portrait features of the reference face image, so it is a zero-sample.

[0112] It should be noted that limited computing resources refers to the process of model fine-tuning. Fine-tuning actually involves the model training process, and the model training process is actually a process that is 3 to 4 times more expensive than the model inference process, because it needs to calculate how much the model parameters need to be updated and then perform the update. In addition, the overall features or intermediate features of the model need to be retained to support the optimization process. Therefore, compared with the inference process, the fine-tuning process involves a huge consumption of resources.

[0113] As can be seen from the above embodiment, in this embodiment, a reference face image, a text prompt word and a source noise data Z are obtained. T ; Among them, the text prompt word is used to describe the characteristics of the custom image corresponding to the reference face image, Z T The noisy data is obtained by gradually adding T time steps of Gaussian noise to the reference face image through the forward diffusion algorithm of the diffusion model, where T is an integer greater than 1; based on the reference face image and the text prompt word, a word unit embedding vector is generated, and the word unit embedding vector is input into the denoiser of the diffusion model to obtain a denoiser conditioned on the word unit embedding vector; Z i Input the denoiser conditioned on the word embedding vector to perform noise estimation and obtain the first noise data ∈ i ; According to ∈ i , Z i and the reference face image to generate the second noise data according to and Z i , generate the noisy data Z at the i-1th time step i-1 , the initial value of i is T, i is reduced by 1, and this step is repeated until i = 0, and the noise-free data Z is obtained. 0 ; Based on Z 0 , generate a customized image corresponding to the reference face image.

[0114] It can be seen that in the embodiment of the present application, when the user can only provide a single reference face image, a word-meta embedding vector is generated based on the single reference face image and the text prompt word, so as to fuse the facial identity features of the reference face image with the text features of the text prompt word, so as to guide the denoiser of the diffusion model to denoise the source noisy data generated based on the reference face image on a time-step basis. In the process of the denoiser denoising the source noisy data on a time-step basis, the reference face image is used as a guide to explicitly align the facial features of the reference face image with the facial features of the image denoised on a time-step basis, thereby ensuring the consistency between the identity of the denoised image and the identity of the reference face image, thereby achieving the generation of fine facial details and high-fidelity customized images based on the single reference face image provided by the user.

[0115] In some other embodiments provided in the present application, a face recognition model can be used to guide a classifier on the denoised image and the reference face image to optimize the similarity between the face identity of the denoised image and the face identity of the reference face image. Accordingly, Figure 6 As shown, the above step 203 may include the following steps: step 601, step 602 and step 603;

[0116] In step 601, according to ∈ i and Z i , generate the denoised image P i .

[0117] For example, the sampler, the following formula (3), the first noise data ∈ i and source noisy data Z i , generate the denoised image P i :

[0118]

[0119] Among them, α i is a predefined value associated with i in the sampler.

[0120] In step 602, P i The face recognition model is input with the reference face image for comparison and processing to obtain the portrait similarity S i .

[0121] In the embodiment of the present application, the above-mentioned face recognition model can be any face recognition model obtained by pre-training.

[0122] In the embodiment of the present application, the portrait similarity S i Used to characterize the denoised image P i The degree of difference between the portrait and the reference face image.

[0123] In step 603, according to ∈ i and S i , generate the second noise data

[0124] It is understandable that the denoised image P i The greater the difference between the portrait of the reference face image, the less the portrait of the final generated custom image is similar to the portrait of the reference face image. On the contrary, the denoised image P i The smaller the difference between the portrait and the reference face image, the more similar the portrait of the customized image finally generated is to the portrait of the reference face image. Therefore, in order to ensure that the portrait of the customized image finally generated is consistent with the portrait of the reference face image, it is necessary to ensure the accuracy of the noise data. Specifically, the portrait similarity S i , the first noise data ∈ output by the denoiser i Correction is performed to obtain the corrected noise data

[0125] For example, the following formula (4) can be used: i Similarity to the portrait S i , generate the second noise data

[0126]

[0127] Among them, ∈ θ is the noise estimation formula of the denoiser, q is the strength value without classifier guidance, u is the strength value guided by the classifier, is the empty set.

[0128] It can be seen that in the embodiment of the present application, the face recognition model can be used to perform classifier guidance on the denoised image and the reference face image, and the noise data output by the denoiser is corrected based on the similarity between the face identity of the denoised image and the face identity of the reference face image, so that the fine facial identity details of the custom image and the reference face image can be more effectively aligned.

[0129] In some other embodiments provided in the present application, considering that the denoiser includes a multi-layer network, each layer of the network will output features, and the features output by the previous layer of the network are used as the input of the next layer of the network. The researchers have found through experiments that the output features of a certain layer of the network in the denoiser are very consistent with the semantic information of the face. Based on this discovery, the classifier can be guided on the output features of the specific layer of the network in the denoiser to optimize the similarity between the face identity of the denoised image and the face identity of the reference face image. Accordingly, Figure 7 As shown, the above step 203 may include the following steps: step 701, step 702, step 703 and step 704;

[0130] In step 701, Z i After the noise estimation is performed on the denoiser conditioned on the word embedding vector, the output feature D of the Nth layer network of the denoiser is obtained. i ; Wherein, N is an integer greater than 1.

[0131] In the embodiment of the present application, the output features of the Nth layer network of the denoiser are very consistent with the semantic information of the face.

[0132] In the embodiment of the present application, it is found through experiments that the output features of the latter network layers of the denoiser are very consistent with the semantic information of the face. Accordingly, the value of N can be set according to the above experimental conditions, K / 2<N<K, wherein the denoiser includes a K-layer network, and K is an integer greater than 1.

[0133] It should be noted that the output feature of the Nth layer network of the denoiser is not an image visible to the human eye or a feature with a specific meaning (for example, color, brightness, texture, etc.), it is just a feature.

[0134] In step 702, after the reference face image is input into the denoiser for processing, the output feature D of the Nth layer network of the denoiser is obtained. 0 .

[0135] In step 703, D is calculated i With D 0 The feature similarity G between i .

[0136] In the embodiment of the present application, the feature similarity G i Used to characterize the degree of difference between noisy data and reference face images in the semantic dimension of the face.

[0137] In the embodiment of the present application, since the output features of the Nth layer network of the denoiser are very consistent with the semantic information of the face, i After the noise estimation is performed on the denoiser conditioned on the word embedding vector, the output feature D of the Nth layer network of the denoiser is obtained. i ; After the reference face image is input into the denoiser for processing, the output feature D of the Nth layer network of the denoiser is obtained 0 , by calculating the similarity between the output features of the noisy data and the reference face image in a specific network layer (i.e., N-layer network) of the denoiser, the degree of difference between the noisy data and the reference face image in the semantic dimension of the face is determined.

[0138] In step 704, according to ∈ i and G i , generate the second noise data

[0139] It can be understood that the lower the similarity between the output features of the noisy data and the reference face image in a specific network layer of the denoiser, the less similar the portrait of the final generated custom image is to the portrait of the reference face image, and the higher the similarity between the output features of the noisy data and the reference face image in the same network layer of the denoiser, the more similar the portrait of the final generated custom image is to the portrait of the reference face image. Therefore, in order to ensure that the portrait of the final generated custom image is consistent with the portrait of the reference face image, it is necessary to ensure the accuracy of the noise data. Specifically, the feature similarity G can be used to obtain the final generated custom image. i , the first noise data ∈ output by the denoiser i Correction is performed to obtain the corrected second noise data

[0140] Exemplarily, the following formula (5) can be used: i and feature similarity G i , generate the second noise data

[0141]

[0142] Among them, ∈ θ is the noise estimation formula of the denoiser, q is the strength value without classifier guidance, v is the strength value guided by the classifier, is the empty set.

[0143] It can be seen that in the embodiment of the present application, considering that the denoiser includes multiple layers of networks, each layer of the network will output features, and the features output by the previous layer of the network are used as the input of the next layer of the network. It is found through experiments that the output features of a certain layer of the network in the denoiser are very consistent with the semantic information of the face. Therefore, the classifier can be guided on the output features of the specific layer of the network in the denoiser. Based on the similarity between the facial identity of the denoised image and the facial identity of the reference facial image, the noise data output by the denoiser is corrected, so that the fine facial identity details of the custom image and the reference facial image can be more effectively aligned.

[0144] In some other embodiments provided in this application, it is possible to integrate Figure 6 and Figure 7 The solution of the embodiment shown, accordingly, as Figure 8 As shown, the above step 203 may include the following steps: step 801, step 802, step 803 and step 804;

[0145] In step 801, Z i After the noise estimation is performed on the denoiser conditioned on the word embedding vector, the output feature D of the Nth layer network of the denoiser is obtained. i; Wherein, N is an integer greater than 1.

[0146] The content of step 801 in the embodiment of the present application is the same as Figure 7 The content of step 701 in the illustrated embodiment is similar and will not be repeated here.

[0147] In step 802, according to ∈ i and Z i , generate the denoised image P i ; P i The face recognition model is input with the reference face image for comparison and processing to obtain the portrait similarity S i .

[0148] The content of step 802 in the embodiment of the present application is Figure 6 The contents of step 601 and step 602 in the illustrated embodiment are similar and will not be described in detail herein.

[0149] In step 803, after the reference face image is input into the denoiser for processing, the output feature D of the Nth layer network of the denoiser is obtained. 0 ; Calculate D i With D 0 The feature similarity G between i .

[0150] The content of step 803 in the embodiment of the present application is the same as Figure 7 The contents of step 702 and step 703 in the illustrated embodiment are similar and will not be described in detail herein.

[0151] In step 804, according to ∈ i , S i and G i , generate the second noise data

[0152] Exemplarily, the following formula (6) can be used: i , Portrait similarity S i and feature similarity G i , generate the second noise data

[0153]

[0154] Among them, ∈ θ is the noise estimation formula of the denoiser, q is the intensity value without classifier guidance, u and v are both the intensity values ​​guided by the classifier, is the empty set.

[0155] Exemplarily, the structure of the classifier guidance module is as follows Fig. 9As shown, it includes a face recognition model, a denoiser and a computing unit, wherein the face recognition model is used to compare the denoised image with the reference face image to obtain the portrait similarity; the denoiser is used to process the reference face image to obtain the output features of the Nth layer network of the denoiser; the computing unit is used to calculate the feature similarity between the intermediate features of the noisy data and the output features of the reference face image in the same network layer (i.e., N layer) of the denoiser, and based on the portrait similarity and the feature similarity, correct the noise data output by the denoiser.

[0156] It can be seen that in the embodiments of the present application, the similarity of the intermediate features output by the specific network layer of the denoiser between the noisy data and the reference face image, as well as the portrait similarity between the denoised image and the reference face image, can be used to correct the noisy data output by the denoiser, thereby more effectively aligning the fine facial identity details of the custom image and the reference face image.

[0157] In some other embodiments provided by the present application, considering that after denoising in the first few time steps of the denoiser, there is usually no face outline in the denoised image, and the portrait cannot be well located. As the denoiser continues to denoise, the face outline gradually appears in the denoised image, and the details of the face are increasingly rich and clear. Therefore, in order to reduce the amount of calculation and ensure the fidelity of the finally generated custom image, in the first few time steps of the denoiser, the denoising is not used. Figure 8 The scheme shown in the figure adopts Figure 8 The scheme shown, accordingly, Fig.10 As shown, the above step 203 may include the following steps: step 1001, step 1002, step 1003 and step 1004;

[0158] In step 1001, when TM≥i≥1, the output feature D of the Nth layer network of the denoiser is obtained. i , where M is an integer, 1<M<T.

[0159] For example, T=50, M=20, at which time the denoiser performs the following operation at and after the 30th time step: Figure 8 Steps shown.

[0160] In step 1002, according to ∈ i and Z i , generate the denoised image P i ; P i The face recognition model is input with the reference face image for comparison and processing to obtain the portrait similarity S i .

[0161] In step 1003, after the reference face image is input into the denoiser for processing, the output feature D of the Nth layer network of the denoiser is obtained. 0 ; Calculate D i With D 0 The feature similarity G between i .

[0162] In step 1004, according to ∈ i , S i and G i , generate the second noise data

[0163] The contents of steps 1001 to 1004 in the embodiment of the present application are Figure 6 , Figure 7 and Figure 8 The contents in the illustrated embodiments are similar and will not be described again here.

[0164] It can be seen that in the embodiments of the present application, after a specific time step of the denoiser, the similarity of the intermediate features output by the noisy data and the reference face image at a specific network layer of the denoiser, as well as the portrait similarity between the denoised image and the reference face image, can be used to correct the noisy data output by the denoiser, thereby more effectively aligning the fine facial identity details of the custom image and the reference face image while reducing the amount of computation.

[0165] Fig.11 is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application, such as Fig.11 As shown, the image generating device 1100 may include: a first acquisition module 1101, a first generating module 1102, a first processing module 1103, a second processing module 1104 and a second generating module 1105;

[0166] The first acquisition module 1101 is used to acquire a reference face image, a text prompt word and a source noisy data Z T ; wherein the text prompt word is used to describe the characteristics of the custom image corresponding to the reference face image, the Z T The noisy data is obtained by gradually adding T time steps of Gaussian noise to the reference face image through the forward diffusion algorithm of the diffusion model, where T is an integer greater than 1;

[0167] A first generating module 1102 is used to generate a word unit embedding vector based on the reference face image and the text prompt word;

[0168] A first processing module 1103 is used to input the word unit embedding vector into the denoiser of the diffusion model to obtain a denoiser conditioned on the word unit embedding vector;

[0169] The second processing module 1104 is used to convert Z i The denoiser conditioned on the word element embedding vector is input to perform noise estimation to obtain the first noise data ∈ i ; According to the ∈ i , the Z i and the reference face image to generate second noise data According to the and the Z i , generate the noisy data Z at the i-1th time step i-1 , the initial value of i is T, i is reduced by 1, and this step is repeated until i = 0, and the noise-free data Z is obtained. 0 ;

[0170] The second generating module 1105 is used to generate a 0 , generating a customized image corresponding to the reference face image.

[0171] As can be seen from the above embodiment, in this embodiment, a reference face image, a text prompt word and a source noise data Z are obtained. T ; Among them, the text prompt word is used to describe the characteristics of the custom image corresponding to the reference face image, Z T The noisy data is obtained by gradually adding T time steps of Gaussian noise to the reference face image through the forward diffusion algorithm of the diffusion model, where T is an integer greater than 1; based on the reference face image and the text prompt word, a word unit embedding vector is generated, and the word unit embedding vector is input into the denoiser of the diffusion model to obtain a denoiser conditioned on the word unit embedding vector; Z i Input the denoiser conditioned on the word embedding vector to perform noise estimation and obtain the first noise data ∈ i ; According to ∈ i , Z i and the reference face image to generate the second noise data according to and Z i , generate the noisy data Z at the i-1th time step i-1 , the initial value of i is T, i is reduced by 1, and this step is repeated until i = 0, and the noise-free data Z is obtained. 0 ; Based on Z 0 , generate a customized image corresponding to the reference face image.

[0172] It can be seen that in the embodiment of the present application, when the user can only provide a single reference face image, a word-meta embedding vector is generated based on the single reference face image and the text prompt word, so as to fuse the facial identity features of the reference face image with the text features of the text prompt word, so as to guide the denoiser of the diffusion model to denoise the source noisy data generated based on the reference face image on a time-step basis. In the process of the denoiser denoising the source noisy data on a time-step basis, the reference face image is used as a guide to explicitly align the facial features of the reference face image with the facial features of the image denoised on a time-step basis, thereby ensuring the consistency between the identity of the denoised image and the identity of the reference face image, thereby achieving the generation of fine facial details and high-fidelity customized images based on the single reference face image provided by the user.

[0173] Optionally, as an embodiment, the second processing module 1104 may be specifically configured to: i and the Z i , generate the denoised image P i ; The P i The reference face image is input into the face recognition model for comparison processing to obtain the portrait similarity S i ; According to the ∈ i and the S i , generate the second noise data

[0174] Optionally, as an embodiment, the image generating device 1100 may further include:

[0175] The second acquisition module is used to obtain the output feature D of the Nth layer network of the denoiser i , wherein N is an integer greater than 1;

[0176] The second processing module 1104 may be specifically configured to: input the reference face image into the denoiser for processing, and then obtain an output feature D of the Nth layer network of the denoiser. 0 ; Calculate the D i With the D 0 The feature similarity G between i ; According to the ∈ i and the G i , generate the second noise data

[0177] Optionally, as an embodiment, the image generating device 1100 may further include:

[0178] The third acquisition module is used to obtain the output feature D of the Nth layer network of the denoiser i , wherein N is an integer greater than 1;

[0179] The second processing module 1104 may be specifically configured to: i and the Z i , generate the denoised image P i ; The P i The reference face image is input into the face recognition model for comparison processing to obtain the portrait similarity S i After the reference face image is input into the denoiser for processing, the output feature D of the Nth layer network of the denoiser is obtained 0 ; Calculate the D i With the D 0 The feature similarity G between i ; According to the ∈ i , the S i and the G i , generate the second noise data

[0180] Optionally, as an embodiment, the third acquisition module may be specifically used to: when TM≥i≥1, acquire the output feature D of the Nth layer network of the denoiser i , wherein M is an integer, 1<M<T.

[0181] Optionally, as an embodiment, the denoiser comprises a K-layer network, wherein K / 2<N<K, and K is an integer greater than 1.

[0182] Optionally, as an embodiment, the first generation module 1102 can be specifically used to: input the reference face image into an image encoder for encoding processing to obtain a visual feature vector; input the text prompt word into a text encoder for encoding processing to obtain a text feature vector; splice the visual feature vector and the text feature vector in the channel dimension, and input the vector splicing result into a feature alignment network based on a multi-layer perceptron for fusion processing to obtain a word unit embedding vector.

[0183] Optionally, as an embodiment, the training process of the image encoder may include the following steps: obtaining a data set, wherein the data set includes multiple sample face images; constructing an initial backbone network corresponding to the image encoder; connecting the denoiser to the initial backbone network to obtain a network to be trained; fixing the network parameters of the denoiser in the network to be trained, performing model training on the network to be trained using the data set until the model converges, and obtaining the image encoder.

[0184] Any step in the embodiment of the image generation method provided in the present application and the specific operation in any step can be completed by the corresponding module in the image generation device. The process of the corresponding operation completed by each module in the image generation device refers to the process of the corresponding operation described in the embodiment of the image generation method. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0185] Fig.12 1 is a block diagram of an electronic device provided by an embodiment of the present application, the electronic device includes a processing component 1222, which further includes one or more processors, and a memory resource represented by a memory 1232, for storing instructions executable by the processing component 1222, such as an application. The application stored in the memory 1232 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1222 is configured to execute instructions to perform the above-mentioned image generation method.

[0186] The electronic device may also include a power supply component 1226 configured to perform power management of the electronic device, a wired or wireless network interface 1250 configured to connect the electronic device to a network, and an input / output (I / O) interface 1258. The electronic device may operate based on an operating system stored in the memory 1232, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or the like.

[0187] According to another embodiment of the present application, the present application further provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps in the image generation method described in any one of the above embodiments.

[0188] According to another embodiment of the present application, the present application further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the image generation method described in any of the above embodiments.

[0189] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0190] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0191] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0192] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0193] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0194] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0195] The above is a detailed introduction to an image generation method, electronic device and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image generation method, characterized in that: The method comprises: Get a reference face image, text prompt word and source noisy data Z T ; wherein the text prompt word is used to describe the characteristics of the custom image corresponding to the reference face image, the Z T The noisy data is obtained by gradually adding T time steps of Gaussian noise to the reference face image through the forward diffusion algorithm of the diffusion model, where T is an integer greater than 1; Generating a word unit embedding vector based on the reference face image and the text prompt word, and inputting the word unit embedding vector into the denoiser of the diffusion model to obtain a denoiser conditioned on the word unit embedding vector; Z i The denoiser conditioned on the word element embedding vector is input to perform noise estimation to obtain the first noise data ∈ i ; According to the ∈ i 、Z i and the reference face image to generate second noise data According to the and the Z i , generate the noisy data Z of the i-1th time step i-1 , the initial value of i is T, i is decremented by 1, and this step is repeated until i = 0, and the noise-free data Z0 is obtained; Based on Z0, a customized image corresponding to the reference facial image is generated.

2. The method according to claim 1, characterized in that According to the ∈ i 、Z i and the reference face image to generate second noise data include: According to the i and the Z i , generate the denoised image P i ; The P i The reference face image is input into the face recognition model for comparison processing to obtain the portrait similarity S i ; According to the i and the S i , generate the second noise data 3. The method according to claim 1, characterized in that In the Z i After the step of inputting the denoiser conditioned on the word unit embedding vector for noise estimation, the method further includes: Get the output feature D of the Nth layer network of the denoiser i , wherein N is an integer greater than 1; According to the ∈ i 、Z i and the reference face image to generate second noise data include: After inputting the reference face image into the denoiser for processing, obtaining an output feature D0 of the Nth layer network of the denoiser; Calculate the D i The feature similarity G between the D0 i ; According to the i and the G i , generate the second noise data 4. The method according to claim 1, wherein In the Z i After the step of inputting the denoiser conditioned on the word unit embedding vector for noise estimation, the method further includes: Get the output feature D of the Nth layer network of the denoiser i , wherein N is an integer greater than 1; According to the ∈ i 、Z i and the reference face image to generate second noise data include: According to the i and the Z i , generate the denoised image P i ; The P i The reference face image is input into the face recognition model for comparison processing to obtain the portrait similarity S i ; After the reference face image is input into the denoiser for processing, the output feature D0 of the Nth layer network of the denoiser is obtained; the D i The feature similarity G between the D0 i ; According to the i 、The S i and the G i , generate the second noise data 5. The method according to claim 4, characterized in that The output feature D of the Nth layer network of the denoiser is obtained i ,include: In the case of TM≥i≥1, the output feature D of the Nth layer network of the denoiser is obtained i , wherein M is an integer, 1<M<T.

6. The method according to any one of claims 3 to 5, characterized in that: The denoiser includes a K-layer network, wherein K / 2<N<K, and K is an integer greater than 1.

7. The method according to any one of claims 1 to 5, characterized in that The generating of a word element embedding vector based on the reference face image and the text prompt word includes: Inputting the reference face image into an image encoder for encoding processing to obtain a visual feature vector; Inputting the text prompt word into a text encoder for encoding to obtain a text feature vector; The visual feature vector and the text feature vector are spliced ​​in the channel dimension, and the vector splicing result is input into a feature alignment network based on a multi-layer perceptron for fusion processing to obtain a word unit embedding vector.

8. The method according to claim 7, characterized in that The training process of the image encoder includes the following steps: Acquire a data set, wherein the data set includes a plurality of sample face images; Constructing an initial backbone network corresponding to the image encoder; Connecting the denoiser to the initial backbone network to obtain a network to be trained; The network parameters of the denoiser in the network to be trained are fixed, and the network to be trained is trained using the data set until the model converges, thereby obtaining the image encoder.

9. An electronic device comprising: A memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the image generation method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the image generation method according to any one of claims 1 to 8 is implemented.

Images