A personalized character image generation method

Abstract

The present invention proposes a personalized character image generation method, which relates to the field of image processing technology. Given a single face image, a pseudo word "S*" is set, and the word embedding vector weight of the pseudo word is initialized as a word embedding vector "face". Then, according to the cross-attention map of the pseudo word "S*", a diffusion model denoising network is divided into multiple stages, and the word embedding vector of "S*" that changes with the time step is trained in the middle stage; dynamic constraints are performed through the changes of text conditions in a text encoder, and a collaborative balance loss function is calculated to achieve a balance of attention weights between the pseudo word "S*" and other text words.

Description

Technical Field

[0001] The invention proposes a method for generating a personalized character image, and relates to the technical field of image processing. Background Art

[0002] In recent years, AI-generated content has made breakthrough progress in many development directions, and image generation is undoubtedly the most eye-catching and innovative field. The diffusion model is a probability-based generative model. The advantage of the diffusion model lies in its training stability and the generation of high-quality results. Compared with the generative adversarial network, the diffusion model does not rely on adversarial training, so it is more stable during the training process and avoids the common mode collapse problem of the generative adversarial network. At the same time, compared with the variational autoencoder, the diffusion model can generate higher quality and more detailed images due to the gradual nature of the denoising process. With the development of generative models, the advantages of diffusion models in controllable generation are particularly significant. When performing text-to-image generation tasks, traditional diffusion models require the model to be able to generate high-quality images that match the text description. However, training in the pixel space of the image has high complexity and computational cost. Although the existing personalized generation methods have made considerable progress, they are still insufficient in some aspects, as follows:

[0003] First, when these methods achieve personalized generation, their generated results are sometimes still difficult to ensure that they are consistent with the user's expectations, especially in personalized face generation. The human face has complex features, which makes it more difficult to maintain the same consistency when generating personalized faces, and existing methods are difficult to handle.

[0004] Secondly, during the training process, these methods usually require users to collect multiple images of people in different scenes and postures, so as to prompt the word embedding vector or the parameters of the diffusion model to find the face area in the image, which makes them have a longer training time.

[0005] Furthermore, when only a single image is used for training, these methods will focus on the entire image in the cross-attention layer of the denoising network, and learn irrelevant content in the background area of ​​the image, causing the generated results to overfit during testing and making it difficult to maintain consistency with the given text description.

[0006] Finally, DB (DreamBooth), CD (Custom Diffusion) and Perfusion are personalized generation methods based on fine-tuning diffusion model parameters. Although they use category prior protection loss to protect the original semantic knowledge of the diffusion model as much as possible, the semantic drift caused by fine-tuning the model cannot be completely avoided, which means that the fine-tuned model is difficult to maintain accuracy and effectiveness in practical applications.

[0007] As an extension of the controllability research of diffusion models, personalized text-guided image generation has become a research direction worthy of attention. However, in terms of personalized generation of character images, most of the existing research techniques perform poorly. Summary of the invention

[0008] To this end, this technical solution proposes a personalized character image generation method. Given a single face image, a pseudo word "S*" is set, the word embedding vector of the pseudo word is initialized to the word "face", and the denoising network of the diffusion model is divided into multiple stages according to the cross-attention map of the pseudo word "S*", and the word embedding vector of the pseudo word "S*" that changes with the time step is trained in the middle stage;

[0009] By dynamically constraining the changes in text conditions in the text encoder of the diffusion model, the collaborative balance loss function is calculated to achieve a balance in the attention weights between the pseudo-word “S*” and other words in the text.

[0010] In a preferred embodiment, the denoising network of the diffusion model is divided into stages and the threshold is set. and Update the training interval, < ;

[0011] The exchange ratio will be lower than The interval is taken as the first static stage, in which no training is performed and the word embedding vector "face" is directly used as a conditional guide during testing;

[0012] When the intersection-and-combination ratio gradually increases and exceeds the threshold The interval after is the second static stage, which also does not participate in the training process. During testing, only a fixed word embedding vector of the pseudo word “S*” is used.

[0013] The remaining intermediate interval is used as the dynamic stage, using a lightweight fully connected network.

[0014] In a preferred embodiment, during the training process, the lightweight fully connected network inputs the time position code and the word embedding vector "face" into a multi-layer perceptron, and the multi-layer perceptron outputs different word embedding vectors according to the different time steps of the time position code;

[0015] The word embedding vector "face" is used as the feature representation of the pseudo word "S*", and is input into the text encoder of the diffusion model together with the word embedding vectors of other text words, and the embedding vector of the entire text sentence is output, which is then input into the cross-attention layer of the diffusion model as a text condition.

[0016] In a preferred embodiment, dynamic constraints through changes in text conditions in a text encoder include:

[0017] During the training process, text conditions with different semantic contents are randomly input into the text encoder, and the self-attention map of the last layer of the text encoder is extracted. The "start of text" and "end of text" are ignored, and the word with the largest attention weight other than the pseudo-word "S*" is found. The word is concatenated with the self-attention map of the pseudo-word "S*" and then split through the output of the softmax function to obtain and , and are the attention weights for the pseudo word “S*” and other texts respectively;

[0018] Calculating the synergy balance loss as follows:

[0019] ;

[0020] in, is the summation function.

[0021] In a preferred embodiment, a mask M of the face image is used to ensure that the denoising network focuses on the face area of ​​the image.

[0022] ;

[0023] in, is the denoising loss function in the diffusion model theory, is the noise added in the noise adding process, is the noise predicted by the denoising network, are the denoising network parameters, is the noise image at time step t, is the sentence embedding vector at time step t.

[0024] In a preferred embodiment, the noise image at time step t is Predict a noise-free image , using the existing face recognition network Arcface to extract The identity features and the training images The identity loss function is calculated based on the identity feature :

[0025] ;

[0026] Introducing hyperparameters To improve the effectiveness of identity loss:

[0027] ;

[0028] in, is the total number of denoising steps of the diffusion model, and the final loss function for:

[0029] ;

[0030] In a preferred embodiment, the image data given by the user is used to train a specific character identity, adjust the parameters of the generation model, and generate an image that meets the specific requirements.

[0031] Compared with the prior art, the present invention has the following beneficial technical effects:

[0032] 1. Stage-based training mechanism:

[0033] Based on the characteristic of the diffusion model that the controllability of text conditions varies according to different time steps, the present invention designs a stage division strategy and performs model training at specific stages, thereby improving the performance of personalized generation in terms of text similarity and identity consistency, while improving the training efficiency.

[0034] 2. Design and implementation of collaborative balance strategy:

[0035] The collaborative balance strategy uses the self-attention map of the text condition in the text encoder and designs a loss function to adjust the degree of attention of the denoising network to the facial content of the character, ensuring that the personalized results are consistent with the description of the text condition. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0037] Figure 1 is the cross attention map of face images at different time steps;

[0038] Figure 2 It is a schematic diagram of the overall framework of the present invention;

[0039] Figure 3 The present invention analyzes the problem of incomplete semantic expression;

[0040] Figure 4 It is a schematic diagram of the steps of the collaborative balance strategy of the present invention;

[0041] Figure 5The impact of the collaborative balancing strategy of the present invention on the cross-attention map; DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] The invention analyzes the influence of the change of time steps on the generation of personalized character images, analyzes the characteristics of the diffusion model denoising process, and sets a pseudo word "S*" to represent the concept of the face in the image given a single face image. The corresponding word embedding vector in the text encoder is initialized to the word embedding vector of the word "face". A fully connected network and a text condition "A photo of S*" are used to train a group of word embedding vectors that vary according to the time steps according to the training method of the personalized character image generation method based on optimization, and then the cross-attention graphs of the pseudo word "S*" at different time steps are visualized.

[0044] When the number of time steps is large, the cross-attention map corresponding to the pseudo-word "S*" is not accurate, and its attention is on the entire image, which means that the word embedding vector may be trained to the content of the background area of ​​the image, which is harmful to the generation of personalized character images. As the number of time steps decreases, the cross-attention of the pseudo-word "S*" gradually finds more accurate face areas, which means that the pseudo-word "S*" can more efficiently obtain accurate character identity information at these time steps.

[0045] The present invention proposes a training network framework from the perspective of the noise level of the noise map.

[0046] In the early stage of the denoising network of the diffusion model, the noise map contains too much noise, which causes the word embedding vector of the pseudo word "S*" to be unable to accurately focus on the face area at this stage, thereby generating a cross-attention map that does not meet expectations. Therefore, it is difficult for the word embedding vector to learn useful conceptual information during training at this stage; as the number of time steps decreases, the noise in the noise map decreases, and the interference of redundant information brought by the noise on the training process is also reduced. However, in the late stage of the denoising network of the diffusion model of the prior art, it is difficult for the word embedding vector to change the semantic content of the image generation. Therefore, the present invention proposes to train the word embedding vector in the middle part of the denoising network process of the diffusion model.

[0047] The present invention proposes to divide the denoising network process of the diffusion model into three stages according to the attention map of the pseudo word "S*", and train the word embedding vector that changes with the time step in the middle stage.

[0048] In different images, the area occupied by the face region is different, so the word embedding vector of the pseudo word "S*" can extract different person identity information from it, which leads to a certain degree of difference in the interval selected for training between training images. In order to reasonably divide the stages, the present invention adopts a stage division strategy:

[0049] Training is performed on the entire time step. After a certain number of training iterations, although the word embedding vector of the pseudo word "S*" has not accurately trained to learn the identity information of the person, its cross-attention map has shown obvious differences at different time steps, and no overfitting has occurred. For example, Figure 1 As shown, the present invention uses an image segmentation model to predict the mask of the face area, and calculates the intersection-over-union ratio with the cross-attention graph at different time steps. The intersection-over-union ratio will gradually increase with the denoising process.

[0050] Specifically, two thresholds are set and ( < ) to update the training interval.

[0051] The intersection-combination ratio is lower than the threshold As the first static stage, we choose not to train in this static stage, but directly use the word embedding vector of "face" as a conditional guide during testing.

[0052] When the intersection-and-combination ratio gradually increases and exceeds the threshold Afterwards, although the word embedding vector of the pseudo word "S*" can accurately find the face area, it also shows that there is a small amount of noise in the noise map. The controllability of the text condition on the diffusion model is reduced. This stage is regarded as the second static stage, which also does not participate in the training process. Instead, only a fixed word embedding vector of the pseudo word "S*" is used during testing.

[0053] For the remaining intermediate interval, it is set as a dynamic stage, in which a lightweight fully connected network is used, such as Figure 2 The input includes a temporal position code, a word embedding vector for "face", and an image embedding vector obtained after the face image passes through the image encoder.

[0054] During the training process, the temporal position encoding and the word embedding vector of "face" are input into a multi-layer perceptron together. According to the difference of the temporal position encoding at different time steps, the multi-layer perceptron will output different word embedding vectors.

[0055] These dynamically changing word embedding vectors are used as feature representations of the pseudo word “S*”, and then input into the text encoder of the diffusion model together with the word embedding vectors of other text words to output the sentence embedding vector of the entire text, which is then input into the cross-attention layer of the diffusion model as a text condition.

[0056] The image embedding vector corresponding to the face image is introduced, concatenated with the word embedding vector of the pseudo word "S*" learned by the network, and then input into another multi-layer perceptron. The output of the multi-layer perceptron is called the residual embedding vector, which is then added to the sentence embedding vector output by the text encoder. The purpose is to capture the image detail information that is difficult for the text encoder to provide through the residual embedding vector.

[0057] Based on the analysis of the characteristics of the diffusion model at different time steps, the present invention learns dynamically changing word embedding vectors in a dynamic interval, which can enable the cross-attention map of the introduced concept to focus more accurately on the face area. This not only helps the present invention avoid overfitting when training on a single image, but also improves its training efficiency because it is trained in a smaller time interval.

[0058] On this basis, the present invention proposes a collaborative balance strategy to solve the problem that the results of the existing methods for generating personalized character images are inconsistent with the text descriptions. The specific contents are as follows:

[0059] like Figure 3-Figure 5 As shown, the present invention first trains a set of word embedding vectors of pseudo words "S*" through given real images and performs tests. Although the results well maintain the identity information of the characters, it is obviously observed that the generated results deviate from the semantics of "wizard".

[0060] The cross-attention map in the denoising network of the diffusion model is visualized, and the L2 norm is used as an indicator to measure the weight of each word in the text on the cross-attention map. It can be found from the figure that the L2 norm of the cross-attention map of the pseudo-word "S*" (0.9596) is much higher than the value of the word "garland" (0.5407), which is the key reason why personalized generation cannot fully express the semantics of the text.

[0061] A threshold is set to limit the attention weight of the pseudo-word "S*" to a specific range. However, the attention weight of the pseudo-word "S*" varies in different text conditions. Therefore, it is not reasonable to use a fixed threshold. Therefore, this embodiment proposes a collaborative balance strategy.

[0062] Dynamic constraints are implemented based on changes in the text condition "A photo of S*":

[0063] During the training process, we first randomly input text conditions with different semantic contents into the text encoder, and extract the self-attention map of the last layer of the text encoder, ignoring the “start of text” and “end of text”, and find the word with the largest attention weight other than the pseudo-word “S*”, for example Figure 4 Then concatenate the self-attention maps of the pseudo word "S*" and the pseudo word "garland" and pass them through the softmax function to get and ; and The attention weights for the pseudo word "S*" and other text words are then used to calculate the collaborative balance loss as follows:

[0064] ;

[0065] in, is the summation function, since and The sum of is fixed, minimizing The two need to be kept as close as possible, and the loss function can ensure that the diffusion model effectively balances the attention weights between "S*" and other texts.

[0066] For the loss function, the mask M of the face image is used to ensure that the denoising network focuses on the face area of ​​the image, so as to better learn the identity information of the face and avoid the interference of the background content in the image:

[0067] ;

[0068] in, for The noise image at time instant, for The sentence embedding vector at the moment.

[0069] In order to ensure that accurate person identity information is learned from the image, the time step is Noise map Predict a noise-free image , using the existing face recognition network Arcface to extract its identity features and compare it with the training image The identity loss function is calculated based on the identity feature:

[0070] ;

[0071] Since the diffusion model has difficulty predicting a clear image when the number of time steps is large, , introduce hyperparameters To improve the effectiveness of identity loss:

[0072] ;

[0073] Final loss function for:

[0074] ;

[0075] In order to further explore the impact of stage division, in a preferred embodiment, a quantitative analysis on interval division is performed, as shown in Table 1.

[0076]

[0077] Initially, this solution sets the threshold =0.5 and =1.0, which means that the denoising process is not divided into intervals and uses dynamically changing word embedding vectors at all time steps. It can be found that the performance of the indicators is unsatisfactory. =0.6 or =0.7, the denoising process is divided into two stages: less than The interval is the static stage, in which the present invention uses the word embedding vector of "face"; greater than The interval is the dynamic stage, in which the present invention adopts dynamically changing word embedding vectors. As the number of time steps increases, the word embedding vector of “face” is used in more time steps, which improves the text similarity and also helps the dynamic stage learn more accurate identity information and improves identity preservation. When it is greater than 0.8, the identity consistency of the personalized generated images begins to decrease. The reason is that in the dynamic stage, the smaller time step limits the diffusion model's ability to control the semantics of the text at the end of denoising, resulting in a decrease in the identity information provided by the word embedding vector of "S*". , then gradually reduce , so that the denoising process is further divided into three stages, the third stage is the static stage, as As is gradually reduced, the identity consistency of the image generated by the present invention is gradually improved. When it is less than 0.8, the range of the dynamic stage of the present invention is too small, resulting in an insufficient number of word embedding vectors generated by the network framework, making it difficult to capture identity details at different time steps. Set to 0.7, Set to 0.8. Since this embodiment divides the intervals based on the intersection-over-intersection ratio of the cross-attention map and the true mask, this method has high flexibility. and The selection of shows strong generalization ability on different data sets.

[0078] This paper further analyzes the collaborative balance strategy by visualizing the cross-attention graph in the denoising network, such as Figure 5 As shown. When the collaborative balance strategy is not adopted, the weight of "S*" in the cross attention layer is 1.2607, which is much higher than the weight of "wizard" 0.4319. This causes the semantics of "wizard" to be ignored, which is reflected in the fact that the attention area on its corresponding cross attention map does not match the text semantics. After the collaborative balance strategy is introduced, the attention weight of "S*" decreases by 0.5221, and the weight of "wizard" increases by 0.1044, which makes the area corresponding to "wizard" on the cross attention map more consistent with its text semantics, and the personalized results are more in line with the text description.

[0079] In order to further verify the effectiveness of each component in the present invention, the present invention conducts a qualitative ablation study to analyze the role of each design, such as Figure 5 As shown, if the present invention does not adopt the stage division strategy ( =0.5 and =1.0), the background content of the generated image will be poor. The reason for this difference is that the word embedding vector of "S*" may focus on the non-face area in the image, thereby interfering with the semantic guidance of other words in the text conditions provided by the user. A similar situation also occurs in the present invention setting =0.5 and =0.8. This highlights the necessity of the first static stage. The present invention generates a spatial layout that conforms to the text description during personalized generation and guides the diffusion model to produce the desired cross-attention map. Increasing from 0.5 to 0.7, the personalized generated results better match the description of the text, although the identity of the character is still slightly insufficient. When it continues to increase to 0.8, the identity of the person in the image becomes inaccurate. = 0.7, the present invention adjusts The value of can be found in =0.8, the results are consistent with the text description and the identities of the characters are more accurate.

[0080] In a preferred embodiment, in order to intuitively reflect the result advantage of this method compared with other methods, the present invention further constructs a qualitative evaluation. For each individual character, this embodiment collects 3 to 5 images of it and randomly selects one of them as the input of this method. In order to ensure fairness, this embodiment also shows the results of the comparison method under the default multiple image training setting. It can be clearly seen that DB (Dreambooth) performs relatively well in identity consistency, but most of the personalized images do not conform to the text description. This is because the training is overfitted during the entire denoising stage, and its cross-attention map focuses on the entire input image. The background content of the training image prevents the model from introducing new semantics in the test. In contrast, this method is only trained in the dynamic stage, and the guided word embedding vector only focuses on the facial area of ​​the human face in the image. The results generated by TI (Textual Inversion), CD (Custom Diffusion) and Perfusion are basically consistent with the text description, but are inaccurate in identity retention. The image embedding vector is introduced in this method to provide assistance, so that the sentence embedding vector learns accurate identity information.

[0081] The above scheme is implemented based on the existing, pre-trained text-generated image diffusion model. With the continuous development of the generative model, a controllable generative model with higher performance is proposed in the preferred embodiment, which directly generates images based on the face image as a condition.

[0082] Domain adaptation technology can learn personalized features from the generative model and adapt the model to different scenarios of user needs. In a preferred embodiment, the specific person identity is learned through the image data given by the user, and the parameters of the generative model are adjusted to generate images that meet specific requirements.

[0083] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0084] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for generating a personalized character image, characterized in that: Given a single face image, set a pseudo word "S*", initialize the word embedding vector of the pseudo word to the word "face", divide the denoising network of the diffusion model into multiple stages according to the cross-attention map of the pseudo word "S*", and train the word embedding vector of the pseudo word "S*" that changes with the time step in the middle stage; Divide the denoising network of the diffusion model into stages and set the threshold and Update the training interval, < ; The exchange ratio will be lower than The interval is taken as the first static stage, in which no training is performed and the word embedding vector "face" is directly used as a conditional guide during testing; When the intersection-and-combination ratio gradually increases and exceeds the threshold The interval after is the second static stage, which also does not participate in the training process. During testing, only a fixed word embedding vector of the pseudo word "S*" is used. The remaining intermediate interval is used as the dynamic stage, using a lightweight fully connected network; During the training process, the lightweight fully connected network inputs the time position code and the word embedding vector "face" into a multi-layer perceptron. The multi-layer perceptron outputs different word embedding vectors according to the different time steps of the time position code. The word embedding vector "face" is used as the feature representation of the pseudo word "S*", and is input into the text encoder of the diffusion model together with the word embedding vectors of other text words, and the embedding vector of the entire text sentence is output, and then it is input into the cross attention layer of the diffusion model as a text condition, and the image embedding vector corresponding to the face image is introduced. After being spliced ​​with the word embedding vector of the pseudo word "S*" learned by the network, it is input into another multi-layer perceptron. The content obtained by the output of the multi-layer perceptron is called the residual embedding vector, which is then added to the sentence embedding vector output by the text encoder; Dynamically constrain the changes in text conditions in the text encoder of the diffusion model, Dynamic constraints that are implemented by changing text conditions in the text encoder include: During the training process, we randomly input text conditions with different semantic contents into the text encoder, extract the self-attention map of the last layer of the text encoder, ignore "start of text" and "end of text", find the word with the largest attention weight other than the pseudo-word "S*", concatenate it with the self-attention map of the pseudo-word "S*", and split it through the output of the softmax function to obtain and , and are the attention weights for the pseudo word "S*" and other texts respectively; Calculate the collaborative balance loss function to achieve a balance between the attention weights of the pseudo-word "S*" and other words in the text; Calculating the synergy balance loss as follows: ； in, is the sum function; The mask M of the face image is used to ensure that the denoising network focuses on the face area of ​​the image. ； in, is the denoising loss function in the diffusion model theory, is the noise added in the noise adding process, is the noise predicted by the denoising network, are the denoising network parameters, is the noise image at time step t, is the sentence embedding vector at time step t; From the noise image at time step t Predict a noise-free image , using the existing face recognition network Arcface to extract The identity features and the training images The identity loss function is calculated based on the identity feature : ； Introducing hyperparameters To improve the effectiveness of identity loss: ； in, is the total number of denoising steps of the diffusion model, and the final loss function for: .

2. The method for generating a personalized character image according to claim 1, characterized in that: Use the image data provided by the user to train a specific character identity, adjust the parameters of the generation model, and generate images that meet specific requirements.

Images