Multi-role customization generation method without fine tuning
Through the multi-role custom generation method without fine-tuning, the multi-role adapter and role attention mechanism are used to solve the problems of high fine-tuning cost and insufficient multi-role generation ability in the existing technology, and achieve a fast, cheap and diverse multi-role role Customize the generation effect.
Patent Information
- Application Number
- CN202411671739.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-21
- Publication Date
- 2025-02-11
AI Technical Summary
The existing multi-character custom generation methods require a lot of time and money to fine-tune the model, and the training-free method has challenges in the ability to generate multiple characters.
Using a multi-role customization generation method without fine-tuning, including image codec, text encoder, noise reduction model, character encoder and multi-role adapter, multi-role customization generation is achieved through multi-role adapter and character attention mechanism. And enrich the sample diversity through clustering algorithm.
It enables fast and inexpensive custom generation for multiple roles, alleviating the problem of character confusion and improving the diversity of generated results.
Smart Images
Figure CN119417949A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer science and technology, and in particular to a method for customizing and generating multiple roles without fine-tuning. Background Art
[0002] Computer science is a science that studies computers and various phenomena and laws surrounding them. It systematically studies the theoretical basis of information and computing and how they are implemented and applied in computer systems; Theoretical computer science: This is the foundation of computer science. It mainly studies the theoretical basis of computing, including computability and complexity theory. It explores the limits and nature of computing and provides theoretical support for other fields of computer science; Computer system structure: It mainly studies the organization and design of computer hardware and software, focusing on how to improve the performance and efficiency of computers, while also paying attention to the sustainability and energy efficiency of computers; Operating system: It is responsible for managing and coordinating the various parts of a computer system, and studies the design, management, and optimization of computer operating systems.
[0003] With the improvement of diffusion model-based image generation technology, the quality of image generation has been significantly improved, and the controllability of generated content has become better and better. Methods for customized character generation have begun to emerge. Common customized generation methods can be roughly divided into two categories: Fine-tuning based methods: such as Lora, Dreambooth, Textual Inversion, Custom Diffusion, etc. The results generated by these methods are outstanding in terms of the fidelity of customized characters, and can achieve good results regardless of customizing a single character or multiple characters. However, these methods have a significant drawback: they need to use multiple samples of the customized characters to fine-tune the model, which not only means that a lot of time and effort is needed to collect samples, but also the process of model fine-tuning itself requires a high monetary cost. Therefore, although these methods perform well in terms of results, their high cost limits their wider application.
[0004] Training-free methods: such as IP-Adapter and InstantID. They inject the information of the reference image directly into the image generation process by training additional cross-attention modules. In this way, only one sample of the customized character is needed to complete the customized generation of the character, and no fine-tuning is required during the generation stage. This method is not only fast in generation speed but also relatively low in cost. However, its generation effect is often not as good as that based on fine-tuning. More importantly, this type of method has not yet formed a relatively complete solution for customized generation of multiple characters, which means that when multiple characters need to be generated at the same time, this type of method may encounter greater challenges. Summary of the invention
[0005] The purpose of the present invention is to provide a multi-role customized generation method that does not require fine-tuning, has the advantages of multi-role customized generation, prevents role confusion and improves generation diversity, and solves the problems raised by the above-mentioned background technology.
[0006] To achieve the above objectives, the present invention provides the following technical solutions: a multi-role customized generation method without fine-tuning, including five modules: image codec, text encoder, denoising model, role encoder, and multi-role adapter. The open source Stable Diffusion is selected as the basic model. The image codec, text encoder, and denoising model directly use the structure and weights of the original model and keep them frozen during the training process.
[0007] Image codec,The image codec is a variational autoencoder (VAE). The encoding part is responsible for encoding the original image into a multi-dimensional tensor in the latent space to reduce the complexity of the generation algorithm. The decoding part is responsible for decoding the multi-dimensional tensor in the latent space into the original image.
[0008] Text Encoder,The text encoder uses the text encoding part of CLIP, which is responsible for encoding the original text into high-dimensional features,,which are used to represent the high-level semantics of the text.
[0009] Denoising model,The denoising model uses U-Net, which gradually denoises the Gaussian noise in the latent space, and obtains the latent space tensor of the image.
[0010] Character Encoder,The character encoder is responsible for encoding the character image into multiple,high-dimensional features.
[0011] Multi-role adapter, the multi-role adapter is an independent cross-attention module for customized roles, responsible for injecting role information into the latent space tensor during the denoising process. It is a learnable module. The multi-role adapter extends the cross-attention module for customizing a single role adopted by IP-Adapter to customize multiple roles, and designs a role attention mechanism to address the problem of role feature confusion that is prone to occur in multi-role generation. The output of the cross-attention module of the multi-role adapter can be defined by the following formula:
[0012] Where Q = ZW q , K = ZW k , V = ZW v is the query, key, value matrix of image features, K m =ZW km is the character attention key matrix, M is the attention score of the image feature to the character feature, Spansum is the operation of summing the attention scores according to the characters they belong to, and Onehot is the operation of setting the maximum number to 1 and the rest to 0.
[0013] Furthermore, as a preferred embodiment of the present invention, a latent space diffusion model is used as a basic framework, and multi-role information is injected into the generation process through an independent self-attention model.
[0014] Furthermore, as a preferred embodiment of the present invention, the encoder used is a fully connected layer added to the end of the CLIP image encoder to change the dimension of the CLIP feature, which is a learnable module.
[0015] Furthermore, as a preferred embodiment of the present invention, the main features of the cross attention module are: 1. It can accept multi-role input; 2. It introduces K m , which is responsible for learning the attention score of image features on the role, and then obtaining the 1-hot role Mask. The role Mask will limit the image latent space features at each position to be affected by only a single role, thereby alleviating the problem of role confusion.
[0016] Further, as a preferred embodiment of the present invention, the process of keeping frozen during training is: S1: Data preparation.
[0017] S1.1: Get the original data by extracting animation frames and crawling the web.
[0018] S1.2: Use pedestrian detector and cartoon character detector to get the accurate position of real person / animated characters.
[0019] S1.3: Extract close-up images of each character and extract features using the CLIP image encoder.
[0020] S1.4: Clustering of CLIP features of all characters.
[0021] S1.5: Use the clustering results to establish (role image, original image) sample pairs to avoid the role image in the sample pair coming from the original image.
[0022] S.1.6: Use a multimodal large model to annotate the original image.
[0023] S2: Model training.
[0024] S2.1: Sample a small batch of training data, the data structure is (character image, original image, original image text annotation).
[0025] S2.2: The character map is encoded into character features by the character encoder, the original image is encoded into a latent space tensor by the image encoder, and the text annotation is encoded into text features by the text encoder.
[0026] S2.3: Sample a diffusion model time step for each data, and superimpose noise on the latent space tensor according to the time step.
[0027] S2.4: The latent space tensor after superimposing noise is input into the denoising model, and the denoising model predicts the superimposed noise in S2.3. The character features and text features are connected to the denoising model by the multi-role adapter and cross-attention respectively. The loss function is the least squares error of the superimposed noise and the predicted noise.
[0028] S2.5: Calculate the gradients of the role encoder and the multi-role adapter, and adjust the weights of the role encoder and the multi-role adapter.
[0029] S2.6: Repeat the above steps until the model converges.
[0030] In the present invention, a method for customizing and generating multiple roles without fine-tuning is provided, and the method comprises the following steps: Step 1: Use text to describe the content of the image to be generated, such as "swimming in a pool".
[0031] Step 2: Select a character. Each character only requires one picture. The picture must contain the complete character image and as little background content as possible.
[0032] Step 3: The text encoder encodes the text description into text features, and the role encoder encodes the role image into role features.
[0033] Step 4: A latent space tensor is randomly generated, and then the denoising model predicts the current step noise and removes it. During the denoising process, the cross-attention module and multi-role adapter inject text and role features into the denoising model.
[0034] Step 5: Repeat step 4 multiple times until the set number of steps is reached, and then decode the denoised latent space tensor into an image.
[0035] Beneficial effects, the technical solution of the present application has the following technical effects: the present invention has the advantages of multi-role customized generation, anti-role confusion and improved generation diversity. By introducing a multi-role adapter, an innovative method that can quickly and cheaply customize and generate multiple roles is realized. As the core innovation of the entire framework, this adapter not only expands the application scope of the traditional cross-attention module, enabling it to support the customized generation of multiple roles, but also introduces a role self-attention mechanism to address the problem that role features are easily confused with each other during the generation of multiple roles. This mechanism calculates the attention score of image features to role features, and then generates a 1-hot role Mask to ensure that the image latent space features at each position are only affected by a single role, thereby effectively alleviating the problem of role confusion; in addition, in order to further improve the diversity of the generated results, a clustering algorithm is used to annotate the sample pairs. Specifically, the traditional method of directly cutting out the original image to construct the sample pair is avoided, but the features of all roles are clustered, and the clustering results are used to establish the (role map, original image) sample pair. This method not only enriches the diversity of the samples, but also helps the model to better learn the feature differences between different roles, thereby further improving the generation effect. The problems raised by the above background technology are solved. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 This is a cross-attention module diagram of the multi-role adapter of the present invention; Figure 2 It is a system framework diagram of the present invention. DETAILED DESCRIPTION
[0037] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. In order to better understand the technical content of the present invention, specific embodiments are cited and explained in conjunction with the drawings as follows. Various aspects of the present invention are described in this disclosure with reference to the drawings, in which many illustrative embodiments are shown. It should be understood that the various concepts and embodiments introduced above, as well as those described in more detail below, can be implemented in any of many ways. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0038] As attached Figure 1 To Attachment Figure 2As shown: This embodiment provides a multi-role customized generation method that does not require fine-tuning, including five modules: image codec, text encoder, denoising model, role encoder, and multi-role adapter. The open source Stable Diffusion is selected as the basic model. The image codec, text encoder, and denoising model directly use the structure and weights of the original model and keep them frozen during the training process.
[0039] Image codec,The image codec is a variational autoencoder (VAE). The encoding part is responsible for encoding the original image into a multi-dimensional tensor in the latent space to reduce the complexity of the generation algorithm. The decoding part is responsible for decoding the multi-dimensional tensor in the latent space into the original image.
[0040] Text Encoder,The text encoder uses the text encoding part of CLIP, which is responsible for encoding the original text into high-dimensional features,,which are used to represent the high-level semantics of the text.
[0041] Denoising model,The denoising model uses U-Net, which gradually denoises the Gaussian noise in the latent space, and obtains the latent space tensor of the image.
[0042] Character Encoder,The character encoder is responsible for encoding the character image into multiple,high-dimensional features.
[0043] Multi-role adapter, the multi-role adapter is an independent cross-attention module for customized roles, responsible for injecting role information into the latent space tensor during the denoising process. It is a learnable module. The multi-role adapter extends the cross-attention module for customizing a single role adopted by IP-Adapter to customize multiple roles, and designs a role attention mechanism to address the problem of role feature confusion that is prone to occur in multi-role generation. The output of the cross-attention module of the multi-role adapter can be defined by the following formula:
[0044] Where Q = ZW q , K = ZW k , V = ZW v is the query, key, value matrix of image features, K m =ZW km is the character attention key matrix, M is the attention score of the image feature to the character feature, Spansum is the operation of summing the attention scores according to the characters they belong to, and Onehot is the operation of setting the maximum number to 1 and the rest to 0.
[0045] Specifically, the latent space diffusion model is used as the basic framework, and multi-role information is injected into the generation process through an independent self-attention model.
[0046] Specifically, the encoder used adds a fully connected layer at the end of the CLIP image encoder to change the dimension of the CLIP feature, which is a learnable module.
[0047] Specifically, the main features of the cross-attention module are: 1. It can accept multi-role input; 2. It introduces K m , which is responsible for learning the attention score of image features on the role, and then obtaining the 1-hot role Mask. The role Mask will limit the image latent space features at each position to be affected by only a single role, thereby alleviating the problem of role confusion.
[0048] Specifically, the process of keeping frozen during training is; S1: Data preparation.
[0049] S1.1: Get the original data by extracting animation frames and crawling the web.
[0050] S1.2: Use pedestrian detector and cartoon character detector to get the accurate position of real person / animated characters.
[0051] S1.3: Extract close-up images of each character and extract features using the CLIP image encoder.
[0052] S1.4: Clustering of CLIP features of all characters.
[0053] S1.5: Use the clustering results to establish (role image, original image) sample pairs to avoid the role image in the sample pair coming from the original image.
[0054] S.1.6: Use a multimodal large model to annotate the original image.
[0055] S2: Model training.
[0056] S2.1: Sample a small batch of training data, the data structure is (character image, original image, original image text annotation).
[0057] S2.2: The character map is encoded into character features by the character encoder, the original image is encoded into a latent space tensor by the image encoder, and the text annotation is encoded into text features by the text encoder.
[0058] S2.3: Sample a diffusion model time step for each data, and superimpose noise on the latent space tensor according to the time step.
[0059] S2.4: The latent space tensor after superimposing noise is input into the denoising model, and the denoising model predicts the superimposed noise in S2.3. The character features and text features are connected to the denoising model by the multi-role adapter and cross-attention respectively. The loss function is the least squares error of the superimposed noise and the predicted noise.
[0060] S2.5: Calculate the gradients of the role encoder and the multi-role adapter, and adjust the weights of the role encoder and the multi-role adapter.
[0061] S2.6: Repeat the above steps until the model converges.
[0062] In the present invention, a method for customizing and generating multiple roles without fine-tuning is provided, and the method comprises the following steps: Step 1: Use text to describe the content of the image to be generated, such as "swimming in a pool".
[0063] Step 2: Select a character. Each character only requires one picture. The picture must contain the complete character image and as little background content as possible.
[0064] Step 3: The text encoder encodes the text description into text features, and the role encoder encodes the role image into role features.
[0065] Step 4: A latent space tensor is randomly generated, and then the denoising model predicts the current step noise and removes it. During the denoising process, the cross-attention module and multi-role adapter inject text and role features into the denoising model.
[0066] Step 5: Repeat step 4 multiple times until the set number of steps is reached, and then decode the denoised latent space tensor into an image.
[0067] It should be noted that, in this document, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.
[0068] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. A person with ordinary knowledge in the technical field to which the present invention belongs may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the definition of the claims.
Claims
1. A method for customizing and generating multiple roles without fine-tuning, characterized in that: It includes five modules: image codec, text encoder, denoising model, role encoder, and multi-role adapter. The open source Stable Diffusion is used as the basic model. The image codec, text encoder, and denoising model directly use the structure and weights of the original model and keep them frozen during the training process. Image codec, the image codec is a variational autoencoder (VAE), the encoding part is responsible for encoding the original image into a multi-dimensional tensor in the latent space to reduce the complexity of the generation algorithm, and the decoding part is responsible for decoding the multi-dimensional tensor in the latent space into the original image; Text encoder: The text encoder uses the text encoding part of CLIP, which is responsible for encoding the original text into high-dimensional features to represent the high-level semantics of the text; Denoising model,The denoising model uses U-Net, which gradually denoises the Gaussian noise in the latent space to obtain the latent space tensor of the image; Role encoder, the role encoder is responsible for encoding the role image into multiple high-dimensional features; Multi-role adapter, the multi-role adapter is an independent cross-attention module for customized roles, responsible for injecting role information into the latent space tensor during the denoising process. It is a learnable module. The multi-role adapter extends the cross-attention module for customizing a single role adopted by IP-Adapter to customize multiple roles, and designs a role attention mechanism to address the problem of role feature confusion that is prone to occur in multi-role generation. The output of the cross-attention module of the multi-role adapter can be defined by the following formula: Where Q = ZW q , K = ZW k , V = ZW v is the query, key, value matrix of image features, K m =ZW km is the character attention key matrix, M is the attention score of the image feature to the character feature, Spansum is the operation of summing the attention scores according to the characters they belong to, and Onehot is the operation of setting the maximum number to 1 and the rest to 0.
2. The method for generating customized multiple roles without fine-tuning according to claim 1, characterized in that: Using the latent space diffusion model as the basic framework, multi-role information is injected into the generation process through an independent self-attention model.
3. The method for generating customized multiple roles without fine-tuning according to claim 1, characterized in that: The encoder used adds a fully connected layer at the end of the CLIP image encoder to change the dimension of the CLIP feature. It is a learnable module.
4. The method for generating customized multiple roles without fine-tuning according to claim 1, characterized in that: The main features of the cross-attention module are:
1. It can accept multi-role input; 2. It introduces K m , which is responsible for learning the attention score of image features on the role, and then obtaining the 1-hot role Mask. The role Mask will limit the image latent space features at each position to be affected by only a single role, thereby alleviating the problem of role confusion.
5. The method for generating customized multiple roles without fine-tuning according to claim 1, characterized in that: The process of keeping frozen during training is; S1: Data preparation; S1.1: Get the original data by extracting animation frames and crawling the web; S1.2: Use pedestrian detector and cartoon character detector to get the accurate position of real person / animation character; S1.3: Extract close-up images of each character and extract features using the CLIP image encoder; S1.4: Clustering of CLIP features of all characters; S1.5: Use the clustering results to establish (role image, original image) sample pairs to avoid the role image in the sample pair coming from the original image; S.1.6: Use a large multimodal model to annotate the original image; S2: Model training; S2.1: Sample a small batch of training data, the data structure is (character image, original image, original image text annotation); S2.2: The character map is encoded into character features by the character encoder, the original image is encoded into a latent space tensor by the image encoder, and the text annotation is encoded into text features by the text encoder; S2.3: Sample a diffusion model time step for each data, and superimpose noise on the latent space tensor according to the time step; S2.4: The latent space tensor after superimposing noise is input into the denoising model, and the denoising model predicts the noise superimposed in S2.
3. The character features and text features are connected to the denoising model by the multi-character adapter and cross attention respectively. The loss function is the least square error of the superimposed noise and the predicted noise. S2.5: Calculate the gradients of the role encoder and the multi-role adapter, and adjust the weights of the role encoder and the multi-role adapter; S2.6: Repeat the above steps until the model converges.
6. A method for customizing and generating multiple roles without fine-tuning, characterized in that: The method comprises the following steps: Step 1: Use words to describe the content of the image to be generated, such as "swimming in a pool"; Step 2: Select a character. Each character only needs one picture. The picture must contain the complete character image and as little background content as possible. Step 3: The text encoder encodes the text description into text features, and the role encoder encodes the role image into role features; Step 4: A latent space tensor is randomly generated, and then the denoising model predicts the current step noise and removes it. During the denoising process, the cross attention module and multi-role adapter inject text and role features into the denoising model; Step 5: Repeat step 4 multiple times until the set number of steps is reached, and then decode the denoised latent space tensor into an image.