Personalized portrait generation and style conversion method and device and medium

Abstract

The invention relates to a personalized portrait generation and style conversion method and device and a medium, and the method comprises the steps: inputting at least one style image into a style low-rank adaptation model, and carrying out the partitioning low-rank adaptation training of the style low-rank adaptation model; linear superposition is carried out on the weight of the trained style low-rank adaptation model and the weight of the corresponding diffusion model U-net for style control in the image generation process; inputting at least one reference portrait into Insight Face, extracting a portrait attitude graph and portrait embedding, and generating attitude control information through a portrait control network; and performing diffusion model U-net image generation according to attitude control information generated by the portrait control network, output after the portrait is embedded into the input cross attention module, text embedding generated by text control statement processing and weight after linear superposition. Compared with the prior art, stable and efficient personalized and stylized picture generation can be realized.

Description

Technical Field

[0001] The present invention relates to the field of controllable generation of image generation models, and in particular to a personalized portrait generation and style conversion method, device and medium. Background Art

[0002] Personalized portrait generation technology is widely used in social media, emoji generation, and short film production to provide higher quality images in multiple styles. This technology not only needs to achieve different image styles, but also accurately generate specified character features. Users can modify postures and expressions, adjust facial features, and achieve natural changes. Expression adjustment, style transfer, age change, and gender conversion can make photos more interesting or meet specific needs. The key challenge is how to achieve reasonable style transfer while retaining personalized facial features under the condition of a single input image.

[0003] When editing and modifying certain specific facial attributes, other facial features are often uncontrollably affected, and unreasonable images may be generated due to changes in the identity information of the original face. Each person's facial features are highly personalized, and these personalized details need to be retained during the editing process to avoid generating common faces or generating distortions. In addition, style changes involve coordinated changes in multiple parts of the image, and unnatural deformations are prone to occur during the editing process, making it difficult to maintain facial coherence. Existing technologies are insufficient in balancing the identity of the person and the editing effect.

[0004] Prior art 1: DreamBooth-LoRA is an improved method that combines the DreamBooth method with the low-rank adaptation (LoRA) technology, aiming to further improve the efficiency and operability of customized image generation. Low-rank adaptation was first used on large language models by "LoRA: Low-Rank Adaptation of Large Language Models". This method is a lightweight fine-tuning technique that can reduce the demand for computing resources by reducing the complexity of model update parameters while maintaining good performance. Combining low-rank adaptation with DreamBooth can significantly reduce training costs and improve customization flexibility and efficiency. Traditional fine-tuning methods usually require updating the parameters of the entire model, while low-rank adaptation introduces low-rank matrices to limit the amount of updated parameters, making model fine-tuning more efficient and computationally cheaper.

[0005] Prior art 2: ControlNet is a control mechanism for image generation and editing, which aims to improve the control ability when generating images based on diffusion models. ControlNet allows users to precisely control the content and structure of generated images by specifying additional conditions or constraints. It combines external conditions with the diffusion model, allowing users to achieve more precise control when generating images, including control over the posture, structure, semantic content, etc. of the image. This allows the generated image to not only rely on text descriptions, but also constrain the generation process based on external guidance information (such as image contours, postures, depth maps, etc.).

[0006] However, in the image generation task of specifying a person and a specified style, it is often necessary to use multiple models together, but the joint use of multiple models has the following problems:

[0007] (1) Models share feature space. When multiple similar models are applied to the same base model, they will occupy the model’s feature space together and may affect each other. Each model is fine-tuned in a specific field (people, objects, style, etc.). When used together, it may be difficult for the model to distinguish these features. For example, if two models are trained on the features of “cat” and “dog” respectively, they cannot be clearly distinguished when the models are used to generate images at the same time, resulting in the generated “animal” with mixed cat and dog features.

[0008] (2) There is an information coverage problem. When using multiple fine-tuned models of the same type, each model will fine-tune and update part of the weights of the pre-trained model. The later trained model may cover some features of the previous model, resulting in the loss of information learned by the model. For example, if a model for generating a specific character is used together with a model for generating a similar style, the superposition effect of the weight values ​​of the two is difficult to predict. It may cause the details of the character to become blurred or the style to be lost.

[0009] (3) Instability of generated results. Since each model has different fine-tuning objectives, unstable effects may occur when multiple models are used together. For example, in the generated image, the model will try to adapt to multiple target features at the same time (such as generating features of a person and the scene at the same time), which may cause the person in the image to look unnatural or the style of the scene to not match the person’s features. Summary of the invention

[0010] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a personalized portrait generation and style conversion method, device and medium, which can realize stable and efficient personalized and stylized image generation.

[0011] The purpose of the present invention can be achieved by the following technical solutions:

[0012] According to a first aspect of the present invention, a personalized portrait generation and style conversion method is provided, comprising:

[0013] Input at least one style image into the style low-rank adaptation model, and perform block low-rank adaptation training on the style low-rank adaptation model; linearly superimpose the weight of the trained style low-rank adaptation model with the weight of the corresponding diffusion model U-net to control the style in the image generation process;

[0014] Input at least one reference portrait into the InsightFace model, extract the portrait pose graph and portrait embedding, and generate pose control information through the portrait control network;

[0015] The diffusion model U-net image generation is performed based on the posture control information generated by the portrait control network, the output of the portrait embedding input into the cross-attention module, the text embedding generated by processing the text control sentence, and the weight after linear superposition.

[0016] Preferably, the method also includes: inputting at least one reference portrait into a portrait low-rank adaptation model, and performing block low-rank adaptation training on the portrait low-rank adaptation model; linearly superimposing the weights of the trained portrait low-rank adaptation model with the weights of the corresponding diffusion model U-net for portrait control during image generation.

[0017] Preferably, the reference portrait input to the portrait low-rank adaptation model and the reference image input to the InsightFace model are images of the same person.

[0018] Preferably, the block low-rank adaptation training is specifically:

[0019] The U-Net of the diffusion model includes an input layer, an intermediate layer, and an output layer; each input layer has a downsampling block, two convolution blocks, and two attention blocks. The output layer uses a convolution block to upsample its input. The intermediate layer does not upsample or downsample the input or output. The features lost by downsampling in the output layer are provided to the output layer through a direct connection between the input layer and the output layer of the same layer.

[0020] The diffusion model U-Net is divided into upper, middle and lower blocks according to the number of downsampling times; for portrait tasks, the upper and lower blocks are trained; for stylized tasks, the middle blocks are trained.

[0021] The parameter update process of the low-rank adaptation is preferably allocated, specifically:

[0022] During training, the network forward calculation process is: h = W0x + ΔWx, W0 is the original network parameter, ΔW is the updated parameter matrix, for the parameter matrix ΔW∈R d×k, using the corresponding two matrices A∈R r×k With B∈R d×r Replace, d, k and r are matrix dimension parameters, satisfy ΔW = BA, and the new forward calculation process is: h = W0x + BAx;

[0023] The block-wise low-rank adaptation method selects different blocks for training for portrait tasks and stylized tasks, and fills the blocks not involved in the training with 0:

[0024]

[0025] Preferably, the text control statement processing generates text embedding, specifically: the text control statement is input into the text embedding generated by the CLIP model.

[0026] Preferably, the diffusion model U-net uses the SDXL network as the backbone network.

[0027] Preferably, during use by the user, a pre-trained portrait control network is directly used; wherein, during the pre-training process of the portrait control network, a five-point posture graph is used to respectively locate the eyes, nose tip and two corners of the mouth of the character.

[0028] According to a second aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, any one of the methods described above is implemented.

[0029] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, any one of the methods described above is implemented.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] (1) The present invention adopts a method of combining block low-rank adaptation with a portrait control network to separate portrait control from style and scene control, thereby avoiding mutual interference between the portrait control model and the style control model. Two different processes are used to control the portrait and style respectively, which greatly reduces the mutual interference problem of similar control signals and improves the generation quality.

[0032] (2) The present invention divides the diffusion model generation network by size, clarifies the impact of different blocks on the generation results, and improves the training efficiency of personalized content.

[0033] (3) The portrait feature information is added to the diffusion model generation network through the cross-attention module, which improves the user's control over the generated portrait.

[0034] (4) During use, the user can directly use the pre-trained portrait control network to generate a single reference portrait without repeated training, thereby protecting the trained portrait control network weight data. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of the personalized portrait generation and style transfer process;

[0036] Figure 2 Generate network and network block structure diagrams for diffusion models;

[0037] Figure 3 It is a schematic diagram of the structure of the low-rank adaptation module;

[0038] Figure 4 Schematic diagram of low-rank adaptation blocks for different tasks;

[0039] Figure 5 This is a schematic diagram of the portrait control network framework structure;

[0040] Figure 6 Schematic diagram of the portrait control network module training process. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0042] Example

[0043] like Figure 1 As shown, this embodiment provides a personalized portrait generation and style conversion method, including:

[0044] Input at least one style image into the style low-rank adaptation model, and perform block low-rank adaptation training on the style low-rank adaptation model; linearly superimpose the weight of the trained style low-rank adaptation model with the weight of the corresponding diffusion model U-net to control the style in the image generation process;

[0045] Input at least one reference portrait into the InsightFace model, extract the portrait pose graph and portrait embedding, and generate pose control information through the portrait control network;

[0046] The diffusion model U-net image generation is performed based on the posture control information generated by the portrait control network, the output of the portrait embedding input into the cross-attention module, the text embedding generated by processing the text control sentence, and the weight after linear superposition.

[0047] In this embodiment, the generation process uses SDXL (stable diffusion XL, diffusion model) as the backbone network. Block low-rank adaptation and portrait control networks are used for generation control respectively. First, block low-rank adaptation training is performed using a reference image and a user-specified identifier (character training is optional), and the training model is directly superimposed on the generation network. The portrait uses the existing InsightFace model to extract the posture control map and facial embedding code, and the generation control is performed through the cross-attention module. The text uses the existing CLIP model to extract the text embedding code and generates control through the cross-attention module.

[0048] Next, the method of this embodiment is introduced in detail.

[0049] (1) Block-based low-rank adaptation (LoRA) style control and portrait control network decoupling control

[0050] This embodiment uses stylization and portrait control to generate images separately.

[0051] In the personalized portrait task, portrait features can be covered by a pre-trained model based on an existing dataset, and the matching portrait encoding model can be used to extract the specified portrait features for controlling image generation. The number of style features is complex and the existing text control is not enough to meet the control requirements, but the identifier method can be used to better control the style.

[0052] (2) Block-wise low-rank adaptation to achieve personalized image elements:

[0053] like Figure 2 As shown in Figure 1, the U-Net of the diffusion model can be divided into three categories: input layer, intermediate layer, and output layer. Each input layer has a downsampling block, two convolution blocks, and two attention blocks. The output layer uses a convolution block to upsample its input, while the intermediate layer does not upsample or downsample the input or output. The features lost by downsampling in the output layer are provided to the output layer through a direct connection between the input layer and the output layer of the same layer.

[0054] like Figure 2As shown in the figure, U-Net is divided into upper layer blocks, middle layer blocks and lower layer blocks through the number of downsampling. The image sizes between each group are similar, and the lower layer blocks have smaller image sizes (lower resolution). The image size is reduced while the window size of each processing remains unchanged, so that the lower layer blocks mainly affect the overall composition of the image, and the higher layer blocks mainly affect the local details of the image.

[0055] Figure 3 The parameter update process of the low-rank adaptation method is shown. For the network forward calculation process h=W0x+ΔWx during training, W0 is the original network parameter and ΔW is the update parameter. Given the matrix ΔW∈R d×k , using A∈R corresponding to the two small matrices r×k With B∈R d×r Replace, where ΔW = BA. Since r < < min (d, k), the number of parameters of the A and B matrices is much smaller than that of the W matrix. The new forward calculation process is h = W0x + ΔWx = W0x + BAx.

[0056] Each module of the generated network is composed of multiple parameter matrices. The block-wise low-rank adaptation method selects different blocks for training for portrait tasks and stylized tasks, and fills the blocks not involved in the training with 0:

[0057]

[0058] The inference process can be used directly like the original low-rank adaptation model. Compared with the original dreambooth-lora, training the entire generative network further reduces the amount of training parameters while ensuring the quality of personalized training. Figure 4 The distribution of training blocks for different tasks is shown. For the portrait task, the upper and lower blocks are trained (this step can be omitted in this method). For the stylization task, the middle block is trained.

[0059] Specifically, this embodiment provides an operation example of a user:

[0060] Steps 1 and 2 are for portrait control network training. Users can directly use the pre-trained network weights during use without repeated training.

[0061] S1. Portrait control network training: CelebA, VGGFace, wikiart, and Danbooru2018 are used to construct portrait datasets. The control signal is the input face pose map. The goal of the model is to generate high-quality human images that meet the specified pose based on these control signals. For example, the head orientation of a given person.

[0062] S2. During the training process of the portrait control network, a five-point posture map is used to locate the eyes, nose tip, and mouth corners of the character respectively, without making specific control over the appearance of the facial features, in order to adapt to the differences in portraits of various styles, such as real portraits, cartoon portraits, and painting portraits.

[0063] S3, stylized low-rank adaptation model training: collect one or more pictures of the same style, and annotate the reference pictures with text using image inference algorithms (such as BLIP, DeepBooru). Manually correct the annotated text and add model identifier text. Note that the identifier text should avoid being similar to natural language as much as possible. Figure 4 Specify the method to train the model in blocks.

[0064] S4, portrait low-rank adaptation model training: collect one or more pictures of the same person, and the rest of the steps are similar to the stylized model. This step is optional.

[0065] S5. Linearly superimpose the low-rank adaptation model weights with the corresponding diffusion model U-net weights to achieve style control during the generation process.

[0066] S6: Input the generated person image. Note that the person in this step is the same person as in S4. Use InsightFace to process the portrait image and output the portrait pose graph and portrait embedding.

[0067] S7. Use InsightFace to process any portrait image and obtain the face posture to control the posture of the generated portrait. This step is optional.

[0068] S8, by default, inputs the posture graph obtained by S6 (or changes the posture through S7), generates posture control information through the portrait control network, and uses it to guide U-net image generation by linearly superimposing it with the denoising result ( Figure 6 ).

[0069] S9, input the portrait embedding obtained in S6, and guide the U-net image generation through the cross attention module.

[0070] S10: Input text containing the identifiers specified in S3 (and S4) and generate text embedding through the CLIP model. Text embedding uses SDXL’s original attention module to guide image generation.

[0071] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0072] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0073] The processing unit performs the various methods and processes described above. For example, in some embodiments, the method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the method described above can be performed. Alternatively, in other embodiments, the CPU can be configured to perform the method in any other appropriate manner (e.g., by means of firmware).

[0074] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), load programmable logic devices (CPLDs), and the like.

[0075] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0076] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0077] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A personalized portrait generation and style transfer method, characterized in that: include: Input at least one style image into the style low-rank adaptation model, and perform block low-rank adaptation training on the style low-rank adaptation model; linearly superimpose the weight of the trained style low-rank adaptation model with the weight of the corresponding diffusion model U-net to control the style in the image generation process; Input at least one reference portrait into the InsightFace model, extract the portrait pose graph and portrait embedding, and generate pose control information through the portrait control network; The diffusion model U-net image generation is performed based on the posture control information generated by the portrait control network, the output of the portrait embedding input into the cross-attention module, the text embedding generated by processing the text control sentence, and the weight after linear superposition.

2. The personalized portrait generation and style transfer method according to claim 1, characterized in that: The method also includes: inputting at least one reference portrait into a portrait low-rank adaptation model, performing block low-rank adaptation training on the portrait low-rank adaptation model; linearly superimposing the weights of the trained portrait low-rank adaptation model with the weights of the corresponding diffusion model U-net for portrait control during image generation.

3. The personalized portrait generation and style conversion method according to claim 2, characterized in that: The reference portrait input to the portrait low-rank adaptation model and the reference image input to the InsightFace model are images of the same person.

4. A personalized portrait generation and style conversion method according to claim 1 or 2, characterized in that: The block-wise low-rank adaptation training is specifically as follows: The U-Net of the diffusion model includes an input layer, an intermediate layer, and an output layer; each input layer has a downsampling block, two convolution blocks, and two attention blocks. The output layer uses a convolution block to upsample its input. The intermediate layer does not upsample or downsample the input or output. The features lost by downsampling in the output layer are provided to the output layer through a direct connection between the input layer and the output layer of the same layer. The diffusion model U-Net is divided into upper, middle and lower blocks according to the number of downsampling times; for portrait tasks, the upper and lower blocks are trained; for stylized tasks, the middle blocks are trained.

5. The personalized portrait generation and style conversion method according to claim 4, characterized in that: The parameter updating process of the allocation low-rank adaptation is specifically as follows: During training, the network forward calculation process is: h = W0x + ΔWx, W0 is the original network parameter, ΔW is the updated parameter matrix, for the parameter matrix ΔW∈R d×k , using the corresponding two matrices A∈R r×k With B∈R d×r Replace, d, k and r are matrix dimension parameters, satisfy ΔW = BA, and the new forward calculation process is: h = W0x + BAx; The block-wise low-rank adaptation method selects different blocks for training for portrait tasks and stylized tasks, and fills the blocks not involved in the training with 0:

6. The personalized portrait generation and style transfer method according to claim 1, characterized in that: The text control statement processing generates text embedding, specifically: the text control statement is input into the text embedding generated by the CLIP model.

7. The personalized portrait generation and style conversion method according to claim 1, characterized in that: The diffusion model U-net uses the SDXL network as the backbone network.

8. The personalized portrait generation and style conversion method according to claim 1, characterized in that: During use by the user, the pre-trained portrait control network is directly used; wherein, during the pre-training process of the portrait control network, a five-point posture graph is used to respectively locate the eyes, nose tip and two corners of the mouth of the character.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images