A virtual hair-changing method and system integrating Transformer architecture and LoRA training

Abstract

The present invention discloses a virtual hairstyle changing method and system integrating Transformer architecture and LoRA training, and relates to the technical field of virtual hairstyle changing. The technical points of the present invention include: obtaining a source image with hair and a hairstyle reference image; extracting hairstyle descriptors from the hairstyle reference image; processing the source image and the hairstyle reference image, and using a hairstyle generation model to generate a first-stage hairstyle changing image; processing the first-stage hairstyle changing image and the source image to obtain an occluded hairstyle image; processing the hairstyle descriptors of the occluded hairstyle image and the hairstyle reference image, and using a hairstyle redrawing DIT model to generate the final hairstyle changing image; wherein the hairstyle redrawing DIT model includes multiple diffusion models based on the Transformer architecture connected in series, and introduces LoRA training for each hairstyle, so that it is more refined when generating more complex hairstyles. The present invention provides users with an efficient and convenient virtual hairstyle changing solution, and also brings an innovative service model to the hairdressing industry.

Description

Technical Field

[0001] The present invention relates to the field of virtual hair-changing technology, and in particular to a virtual hair-changing method and system integrating Transformer architecture and LoRA training. Background Art

[0002] The right hairstyle can well reflect a person's style, and it also plays a very important role in the overall outfit. As the pursuit of beauty continues to deepen, people pay more and more attention to their hairstyle choices. People want to preview the hairstyle effect before trying a new hairstyle, which greatly reduces the unsatisfactory haircut experience. Traditional hairstyle changing technology is usually completed with the help of photo editing tools, especially for more complex hairstyles, which takes a lot of time to process the details.

[0003] Traditional hairstyle changing technology is usually done with the help of photo editing tools. It not only requires finding a picture that matches the angle of the new hairstyle and the person's photo, but also takes a certain amount of time to edit the picture to make it look real and natural. With the development of artificial intelligence technology, virtual hairstyle changing technology has emerged. It can integrate the target hairstyle into the user's photo in a natural way while maintaining the details of the hairstyle and the clarity of the user's facial features. In recent years, although the method based on generative adversarial network (GAN) has dominated the field of virtual hairstyle changing, it still faces challenges in processing hairstyle details and is prone to unnatural artifacts. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a virtual hair style changing method and system that integrates Transformer architecture and LoRA training.

[0005] According to one aspect of the present invention, a virtual hair style change method integrating Transformer architecture and LoRA training is proposed, the method comprising:

[0006] Get source pictures with hair and hairstyle reference pictures;

[0007] Extract hairstyle description words from hairstyle reference pictures;

[0008] Processing the source image and the hairstyle reference image, and generating a first-stage hairstyle change image using a hairstyle generation model;

[0009] Processing the hairstyle change picture in the first stage and the source picture to obtain the blocked hairstyle picture;

[0010] The hairstyle description words of the occluded hairstyle image and the hairstyle reference image are processed, and a final hairstyle change image is generated using a hairstyle redrawing DIT model; wherein the hairstyle redrawing DIT model includes multiple serially connected diffusion models based on the Transformer architecture.

[0011] Furthermore, the processing of the source image and the hairstyle reference image and the use of the hairstyle generation model to generate the first stage hairstyle change image includes: using a baldness generator to process the source image to generate a baldness image; generating the first stage hairstyle change image using the hairstyle generation model according to the hairstyle reference image and the baldness image; wherein the baldness generator includes a VAE encoder, a baldness generation model, a baldness ControlNet, and a VAE decoder, and the baldness generation model and the baldness ControlNet both include multiple serially connected diffusion models based on the Transformer architecture, and the baldness ControlNet is a trainable copy of the baldness generation model.

[0012] Furthermore, the baldness generator is used to process the source image to generate the baldness image, including: inputting the source image into the VAE encoder to obtain a latent space code; inputting the latent space code into the baldness ControlNet, and after block processing and linear layer processing, inputting it into multiple series-connected diffusion models based on the Transformer architecture for processing to obtain source image reference information; inputting the source image reference information into the baldness generation model; randomly generating latent space Gaussian noise, and inputting the noise into the baldness generation model, after block processing and linear layer processing, obtaining a feature map, and inputting the feature map and the source image reference information output by the baldness ControlNet into multiple series-connected diffusion models based on the Transformer architecture for processing, and the obtained output is processed by a multi-layer perceptron and then deblocked; the result after deblocking processing is input into the VAE decoder to obtain a baldness image corresponding to the source image.

[0013] Furthermore, the hairstyle generation model includes multiple diffusion models based on the Transformer architecture connected in series; the generation of the first stage hairstyle change picture using the hairstyle generation model according to the hairstyle reference picture and the bald picture includes: inputting the hairstyle reference picture and the bald picture generated by the bald generator into the pre-trained VAE encoder respectively to obtain the corresponding latent space code; inputting the latent space code corresponding to the hairstyle reference picture into the hairstyle reference network for processing to obtain the hairstyle detail feature; and inputting the hairstyle detail feature into the hairstyle generation model; randomly generating latent space Gaussian noise, and inputting the noise and the latent space code corresponding to the bald picture into the hairstyle generation model together, after block processing and linear layer processing, a feature map is obtained, and the feature map and the hairstyle detail feature output by the hairstyle reference network are input into multiple diffusion models based on the Transformer architecture connected in series for processing, and the obtained output is processed by a multi-layer perceptron and then de-blocked; the result after de-blocking processing is input into the VAE decoder to obtain the first stage hairstyle change picture corresponding to the source picture.

[0014] Furthermore, the processing of the first-stage hairstyle change picture and the source picture to obtain the blocked hairstyle picture includes:

[0015] Perform image segmentation on the hairstyle change picture and the source picture in the first stage respectively to obtain the corresponding segmented pictures;

[0016] Superimpose the two segmented images to obtain the hairstyle binary mask ;

[0017] Hairstyle binary mask Expand and obtain the binary mask of the expanded hairstyle ;

[0018] The binary mask of the expanded hairstyle Superimpose it with the hairstyle change picture in the first stage to obtain the blocked hairstyle picture.

[0019] Furthermore, the processing of the hairstyle description words of the blocked hairstyle picture and the hairstyle reference picture and using the hairstyle redrawing DIT model to generate the final hairstyle change picture includes:

[0020] Perform text encoding on the hairstyle description words to obtain the text encoding ;

[0021] Input the occluded hairstyle image into the VAE encoder to obtain the redrawn background latent code;

[0022] Generate random Gaussian noise in the latent space, redraw the background hidden code, the random Gaussian noise in the latent space, and the binary mask of the hairstyle after expansion Splice along the channels to get a multi-channel input matrix;

[0023] The multi-channel input matrix is ​​divided into blocks, then processed by the linear layer, and the processed feature maps and text encoding The data are then fed into multiple serially connected diffusion models based on the Transformer architecture for cyclic denoising.

[0024] The obtained output is processed by a multi-layer perceptron and then deblocked to restore it to the size of the input occluded hairstyle image; the result after deblocking is input into the VAE decoder to obtain the final hairstyle image.

[0025] Furthermore, the diffusion model based on the Transformer architecture is divided into an encoding block and a decoding block; wherein the encoding block is used to compress the input image to obtain features of different levels of the image, and the encoding block includes a self-attention module, a cross-attention module, and a forward propagation network; the decoding block is used to restore the image size, and the decoding block includes a self-attention module, a cross-attention module, a forward propagation network, and a jump module.

[0026] Furthermore, the hairstyle generation model and the hairstyle redrawing DIT model are both pre-trained models, and in the inference process of the hairstyle redrawing DIT model, the trained low-rank adaptive fine-tuning model is used to fine-tune the weight parameters of the diffusion model based on the Transformer architecture in the hairstyle redrawing DIT model, and the fine-tuning formula is:

[0027]

[0028] In the formula, represents the weight after fine-tuning; represents the original weight parameters of the diffusion model based on the Transformer architecture, represents the parameters of the low-rank adaptive fine-tuning model, for The two matrices obtained by decomposition.

[0029] Furthermore, the loss function in the hairstyle generation model training process is as follows:

[0030]

[0031] in, represents Gaussian noise; represents the VAE encoder; Represents the diffusion model based on the Transformer architecture in the hairstyle generation model; represents the hairstyle reference network; They represent the hairstyle reference pictures and bald pictures respectively; represents the latent space encoding; t represents the time step; express Distribution of expectations;

[0032] The loss function during the training of the low-rank adaptive fine-tuning model is as follows:

[0033]

[0034] in, Represents the diffusion model based on Transformer architecture in the low-rank adaptive fine-tuning model; Indicates text encoding; express Distribute expectations.

[0035] According to another aspect of the present invention, a virtual hair-changing system integrating Transformer architecture and LoRA training is proposed, the system comprising:

[0036] A picture acquisition module, configured to acquire a source picture with hair and a hairstyle reference picture;

[0037] a hairstyle description extraction module configured to extract hairstyle description words of a hairstyle reference picture;

[0038] an initial hairstyle change picture generation module, configured to process the source picture and the hairstyle reference picture, and generate a first-stage hairstyle change picture using a hairstyle generation model;

[0039] A masked hairstyle picture generation module, configured to process the first-stage hairstyle change picture and the source picture to obtain a masked hairstyle picture;

[0040] The hairstyle change picture generation module is configured to process the hairstyle description words of the occluded hairstyle picture and the hairstyle reference picture, and generate the final hairstyle change picture using the hairstyle redrawing DIT model; wherein the hairstyle redrawing DIT model includes multiple serially connected diffusion models based on the Transformer architecture.

[0041] The present invention has the following technical effects:

[0042] The present invention proposes a virtual hairstyle changing method and system that integrates Transformer architecture and LoRA training. First, a source image with hair and a hairstyle reference image are obtained; and the hairstyle descriptor of the hairstyle reference image is extracted; then the source image and the hairstyle reference image are processed, and a first-stage hairstyle changing image is generated using a hairstyle generation model; the first-stage hairstyle changing image and the source image are processed to obtain an occluded hairstyle image; the hairstyle descriptors of the occluded hairstyle image and the hairstyle reference image are processed, and the final hairstyle changing image is generated using a hairstyle redrawing DIT model; wherein the hairstyle redrawing DIT model includes multiple diffusion models based on the Transformer architecture connected in series, and introduces LoRA training for each hairstyle, so that it is more refined when generating more complex hairstyles.

[0043] The present invention proposes a two-stage hairstyle changing method. In the first stage, a rough hairstyle changing picture is generated: first, a bald picture is generated to reduce the influence of the original user's hairstyle, and then a hairstyle changing picture is generated, and the hairstyle information is injected through the hairstyle cross-attention module; in the second stage, for each hairstyle's LoRA, during the inference process, a hairstyle redrawing DiT model loaded with the hairstyle LoRA is used to redraw the hairstyle changing picture more finely, which is more friendly to complex hairstyles.

[0044] The present invention can provide users with an efficient and convenient virtual hairstyle changing solution, and also bring an innovative service model to the hairdressing industry; the present invention can bring a new hairdressing experience to more people and promote the development of personalized hairdressing services. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 It is a flowchart of a virtual hair style changing method that integrates Transformer architecture and LoRA training provided by an embodiment of the present invention.

[0047] Figure 2 This is a complete flowchart of a virtual hair style changing method that integrates Transformer architecture and LoRA training provided by an embodiment of the present invention.

[0048] Figure 3 This is a flow chart of using a hairstyle generation model to generate a first-stage hairstyle change picture in an embodiment of the present invention.

[0049] Figure 4 It is a flowchart of a baldness generator generating a baldness picture in an embodiment of the present invention.

[0050] Figure 5 It is a flowchart of the hairstyle generation model generating a hairstyle change picture in an embodiment of the present invention.

[0051] Figure 6 It is a flowchart of the training process of the baldness generator and the hairstyle generation model in the embodiment of the present invention.

[0052] Figure 7 It is a flowchart of using the hairstyle redrawing DIT model to generate the final hairstyle change picture in an embodiment of the present invention.

[0053] Figure 8 It is a flowchart of the training process of the hairstyle redrawing DIT model in an embodiment of the present invention.

[0054] Fig. 9 It is a structural diagram of a virtual hair-changing system that integrates Transformer architecture and LoRA training provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0056] The embodiment of the present invention proposes a virtual hair-changing method that integrates the Transformer architecture and LoRA training, such as Figure 1 As shown, the method includes:

[0057] S1. Obtain a source image with hair and a hairstyle reference image;

[0058] S2, extracting hairstyle description words from the hairstyle reference picture;

[0059] S3, processing the source image and the hairstyle reference image, and generating a first-stage hairstyle change image using a hairstyle generation model;

[0060] S4, processing the hairstyle change picture in the first stage and the source picture to obtain an occluded hairstyle picture;

[0061] S5. Process the hairstyle description words of the occluded hairstyle image and the hairstyle reference image, and generate a final hairstyle change image using a hairstyle redrawing DIT model; wherein the hairstyle redrawing DIT model includes multiple serially connected diffusion models based on the Transformer architecture.

[0062] Figure 2 A complete flowchart of a virtual hair style change method integrating Transformer architecture and LoRA training provided by an embodiment of the present invention is shown.

[0063] The method starts from S1. In S1, a source image with hair and a hairstyle reference image are obtained.

[0064] According to an embodiment of the present invention, after acquiring a source image of a person with hair, the source image can be processed to obtain a source image that meets the size requirements. For example, the source image is a person image uploaded by a user. , first detect the face, then align the face, and then crop and resize it to the required size.

[0065] The hairstyle reference picture is a pre-processed picture with a specific hairstyle. For example, a user can select a hairstyle picture of his / her own preference from a plurality of hairstyle reference pictures in the hairstyle library.

[0066] Then S2 is executed, in which the hairstyle description words of the hairstyle reference image are extracted.

[0067] According to an embodiment of the present invention, the hairstyle reference picture selected by the user is input into a pre-trained model for descriptive word extraction. The pre-trained model may be, for example, a Blip large model. During training, multiple hairstyle reference pictures are prepared for each hairstyle as a training set, for example, 5-10 hairstyle reference pictures from different angles; the Blip large model is used to mark each frontal hairstyle picture in the training set, and the keywords of the hairstyle reference picture, i.e., the hairstyle descriptive words, are output, such as: black, long straight hair, etc.

[0068] Then execute S3, in which the source image and the hairstyle reference image are processed, and the hairstyle generation model is used to generate the first stage hairstyle change image. The specific process includes: S31, using the baldness generator to process the source image to generate a baldness image; S32, according to the hairstyle reference image and the baldness image, using the hairstyle generation model to generate the first stage hairstyle change image; wherein, the baldness generator includes a VAE encoder, a baldness generation model, a baldness ControlNet, and a VAE decoder, and the baldness generation model and the baldness ControlNet both include multiple serially connected diffusion models based on the Transformer architecture, and the baldness ControlNet is a trainable copy of the baldness generation model.

[0069] Figure 3 A flow chart of using a hairstyle generation model to generate a first-stage hairstyle change picture in an embodiment of the present invention is shown.

[0070] According to an embodiment of the present invention, in order to avoid the influence of the user's own hairstyle, the baldness generator will first generate a picture of the user without hair, and then input the generated baldness picture into the hairstyle generation model to generate a hairstyle change picture. The baldness generator includes a VAE encoder - , Baldness Generative Model - , Bald ControlNet- and a VAE decoder. Its function is to generate a bald picture of the person in the source picture, that is, a picture without hair. In this way, in the subsequent hairstyle generation process, the influence of the hairstyle of the user's original picture can be avoided.

[0071] Usually, diffusion models use the latent space diffusion model - Stable Diffusion, which is the classic latent space diffusion model (LDM, Latent Diffusion Model). LDM uses variational autoencoder VAE. Given an input image, the image is first mapped to the latent space through the encoder, so that diffusion is carried out in the latent space. In the forward process of diffusion, Gaussian noise will be gradually added to the latent variable in T iterations. The reverse diffusion process is to gradually remove noise, and finally restore it to RGB space through the decoder to obtain the final image.

[0072] The backbone structure of the diffusion model in the embodiment of the present invention adopts Transformer, and this architecture is called DiT (Diffusion Transformer, a diffusion model based on Transformer architecture). Compared with the traditional diffusion model structure, the Transformer structure has better effects in maintaining spatial features and generating stability. At the same time, the Transformer structure has strong scalability. The more high-quality training data, the more information can be learned; the larger the architecture, the better the generation effect. Diffusion in the latent space at the same time can save a lot of calculations and obtain the same good generation effect.

[0073] Baldness Generative Model - There are 32 DiT modules in it, of which there are two types of DiT modules, namely encoding block and decoding block. Both the encoding block and the decoding block contain self-attention module, cross-attention module and forward propagation network. The decoding block additionally contains a skip module to add the information of the encoding block to the decoding block. The encoding block will gradually compress the size of the image and obtain the features of different levels of the image; the decoding block will gradually restore the size of the image.

[0074] The ControlNet branch of DiT has the same effect as the ControlNet branch of the traditional diffusion model, creating a trainable copy of some DiT modules (i.e., the bald ControlNet- ), and add an additional zero convolution layer at the connection with each decoding block of the bald generative model. When the bald ControlNet starts training, the weights and biases of the zero convolution layer are initialized to zero, so the feedforward process is the same as the process without the ControlNet branch. Before any optimization, there will be no impact on the deep neural features; the output of each replica block is added to the jump connection of the bald generative model, so that the feature map output by the encoding block of the bald ControlNet is added to the feature map output by the decoding block of the DiT module in the bald generative model.

[0075] In this embodiment, preferably, Figure 4 As shown, in S31, the source image is processed by the baldness generator, and the process of generating the baldness image includes:

[0076] S311. Input the source image into the VAE encoder to obtain the latent space encoding ; Encode the latent space After block processing, it is processed through a linear layer and input into the baldness ControlNet, which is a trainable copy of the baldness generation model. After a zero convolution layer, the baldness ControlNet is connected to the baldness generation model through a residual connection, and the reference information of the source image is input into the baldness generation model.

[0077] S312, Randomly sampled 4-channel latent space Gaussian noise , input into the baldness generation model, and after block division, the input Blocking ( are the width and height of the input image respectively) blocks, It is usually set to 2, and then processed by a linear layer to obtain a feature map; then it passes through multiple DiT modules (that is, multiple diffusion models based on the Transformer architecture connected in series), and the output is processed by a multi-layer perceptron (MLP) and then de-blocked, that is, the blocks are rearranged into their original spatial layout.

[0078] S313, then pass through the VAE decoder to output the bald picture of the person in the source picture in RGB space .

[0079] In this embodiment, the baldness generator and the hairstyle generation model are both pre-trained models, wherein only the baldness ControlNet is trained during the training of the baldness generator. The weights of the baldness generation model, VAE encoder, and VAE decoder are fixed, and the pre-trained weights of the mixed-element DiT are used. The baldness ControlNet participates in the training, and the initial weights also use the pre-trained weights of the mixed-element DiT. By minimizing the loss function, the Adam optimizer is used for optimization iteration. The loss function for training the baldness generator is as follows:

[0080]

[0081] in, represents Gaussian noise; represents the VAE encoder; Represents the diffusion model based on the Transformer architecture in the bald generator; It means bald ControlNet; Indicates the source image; represents the latent space encoding; t represents the time step; express Distribute expectations.

[0082] In this embodiment, preferably, Figure 5 As shown, the process of generating a hairstyle change picture using a hairstyle generation model according to the hairstyle reference picture and the bald picture in S32 includes:

[0083] S321. Use the hairstyle reference picture And bald pictures generated by baldness generator Input the pre-trained VAE encoder respectively to obtain the corresponding latent space encoding and ;here;

[0084] S322, encoding the latent space corresponding to the hairstyle reference image Input hairstyle reference network Processing is performed to obtain detailed features of the hairstyle ; and the hairstyle details feature Input into the hairstyle generation model;

[0085] S323, Randomly generate Gaussian noise in latent space , and encode the latent space corresponding to the noise and bald picture The hair style generation model is input together, and after block processing and linear layer processing, a feature map is obtained. The feature map and the hair style detail features output by the hair style reference network are combined. The data is then fed into multiple serially connected diffusion models based on the Transformer architecture (i.e., DiT modules) for denoising. The output is processed by a multi-layer perceptron and then deblocked to restore it to the input size.

[0086] S324. Input the result after the deblocking process into the pre-trained VAE decoder to obtain a hair-changing picture corresponding to the RGB space source picture.

[0087] In this embodiment, the hairstyle reference network The structure of is the same as the hairstyle generation model, which is also trained based on the pre-trained weights of Hunyuan DiT. Figure 5 As shown, the specific process of S322 includes:

[0088] S3221. Encode the latent space It is divided into blocks and then processed by the linear layer;

[0089] S3222, after being processed by the linear layer, is input into multiple serially connected diffusion models based on the Transformer architecture to extract the detailed features of the hairstyle , and is input into the hairstyle generation model through its cross-attention module.

[0090] In this embodiment, the hairstyle generation model It is trained based on the pre-trained weights of the mixed DiT. Its function is to generate pictures of users with changed hairstyles. During the training of the bald generator, both the hairstyle reference network and the hairstyle generation model participate in the training. The weights of the VAE encoder and the VAE decoder are fixed, and the hairstyle reference network and the hairstyle generation model use the pre-trained weights of the mixed DiT. By minimizing the loss function, the Adam optimizer is used for optimization iteration. The loss function for the hairstyle generation model training is as follows:

[0091]

[0092] in, Represents the diffusion model based on the Transformer architecture in the hairstyle generation model; represents the hairstyle reference network; They represent the hairstyle reference pictures and bald pictures respectively; represents the latent space encoding; t represents the time step; express Distribute expectations.

[0093] In the process of pre-training the baldness generator and the hairstyle generation model, the baldness generator generates bald pictures from pictures with hair, and the hairstyle generation model generates pictures with changed hairstyles from bald pictures. Therefore, the training data set selects bald pictures and pictures with hair of the same person; in the inference stage, the hairstyle reference picture and the source picture may not be pictures of the same person. It should be noted that during the training process, the source picture , Hairstyle Reference Pictures And bald pictures Both require image processing to crop and resize to the required size. Figure 6 A flowchart of the training process of the baldness generator and the hairstyle generation model is shown.

[0094] Then, S4 is executed, in which the first-stage hairstyle change picture and the source picture are processed to obtain the blocked hairstyle picture.

[0095] According to the embodiment of the present invention, since the hair-changing picture generated in the first stage is relatively rough, in order to solve this problem, a refined redrawing process in the second stage is introduced.

[0096] First, perform the following processing on the first stage hairstyle change picture and the source picture to obtain the occluded hairstyle picture:

[0097] S41, performing image segmentation on the hairstyle change picture and the source picture in the first stage respectively to obtain corresponding segmented pictures;

[0098] S42, superimpose the two segmented images to obtain the hairstyle binary mask ;

[0099] S43, binary mask for hairstyle Expand and obtain the binary mask of the expanded hairstyle ;

[0100] S44, binary mask the hairstyle after expansion Superimpose it with the hairstyle change picture in the first stage to obtain the blocked hairstyle picture.

[0101] Then, S5 is executed. In S5, the hairstyle description words of the occluded hairstyle image and the hairstyle reference image are processed, and a hairstyle redrawing DIT model is used to generate a final hairstyle change image; wherein the hairstyle redrawing DIT model includes multiple serially connected diffusion models based on the Transformer architecture. Figure 7 A flowchart of generating a final hairstyle change picture by using a hairstyle redrawing DIT model in an embodiment of the present invention is shown.

[0102] According to an embodiment of the present invention, the goal of the redrawing diffusion model is to redraw the image required by the user in the user-specified area of ​​a picture, and ensure that the drawn area can be harmoniously integrated with the original image. The redrawing diffusion model is used in virtual hairstyle change, aiming to strip the hairstyle of the source picture, redraw the target hairstyle trained by the hairstyle LoRA on the model, so that the other parts remain in the original state, and the generated hairstyle is more realistic and natural.

[0103] like Figure 7 As shown in FIG. 1 , the specific process of processing the hairstyle description words of the occluded hairstyle image and the hairstyle reference image and using the hairstyle redrawing DIT model to generate the final hairstyle change image includes:

[0104] S51, text encoding the hairstyle description word to obtain a text encoding Here, for example, CLIP text encoder (a pre-trained model based on contrastive text-image pairs) and T5 (text-to-text transfer Transformer) can be used to encode the hairstyle description words to obtain the text encoding .

[0105] S52, cover the hairstyle picture Input to VAE encoder In the above example, we get the 4-channel redrawn background hidden code .

[0106] S53. Generate random Gaussian noise in latent space , implicitly encode the 4-channel redrawn background , 4-channel latent spatial noise And the 1-channel hairstyle binary mask map obtained by S43 , spliced ​​along the channel to obtain 9-channel input.

[0107] S54, the 9-channel input is divided into blocks, then processed by the linear layer, and the processed feature map and text are encoded They are then input into multiple serially connected diffusion models based on the Transformer architecture for cyclic denoising.

[0108] S55, after all DiT modules of the hairstyle redrawing DiT model are processed by a multi-layer perceptron, then deblocked and restored back to the input size, and finally passed through a VAE decoder to obtain a more refined hairstyle change picture.

[0109] The above hairstyle redrawing DIT model is a pre-trained model, in which multiple diffusion models based on the Transformer architecture are connected in series The pre-trained mixed DiT is used as the pre-trained weight. Among them, for each hairstyle, a low-rank adaptive fine-tuning (LoRA) model is pre-trained. LoRA is a lightweight model fine-tuning method that improves parameter efficiency by fixing the pre-trained weights and integrating additional trainable low-rank matrices in the neural network. Specifically, the fine-tuning formula is as follows:

[0110]

[0111] In the formula, represents the weight after fine-tuning; Represents the original weight parameters of the diffusion model based on the Transformer architecture, that is, the pre-trained weights, ; represents the parameters of the low-rank adaptive fine-tuning model, for The two matrices obtained by decomposition are: , and the rank .

[0112] This approach not only reduces the training burden, but also enables the model to quickly capture the unique characteristics of each hairstyle, thereby significantly improving the quality of generated images while maintaining training efficiency.

[0113] The weights trained with LoRA are applied during the inference process of the hairstyle redrawing DiT model, such as Figure 8 As shown, the reasoning process of the hairstyle redrawing DiT model is as follows:

[0114] First, the hairstyle description words are input into the pre-trained CLIP text encoder and T5 to obtain the text encoding , which is input into the hairstyle redrawing DiT model through the text cross-attention module;

[0115] Then, the hairstyle picture Input into the VAE encoder to get the latent space encoding , plus randomly generated latent space Gaussian noise , block processing, and then processed by linear layer, input into multiple series-connected diffusion models based on Transformer architecture, and apply the above-mentioned fine-tuned weight loading to the diffusion model based on Transformer architecture; here, the hairstyle picture The image segmentation is obtained by performing image segmentation on the hairstyle reference image. For example, an open source image segmentation model - SAM (Segment Anything) can be used to perform image segmentation on the hairstyle reference image.

[0116] Finally, the output of all DiT modules is processed by a multi-layer perceptron, then restored to the input size through deblocking, and passed through a VAE decoder to obtain the final generated hairstyle image in RGB space.

[0117] In the process of training the hairstyle redrawing DiT model, the Adam optimizer is used to perform optimization iterations by minimizing the loss function. The weights of the diffusion model are fixed, and only the hairstyle LoRA model (i.e., the low-rank adaptive fine-tuning model) is trained. The loss function for training the hairstyle LoRA model is as follows:

[0118]

[0119] in, Represents the diffusion model based on Transformer architecture in the low-rank adaptive fine-tuning model; Indicates text encoding; express Distribute expectations.

[0120] The present invention uses the Transformer architecture to provide users with an efficient, economical and highly realistic hairstyle selection service. Through this service, users can select their favorite styles from a variety of hairstyles and use the virtual try-on function to directly preview the effects of these hairstyles on their own heads. This service not only allows users to preview and create new virtual images, but also helps avoid disappointment caused by hairstyles that are not as expected. Through precise simulation technology, it is ensured that what users see is what they will get, and the hairstyle effects are lifelike, bringing users an unprecedented personalized experience.

[0121] Barber shops or wig sellers can use this tool to put all popular hairstyles on the shelves of the hairstyle library. Users can choose their favorite styles from the rich hairstyle library and use the virtual try-on function to intuitively see the effects of these hairstyles on their own image. By intuitively feeling what the hairstyle will look like after changing, it can help users make decisions and help merchants reduce the rate of negative reviews, etc. This technology provides merchants with a new service idea through convenient and natural hairstyle changing technology, providing customers with a better consumption experience.

[0122] Another embodiment of the present invention proposes a virtual hair-changing system that integrates the Transformer architecture and LoRA training, such as Fig. 9 As shown, the system includes:

[0123] The picture acquisition module 910 is configured to acquire a source picture with hair and a hairstyle reference picture;

[0124] a hairstyle description extraction module 920 configured to extract hairstyle description words of the hairstyle reference picture;

[0125] An initial hairstyle picture generation module 930 is configured to process the source picture and the hairstyle reference picture, and generate a first-stage hairstyle picture using a hairstyle generation model;

[0126] The occluded hairstyle picture generating module 940 is configured to process the first-stage hairstyle changing picture and the source picture to obtain the occluded hairstyle picture;

[0127] The hairstyle change picture generation module 950 is configured to process the hairstyle description words of the occluded hairstyle picture and the hairstyle reference picture, and generate the final hairstyle change picture using the hairstyle redrawing DIT model; wherein the hairstyle redrawing DIT model includes multiple serially connected diffusion models based on the Transformer architecture.

[0128] For the undetailed parts of a virtual hair-changing system that integrates Transformer architecture and LoRA training according to an embodiment of the present invention, please refer to the above detailed description of the method embodiment.

[0129] It should be noted that the terms used in the present invention are only for describing specific embodiments, rather than limiting the scope of the present application. As shown in the present specification, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular, but may also include the plural. The terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of more restrictions, the elements defined by the sentence "include one..." do not exclude the presence of other identical elements in the process, method or device including the elements.

[0130] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments may still be modified, or some or all of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A virtual hair-changing method integrating Transformer architecture and LoRA training, characterized in that: include: Get source pictures with hair and hairstyle reference pictures; Extract hairstyle description words from hairstyle reference pictures; Processing the source image and the hairstyle reference image, and generating the first-stage hairstyle change image using the hairstyle generation model; the hairstyle generation model includes a plurality of serially connected diffusion models based on the Transformer architecture; Process the hairstyle change pictures and source pictures in the first stage to obtain the occluded hairstyle pictures; The hairstyle description words of the occluded hairstyle picture and the hairstyle reference picture are processed, and the final hairstyle change picture is generated by using the hairstyle redrawing DIT model; wherein the hairstyle redrawing DIT model includes multiple serially connected diffusion models based on the Transformer architecture; including: text encoding the hairstyle description words to obtain the text encoding ; Input the occluded hairstyle image into the VAE encoder to obtain the redrawn background hidden code; Generate latent space random Gaussian noise, and use the redrawn background hidden code, latent space random Gaussian noise and the dilated hairstyle binary mask Splice along the channel to get a multi-channel input matrix; divide the multi-channel input matrix into blocks, then process it through the linear layer, and encode the processed feature map and text The images are input into multiple serially connected diffusion models based on the Transformer architecture for cyclic denoising. The output is processed by a multi-layer perceptron and then deblocked to restore the input occluded hairstyle image size. The deblocked result is input into the VAE decoder to obtain the final hairstyle image. The hairstyle generation model and the hairstyle redrawing DIT model are both pre-trained models, and the loss function in the hairstyle generation model training process is as follows: ； In the formula, represents Gaussian noise; represents the VAE encoder; Represents the diffusion model based on the Transformer architecture in the hairstyle generation model; represents the hairstyle reference network; They represent the hairstyle reference pictures and bald pictures respectively; represents the latent space encoding; t represents the time step; express Distribution of expectations; During the inference process of the hairstyle redrawing DIT model, the trained low-rank adaptive fine-tuning model is used to fine-tune the weight parameters of the diffusion model based on the Transformer architecture in the hairstyle redrawing DIT model. The fine-tuning formula is: ； In the formula, represents the weight after fine-tuning; represents the original weight parameters of the diffusion model based on the Transformer architecture, represents the parameters of the low-rank adaptive fine-tuning model, for The two matrices obtained by decomposition; The loss function during the training of the low-rank adaptive fine-tuning model is as follows: ； In the formula, Represents the diffusion model based on Transformer architecture in the low-rank adaptive fine-tuning model; Indicates text encoding; express Distribute expectations.

2. According to claim 1, a virtual hair style change method integrating Transformer architecture and LoRA training is characterized in that: The method of processing the source image and the hairstyle reference image and generating the first-stage hairstyle change image using the hairstyle generation model includes: processing the source image using a baldness generator to generate a baldness image; generating the first-stage hairstyle change image using the hairstyle generation model according to the hairstyle reference image and the baldness image; wherein the baldness generator includes a VAE encoder, a baldness generation model, a baldness ControlNet, and a VAE decoder, and the baldness generation model and the baldness ControlNet both include multiple serially connected diffusion models based on the Transformer architecture, and the baldness ControlNet is a trainable copy of the baldness generation model.

3. According to claim 2, a virtual hair style change method integrating Transformer architecture and LoRA training is characterized in that: The method uses the baldness generator to process the source image to generate the baldness image, including: inputting the source image into the VAE encoder to obtain a latent space code; inputting the latent space code into the baldness ControlNet, and after block processing and linear layer processing, inputting it into multiple diffusion models based on the Transformer architecture in series for processing to obtain source image reference information; inputting the source image reference information into the baldness generation model; randomly generating latent space Gaussian noise, and inputting the noise into the baldness generation model, after block processing and linear layer processing, obtaining a feature map, and inputting the feature map and the source image reference information output by the baldness ControlNet into multiple diffusion models based on the Transformer architecture in series for processing, and the obtained output is processed by a multi-layer perceptron and then deblocked; inputting the result after the deblocking processing into the VAE decoder to obtain the baldness image corresponding to the source image.

4. According to claim 3, a virtual hair style change method integrating Transformer architecture and LoRA training is characterized in that: The method of generating a first-stage hairstyle change picture using a hairstyle generation model according to a hairstyle reference picture and a bald picture includes: inputting the hairstyle reference picture and the bald picture generated by a bald generator into a pre-trained VAE encoder respectively to obtain corresponding latent space codes; inputting the latent space codes corresponding to the hairstyle reference picture into a hairstyle reference network for processing to obtain hairstyle detail features; and inputting the hairstyle detail features into the hairstyle generation model; randomly generating latent space Gaussian noise, and inputting the noise and the latent space codes corresponding to the bald picture into the hairstyle generation model together, and obtaining a feature map after block processing and linear layer processing, and inputting the feature map and the hairstyle detail features output by the hairstyle reference network into a plurality of serially connected diffusion models based on a Transformer architecture for processing, and the obtained output is processed by a multi-layer perceptron and then deblocked; and inputting the result after deblocking into a VAE decoder to obtain a first-stage hairstyle change picture corresponding to the source picture.

5. According to claim 1, a virtual hair style change method integrating Transformer architecture and LoRA training is characterized in that: The processing of the first-stage hairstyle change picture and the source picture to obtain the blocked hairstyle picture includes: Perform image segmentation on the hairstyle change picture and the source picture in the first stage respectively to obtain the corresponding segmented pictures; Superimpose the two segmented images to obtain the hairstyle binary mask ; Hairstyle binary mask Expand and obtain the binary mask of the expanded hairstyle ; The binary mask of the expanded hairstyle Superimpose it with the hairstyle change picture in the first stage to obtain the blocked hairstyle picture.

6. A virtual hair style change method integrating Transformer architecture and LoRA training according to any one of claims 1-5, characterized in that: The diffusion model based on the Transformer architecture is divided into an encoding block and a decoding block; wherein the encoding block is used to compress the input image to obtain features of different levels of the image, and the encoding block includes a self-attention module, a cross-attention module, and a forward propagation network; the decoding block is used to restore the image size, and the decoding block includes a self-attention module, a cross-attention module, a forward propagation network, and a jump module.

7. A virtual hair-changing system integrating Transformer architecture and LoRA training, characterized in that: include: A picture acquisition module, configured to acquire a source picture with hair and a hairstyle reference picture; a hairstyle description extraction module configured to extract hairstyle description words of a hairstyle reference picture; The initial hairstyle picture generation module is configured to process the source picture and the hairstyle reference picture, and generate the first stage hairstyle picture using the hairstyle generation model; the hairstyle generation model includes a plurality of serially connected diffusion models based on the Transformer architecture; the hairstyle generation model is a pre-trained model, and the loss function in the hairstyle generation model training process is as follows: ； In the formula, represents Gaussian noise; represents the VAE encoder; Represents the diffusion model based on the Transformer architecture in the hairstyle generation model; represents the hairstyle reference network; They represent the hairstyle reference pictures and bald pictures respectively; represents the latent space encoding; t represents the time step; express Distribution of expectations; A masked hairstyle picture generation module, configured to process the first-stage hairstyle change picture and the source picture to obtain a masked hairstyle picture; The hairstyle change picture generation module is configured to process the hairstyle description words of the occluded hairstyle picture and the hairstyle reference picture, and generate the final hairstyle change picture using the hairstyle redrawing DIT model; wherein the hairstyle redrawing DIT model includes a plurality of diffusion models based on the Transformer architecture connected in series; including: text encoding the hairstyle description words to obtain the text encoding ; Input the occluded hairstyle image into the VAE encoder to obtain the redrawn background hidden code; Generate latent space random Gaussian noise, and use the redrawn background hidden code, latent space random Gaussian noise and the dilated hairstyle binary mask Splice along the channel to get a multi-channel input matrix; divide the multi-channel input matrix into blocks, then process it through the linear layer, and encode the processed feature map and text The images are input into multiple serially connected diffusion models based on the Transformer architecture for cyclic denoising. The output is processed by a multi-layer perceptron and then deblocked to restore the input occluded hairstyle image size. The deblocked result is input into the VAE decoder to obtain the final hairstyle image. The hairstyle redrawing DIT model is a pre-trained model, and in the inference process of the hairstyle redrawing DIT model, the trained low-rank adaptive fine-tuning model is used to fine-tune the weight parameters of the diffusion model based on the Transformer architecture in the hairstyle redrawing DIT model. The fine-tuning formula is: ； In the formula, represents the weight after fine-tuning; represents the original weight parameters of the diffusion model based on the Transformer architecture, represents the parameters of the low-rank adaptive fine-tuning model, for The two matrices obtained by decomposition; The loss function during the training of the low-rank adaptive fine-tuning model is as follows: ； In the formula, Represents the diffusion model based on Transformer architecture in the low-rank adaptive fine-tuning model; Indicates text encoding; express Distribute expectations.

Images