Virtual fitting method and device

Abstract

The invention provides a virtual fitting method and device, multi-target personalized virtual fitting is realized through a multi-target and multi-mode virtual fitting model, and the virtual fitting device comprises a multi-mode feature interlaced learning attention module (MFILA) and a parallel denoising UNet module. Wherein the MFILA module realizes multi-modal condition feature coding through a multi-modal feature fusion module (MFF) and a multi-reference target texture feature extraction module (MRSTF), and the parallel denoising UNet module is mainly responsible for denoising generation. A user uploads a model clothes graph, a target fitting clothes style graph and corresponding text cues, and the system gradually generates a denoised virtual fitting result from pure noise through an MM-VTON model by taking the inputs as conditions. According to the method, the reality sense and the accuracy of virtual fitting are improved by utilizing an advanced image generation technology and multi-modal learning, meanwhile, the demand of computing resources is reduced, and the method has relatively high practical application value and a wide market prospect.

Description

Technical Field

[0001] The present invention belongs to the field of computer and network technology, and in particular to a virtual fitting method and device. Background Art

[0002] With the rapid development of e-commerce, consumers' demands and expectations for online shopping are constantly increasing. Traditional online shopping methods lack the actual trial experience, which makes it difficult for consumers to accurately judge the fit and style of clothing, greatly limiting consumers' shopping experience and purchasing desire.

[0003] Virtual fitting technology can match the user's body data with the digital model of clothing by using visual modeling and deep learning algorithms to generate realistic fitting effects. However, current methods usually only support single-target fitting tasks (tops, bottoms or dresses), and cannot customize dressing styles and clothing details based on text prompts. It is difficult to meet the high fidelity and robustness required for applications in the real world, which hinders the application scope and user experience of virtual fitting technology. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention proposes a virtual fitting method and device.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] In one aspect, the present invention provides a virtual fitting method, comprising the following steps:

[0007] S1: In order to meet the model's requirements for multi-target and multi-modality, starting from the existing model image and clothing style image datasets, the multimodal models GPT-4V and CogVLM are used to describe the model's clothing in the required format as text prompts. Then, the open source vocabulary detection model (BERT / SpaCy) is used to highlight the key prompts of each target clothing body. Next, the SAM model is used to generate clothing image bounding boxes corresponding to the key prompts on the existing model images. Finally, the clothing style images, the text prompts that have highlighted the keywords, and the model images containing the clothing bounding boxes are combined to construct data sample pairs of image bodies, text entities, and target spatial layouts.

[0008] S2: The MM-VTON model (Multi-subject, multi-modal-Virtual Try-on) is established to realize multi-target personalized virtual fitting. The MM-VTON consists of a multi-modal feature interleaved learning attention module (MFILA) and a parallel denoising UNet module, which respectively realize multi-modal conditional feature encoding and diffusion denoising probability generation of virtual fitting images.

[0009] S3: The data sample is input into the MM-VTON model for multimodal feature encoding. The MFILA module is responsible for processing virtual fitting of multiple garments, and the potential distortions of different types of garments are aligned through the self-attention layer, while other network architectures responsible for general feature extraction are shared between different conditional encoding branches to reduce the number of parameters and computing resource requirements;

[0010] S4: Implementing feature injection of the multimodal feature encoding in the parallel denoising UNet module through the position feature encoding module to enhance the scalability of the model under multi-objective conditions;

[0011] S5: Design the total loss function L of the MM-VTON model enc To optimize the parameters of each module of S2, and further improve the ability of MM-VTON model to extract local texture features and retain global semantic information;

[0012] S6: Load the model weights and network structure trained in S3, obtain the model clothing image input by the user, the multi-target clothing style images to be tried on and the corresponding text prompt words, and save them in the system;

[0013] S7: During the MM-VTON model inference process, the model clothing image, the multi-target clothing style image and the text prompt words input by the user are used as conditions, and the MM-VTON model is used to gradually estimate the denoised version of the virtual fitting result from the pure noise, and output the virtual fitting result image.

[0014] Optionally, the specific operation steps of generating data sample pairs for the existing model image and the clothing style image in step S1 are:

[0015] S11: extracting corresponding images from existing model images and clothing style image datasets;

[0016] S12: Input the model image into GPT-4V and CogVLM to generate detailed text prompts describing what the model is wearing;

[0017] S13: inputting the generated text prompt words into the open vocabulary detection model to generate key prompt words related to each target clothing subject;

[0018] S14: Input the model image into the SAM model to generate a clothing image bounding box corresponding to the key prompt word to highlight the target clothing body;

[0019] S15: combining the model image, clothing style image, model image with clothing bounding box, and the generated text prompt words and key prompt words, and constructing a data sample pair including the image body, text entity, and target space layout;

[0020] S16: Perform data enhancement on the generated data sample pairs, such as rotation, scaling, contrast and color adjustment, and perform manual review and automated inspection to ensure the quality and consistency of the data sample pairs.

[0021] Optionally, the method based on the multimodal feature interleaved learning attention module (MFILA) described in step S2 is:

[0022] Firstly, the multimodal feature fusion module (MFF) is used to encode the input text prompt words and clothing images, and output the multimodal conditional embedding. Then, the multi-reference target texture feature extraction module (MRSTF) is used to extract the features of specific style from the encoded key prompt words and multi-target reference images, and the features of different styles are spliced ​​in the channel dimension to obtain the texture features of multi-reference targets. Next, a parallel multimodal attention layer is designed to process the self-attention calculation of text and image features to realize the interaction of multimodal features. Finally, the multi-target cross-attention mechanism (MS-CA) is used to calculate the cross-attention, and the attention mask is introduced to limit the context to the specified target area. The multimodal conditional feature encoding after interleaved learning and fusion processing is output for downstream tasks.

[0023] Optionally, the method of the parallel denoising UNet module described in step S2 is:

[0024] The parallel denoising UNet module designed in the present invention retains the network architecture responsible for general feature extraction in the original SDXL model and shares it between different conditional coding branches, and adopts the multimodal attention and multi-target cross-attention methods described in step S2 to replace the original self-attention and cross-attention methods; during the reasoning process, the trained parallel denoising UNet model is used to denoise the input noisy image to generate high-quality virtual fitting results.

[0025] Optionally, the multimodal feature fusion module (MFF) in the multimodal feature interleaved learning attention module (MFILA) described in step S3 is:

[0026] Firstly, the input text prompt words are encoded using a well-designed multimodal prompt method using the pre-trained CLIP text encoder, and the text prompt words are converted into text embeddings for multimodal fusion. Then, the pre-trained image encoder is used to extract clothing image features and map them into the latent feature space to preserve the detailed features. Next, the text embeddings are fused with the image features to form multimodal conditional embedding features, retaining the special placeholder markers in the original text prompts and aligning the keyword embeddings with the relevant target clothing image features. Finally, the multimodal conditional embedding features are downsampled using a perceptual resampler to balance the embedding length and image feature granularity to generate reliable multimodal conditional embeddings for further processing.

[0027] Optionally, the multi-reference target texture feature extraction module (MRSTF) in the multimodal feature interleaved learning attention module (MFILA) described in step S3 is:

[0028] Firstly, the clothing encoder is initialized using the UNet encoder from the diffusion model, and the clothing style image, the model image with clothing bounding box and the key prompt words are taken as input; then, the text encoder CLIP is used to encode the text entity containing the key prompt word and input it into the initialized clothing encoder together with the multi-target clothing images and the model image with clothing bounding box to extract the style features; next, the clothing features of different styles are spliced ​​in the channel dimension to obtain the texture features of multiple reference targets to ensure the consistency of texture; finally, the text query loss L is used to extract the style features of the multi-reference target. query To supervise the clothing encoder separately to improve the accuracy and feature extraction ability of text query.

[0029] Optionally, the total loss function L of the MM-VTON model in step S5 is enc The calculation formula is:

[0030] L enc =L dm +L query

[0031] Where, L dm is the diffusion loss function, L query The query loss function can be calculated by the following formulas:

[0032]

[0033]

[0034] Where c is the text condition associated with the latent image feature z, t is the time step, and z trepresents the encoded image ε(I) with random Gaussian noise ∈~N(0,1) added, D is the training set, ⊙ represents element-by-element multiplication, σ represents the sigmoid activation function. F represents clothing features, M represents the target region prior mask corresponding to the key prompt word, and N represents the number of elements in F and M.

[0035] Optionally, in step S7, the reasoning method of the MM-VTON model is performed according to the model clothing picture input by the user, the multi-target clothing style pictures to be tried on and the corresponding text prompt words:

[0036] S71: Encode the text prompt words through a pre-trained text encoder (such as CLIP) to generate a high-dimensional text feature vector;

[0037] S72: using an image encoder to extract features from the model clothing image and the reference clothing image respectively, to generate latent space features and a high-dimensional image feature vector;

[0038] S73: Based on the MFILA module, the multimodal attention mechanism is used to align and fuse text features and image features to generate multimodal conditional embedding features. The perceptual resampler is used to downsample the fused features to balance the embedding length and image feature granularity.

[0039] S74: Initialize the generated latent space feature into a pure noise feature using the forward process of diffusion as the initial input of the parallel UNet module;

[0040] S75: Use the parallel UNet module to gradually denoise the input noisy features, and estimate the denoised version of the virtual fitting model image from the pure noise as the final result output.

[0041] On the other hand, the present invention also provides a virtual fitting device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned virtual fitting method based on the MM-VTON model when executing the computer program.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] Through the MM-VTON model, an efficient virtual fitting process is realized. First, in the training stage, the model images and clothing style images in the database are used to generate model images with clothing bounding boxes, generated text prompts and key prompts using existing advanced methods, and data sample pairs containing image bodies, text entities, and target spatial layouts are constructed; then, the text prompts are encoded using the pre-trained text encoder, and the image encoder and the initialized clothing encoder are used to extract the image features of the clothing style and the texture features of multiple reference targets, and the text prompts and image features are fused as multimodal conditional embeddings. Subsequently, the multimodal feature interleaved learning attention module (MFILA) uses multimodal attention and multi-target cross-attention to fuse the multimodal conditional embedding and the texture features of multiple reference targets, and outputs the multimodal conditional feature encoding; then, the parallel denoising UNet module is used to gradually generate high-quality denoised images from pure noise based on the multimodal conditional feature encoding information, and the loss is calculated with the model image to improve the model's ability to extract image texture details and retain global semantic information, significantly improving the robustness and image clarity of the virtual fitting system under complex backgrounds and postures of multiple targets. In the inference stage, first, the system loads the trained MM-VTON model weights and network structure, obtains the model clothing image, multi-target clothing style images that need to be tried on, and the corresponding text prompt words input by the user, and inputs them into the MM-VTON model. Through the diffusion mechanism, it generates a virtual fitting effect that better meets the user's expectations, thereby optimizing the user's online shopping experience.

[0044] The present invention significantly improves the realism of clothing and the number of tries on through an innovative virtual fitting method, while optimizing the user experience and reducing inventory and logistics costs. It also improves the robustness of the system and the clarity of the image through efficient prompt word segmentation and parallel diffusion technology, thereby bringing cost-effectiveness and market competitiveness to the clothing industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a logic flow chart of the present invention;

[0046] Figure 2 It is a structural diagram of the MM-VTON training phase model of the present invention;

[0047] Figure 3 It is a structural diagram of the MM-VTON reasoning stage model of the present invention;

[0048] Figure 4 is a flow chart of constructing data sample pairs of the present invention;

[0049] Figure 5 It is a structural diagram of the MRSTF module in the MM-VTON model of the present invention;

[0050] Figure 6 It is a structural diagram of MS-CA in the MFILA module of the present invention;

[0051] Figure 7 This is a virtual fitting effect diagram of the present invention. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0053] like Figure 1-7 As shown, a virtual fitting method provided in an embodiment includes the following steps:

[0054] (1) First, in order to meet the model's requirements for multi-objective and multi-modal features, the traditional method is to construct the dataset in the same format. However, it is difficult to obtain existing fitting datasets in this format. To this end, for a given set of model images and clothing style images I a ,I b , this embodiment starts from the existing image dataset of models and clothing styles, and uses the advanced large-scale multimodal models GPT-4V and CogVLM to describe the model's clothing in the required format; then, the description of the main clothing body and style is extracted as text prompt words, recorded as {obj1, obj2, ..., obj n}; Then, an advanced open source vocabulary detection model is used to highlight the key words of each target clothing body in the text prompt words, and the SAM model is used to generate clothing image bounding boxes corresponding to the key words on the existing model images to extract the text entities of the relevant target clothing and generate accurate mask annotations; Next, the clothing style images, the text prompt words with the highlighted keywords, and the model images with clothing bounding boxes are combined to construct a series of target model images with text descriptions I a With from I b The correspondence between all reference target garments, that is, [image subject, text entity, target space layout] is used as a data set. It is worth noting that the present invention establishes a correspondence between two different postures, otherwise the virtual fitting task will degenerate into a copy and paste problem; finally, the produced data set is divided into a training set, a validation set and a test set according to the ratio of 8:1:1.

[0055] (2) The user manually uploads the model clothing image that needs to be virtually tried on, the multi-target clothing style images that need to be tried on, and the corresponding text prompt words in the system, and the image and text data entered by the user are stored in the system memory for subsequent operations.

[0056] (3) The MM-VTON model (Multi-subject, multi-modal-Virtual Try-on) is established to realize multi-objective personalized virtual fitting. The model is mainly composed of a multi-modal feature interleaved learning attention module (MFILA) and a parallel denoising UNet module, which respectively realize multi-modal conditional feature encoding and diffusion generation of virtual fitting images.

[0057] In the process of training the conditional diffusion probability model (DDPM), traditional methods mainly rely on using the CLIP encoder to convert pixel features into text-aligned features and inject them into the cross-attention layer as conditions. However, this method often leads to a decrease in the model's attention to details. Recent studies have shown that the cross-attention layer tends to learn the correspondence between image features and text embedding tokens to generate the semantic structure of the overall image, while the self-attention layer is more inclined to learn detailed detail features to generate high-quality texture styles. In order to achieve the seamless transfer of detailed texture features of clothing images to the required model images, for this purpose, based on the idea of ​​multimodal learning, the present invention introduces multimodal fusion features into the training process of MM-VTON to enhance the cross-attention layer conditioned on the original text, and also carefully designs and trains a clothing encoder to align image features with diffusion features to synchronously enhance the self-attention layer. As a result, the condition of the input model is not just a text condition, but a multimodal condition, which includes the texture information of multi-target clothing images and the semantic information of text prompt words.

[0058] (4) The multimodal feature interleaved learning attention module (MFILA) for implementing multimodal conditional feature encoding in the MM-VTON model is:

[0059] When dealing with virtual fitting of multiple garments, traditional methods usually need to replicate multiple garment encoding networks to manage different conditions. However, this approach significantly increases the number of parameters, resulting in a waste of computational resources. Recent studies have found that for conditions with similar content (such as different types of clothing), the self-attention layer plays a key role in the potential distortion alignment process of the image, while other network architectures responsible for general feature extraction can be shared between different conditional encoding branches without affecting performance. To this end, this embodiment proposes MFILA for multimodal conditional feature encoding. MFILA mainly includes two parts: a multimodal feature fusion module (MFF) and a multi-reference target texture feature extraction module (MRSTF). MFF maps multimodal data into the latent feature space, and MRSTF uses an innovative clothing encoder module to encode the conditional information of multiple targets. Then, the number of parallelized attention modules is selected according to the number of input conditions to accurately retain the details of the clothing. Specifically, firstly, for text information, the present invention designs a complex multimodal prompt word method, and uses the pre-trained CLIP text encoder and vocabulary detection model to encode the prompt word T input by the user and divide the text entity; for image information, the present invention uses a pre-trained image encoder to extract a set of clothing image features and map the original image to the latent space feature; then, the self-attention matrix with the same initial weight is parallelized, and the multi-condition key and value features are combined into a matrix. After splicing along the l-dimensional, a key-value combination of multiple targets is formed as follows:

[0060]

[0061]

[0062] Where N represents the total number of input conditions. Representation splicing, using attention calculation to encode fine-grained features of clothing.

[0063] Next, the encoded features are input into a multi-target cross-attention (MS-CA) to decouple the conditions of different modalities. Different from the traditional cross-attention method, the present invention introduces an attention mask M in the cross-attention mechanism to limit the context of each object to the prior mask corresponding to the specified target area text, which helps to exclude filler tokens from the conditions and minimize the impact of condition conflicts. The mask mechanism involves bilateral neglect of tokens in the query and key matrices. The calculation method of the attention mask M corresponding to the jth reference clothing is as follows:

[0064]

[0065] In the formula, B jrepresents the set of coordinates belonging to the jth reference clothing bounding box. In this way, the conditional image latent representation z attn It can be exported in the following ways:

[0066]

[0067] Where, Q = zW q , W q ,W k ,W v is the corresponding projection weight matrix, d is the dimension of the key vector, c i Represents a multimodal conditional feature encoding of the input, where the key and value matrices corresponding to different target features are independent of each other.

[0068] Finally, this embodiment implements feature injection between two parallel UNets through a position feature encoding module and cascaded attention. By introducing learnable position embedding to distinguish the features of different target conditions, the superscript i represents different input conditions. Then concatenate the keys and values ​​along the l dimension to obtain the final multi-target key-value combination as follows:

[0069]

[0070]

[0071] Where N represents the total number of input conditions, PE represents the positional encoding, Represents concatenation. Combine it with the key and value features from the second UNet Splicing along the l dimension is and use the query z mq Subsequent attention and multi-target cross attention (MS-CA) calculations are performed. With the innovative design of parallelism and cascading, it can be effortlessly extended to perform feature injection under any number of conditions, thus having wider application potential in the field of virtual fitting.

[0072] (5) The multimodal feature fusion module (MFF) for implementing multimodal conditional embedding extraction in the MFILA module is:

[0073] In the process of virtual fitting, due to the ambiguity of the high-level description of clothing types, it is impossible to require users to accurately describe the desired clothing from the reference image using only text. Therefore, the image generation method that relies solely on text prompts cannot guarantee the fidelity of the clothing. In order to alleviate this problem, the present invention proposes a multimodal prompt method:

[0074] A person wearing<garment> <style>[Ref#1],…,and<garment><style

[0075] >[Ref#N]

[0076] 给定一个句子，"A person wearing a jacket,tucked in,[Ref#1],pants[Ref#2]and shoes[Ref#3].”这里[Ref#N]是为相应的参考服装图像准备的特殊占位符标记，通过利用CLIP文本编码器将该文本提示编码为文本嵌入T∈RL×D，其中L是token的长度，D是嵌入维数；然后，将输入的多目标服装图像利用图像编码器进行服装图像特征的提取；

[0077] 最后，将文本中[Ref#N]占位符token对应的嵌入与相关服装图像的图像特征进行融合为多模态条件嵌入特征。为了平衡文本嵌入长度和图像特征粒度，并更好地将图像特征与文本嵌入对齐，本发明不仅保留了原本的[CLS]标记，还采用了感知重采样器进一步对多模态条件嵌入特征进行下采样。

[0078] 本发明利用这种多模态条件嵌入的方式以替代稳定扩散模型中原始文本嵌入方式，有助于生成可靠的着装风格与增强虚拟试穿的稳定性。

[0079] (6)所述MFILA模块中多参考目标纹理特征提取模块(MRSTF)为：

[0080] 首先，为了充分利用图像信息，本发明在大数据集上预先训练了一个鲁棒的服装编码器。具体地，本发明使用来自扩散模型的UNet编码器初始化服装编码器，该编码器将服装图像和文本查询作为输入，并将每个Transformer块的交叉注意层输出的特征作为服装特征，通过计算扩散损失Ldm进行监督，以保持特征纹理的一致性。同时，该模块可以利用文本作为查询来提取必要的图像特征，以消除对服装分割模型的依赖。为了提高文本查询的准确性，本发明引入文本查询损失，通过抑制注意力掩码区域之外的特征，以鼓励与文本无关区域的激活最小化。文本查询损失Lquery可以表示为：

[0081]

[0082] 式中，⊙表示逐元素乘法，σ表示sigmoid激活函数。F表示服装特征，M表示与目标区域文本对应的先验掩码，N表示F和M中的元素数量。最终的损失可以通过如下公式进行计算：

[0083] Lenc＝Ldm+Lquery

[0084] 式中，Ldm是扩散损失函数，将在接下来的步骤中进行详细说明。通过最小化Lenc损失函数，便可以获得一个高质量的服装编码器，该编码器可以提取与文本查询相对应的必要特征，并且通过判别监督和生成监督的组合来保持低级像素的一致性。

[0085] 然后，利用预训练完成的服装编码器从编码后的文本实体标签Pi与对应的参考图像Gi中提取特定风格的特征，以保证纹理的一致性；最后，将不同风格的特征在通道维度进行拼接，作为多参考目标的纹理特征。

[0086] (7)所述MM-VTON模型中并行去噪UNet模块的方法为：

[0087] MM-VTON模型与传统的稳定扩散模型性质相同，均为条件化的潜在空间概率扩散模型。与传统的稳定扩散模型(DDPM、Stable Diffusion、SDXL)不同的是，本发明将单个去噪UNet修改为两个并联的去噪UNet网络，通过保留其初始化权重相同且共享的性质，以结合MFILA模块的注意力特征注入有利于更稳定的生成高质量的虚拟试衣效果。在训练过程中，首先，利用MFILA来编码多模态的潜在特征z，然后，通过根据预定义的噪声调度器αt来添加噪声以执行前向扩散过程：

[0088]

[0089] 接下来，在逆向的扩散过程，通过并行去噪UNet参数化的噪声估计器∈θ()以学习预测具有目标函数的前向添加的噪声θ，噪声估计器∈θ()在并行去噪UNet中共享。并行去噪UNet的训练目标是获得一个稳定的噪声估计器，其可以最小化预测噪声与添加到真实图像中噪声之间的差异，损失函数可以定义为，

[0090]

[0091] 式中，c是与潜层图像特征z相关联的文本条件，t为时间步长，zt表示添加了随机高斯噪声∈～N(0,1)的编码图像ε(I)，D是训练集。

[0092] 在推理过程中，给定目标模特图像I、试穿服装的多个参考图像以及来自用户输入的关于试穿服装参考图像的文本指令T，MM-VTON旨在无缝地跟随T以合成照片般逼真的试穿结果I′。为此，在图像生成部分，本发明基于开源的稳定扩散文本到图像模型(SDXL)来建模上述任务，该模型将I，和T作为条件，并从纯噪声中逐渐估计I′的去噪版本，再将I′利用解码器进行像素空间的还原，输出虚拟试衣的结果。

[0093] 以上所述，仅为本发明较佳的具体实施方式，但本发明的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本发明揭露的技术范围内，根据本发明的技术方案及其发明构思加以等同替换或改变，都应涵盖在本发明的保护范围之内。< / style> < / garment>

Claims

1. A virtual fitting method, characterized in that: The following steps are involved: S1: Use the large text model GPT-4V and CogVLM to describe the model's clothing in the required format as text prompts, use the open source vocabulary detection model to highlight the key prompts of each target clothing body, use the SAM model to generate clothing image bounding boxes corresponding to the key prompts on the existing model image, and combine the clothing style image, the text prompts that have highlighted the keywords, and the model image containing the clothing bounding box to construct the image body, text entity, and target space layout data sample pairs; S2: The MM-VTON model is established to realize multi-objective personalized virtual fitting. The MM-VTON model consists of a multimodal feature interleaved learning attention module and a parallel denoising UNet module, which respectively realize multimodal conditional feature encoding and diffusion denoising probability generation of virtual fitting images; S3: The data sample is input into the MM-VTON model for multimodal feature encoding. The MFILA module is responsible for processing virtual fitting of multiple garments, and the potential distortions of different types of garments are aligned through the self-attention layer, while other network architectures responsible for general feature extraction are shared between different conditional encoding branches to reduce the number of parameters and computing resource requirements; S4: Implementing feature injection of the multimodal feature encoding in the parallel denoising UNet module through the position feature encoding module to enhance the scalability of the model under multi-objective conditions; S5: Design the total loss function L of the MM-VTON model enc To optimize the parameters of each module of step S2, and further improve the ability of the MM-VTON model to extract local texture features and retain global semantic information; S6: Load the model weights and network structure trained in step S3, obtain the model clothing image input by the user, the multi-target clothing style images to be tried on and the corresponding text prompt words, and save them in the system; S7: During the MM-VTON model inference process, the model clothing image, the multi-target clothing style image and the text prompt words input by the user are used as conditions, and the MM-VTON model is used to gradually estimate the denoised version of the virtual fitting result from the pure noise, and output the virtual fitting result image.

2. A virtual fitting method according to claim 1, characterized in that: The specific operation steps for generating data sample pairs of the existing model image and clothing style image in step S1 are: S11: extracting corresponding images from existing model images and clothing style image datasets; S12: Input the model image into GPT-4V and CogVLM to generate detailed text prompts describing what the model is wearing; S13: Input the generated text prompt words into the open source vocabulary detection model to generate key prompt words related to each target clothing subject; S14: Input the model image into the SAM model to generate a clothing image bounding box corresponding to the key prompt word to highlight the target clothing body; S15: combining the model image, clothing style image, model image with clothing bounding box, and the generated text prompt words and key prompt words, and constructing a data sample pair including the image body, text entity, and target space layout; S16: Rotate, scale, adjust contrast and color of the generated data sample pairs to achieve data enhancement, and perform manual review and automated inspection to ensure the quality and consistency of the data sample pairs.

3. A virtual fitting method according to claim 1, characterized in that: The method for interleaving learning of attention modules based on multimodal features described in step S2 is: Firstly, the input text prompt words and clothing images are encoded using the multimodal feature fusion module, and the multimodal conditional embedding is output; Then, a multi-reference target texture feature extraction module is used to extract the features of a specific style from the encoded key prompt words and multi-target reference images, and the features of different styles are spliced ​​in the channel dimension to obtain the texture features of the multi-reference targets. Next, a parallel multimodal attention layer is designed to handle the self-attention calculation of text and image features to achieve the interaction of multimodal features; Finally, a multi-target cross-attention mechanism is used to calculate the cross-attention, and an attention mask is introduced to limit the context to the specified target area, outputting the multimodal conditional feature encoding after interleaved learning and fusion processing for downstream tasks.

4. A virtual fitting method according to claim 1, characterized in that: The method of the parallel denoising UNet module described in step S2 is: The network architecture responsible for general feature extraction in the original SDXL model is retained and shared between different conditional encoding branches, and the multimodal attention and multi-target cross-attention methods described in step S2 are used to replace the original self-attention and cross-attention methods; During the inference process, the pre-trained parallel denoising UNet model is used to denoise the input noisy image to generate high-quality virtual fitting results.

5. A virtual fitting method according to claim 1, characterized in that: The multimodal feature fusion module in the multimodal feature interleaved learning attention module described in step S3 is: Firstly, the pre-trained CLIP text encoder is used to encode the input text prompt words using a carefully designed multimodal prompt method, and the text prompt words are converted into text embeddings for multimodal fusion. Then, the clothing image features are extracted using a pre-trained image encoder and mapped into the latent feature space to preserve the detailed features; Next, the text embeddings are fused with the image features to form a multimodal conditional embedding feature, preserving the special placeholder markers in the original text prompt and aligning the keyword embeddings with the relevant target clothing image features; Finally, the multimodal conditional embedding features are downsampled using a perceptual resampler to balance the embedding length and image feature granularity to generate reliable multimodal conditional embeddings for further processing.

6. A virtual fitting method according to claim 1, characterized in that: The multi-reference target texture feature extraction module in the multi-modal feature interleaved learning attention module described in step S3 is: First, the clothing encoder is initialized using the UNet encoder from the diffusion model, taking clothing style images, model images with clothing bounding boxes, and key prompt words as input; Then, the text entity containing the key prompt words is encoded using the text encoder CLIP and input into the initialized clothing encoder together with the multi-target clothing images and the model images containing clothing bounding boxes to extract style features; Next, the clothing features of different styles are spliced ​​in the channel dimension to obtain the texture features of multiple reference targets to ensure the consistency of texture; Finally, using the text query loss L query To supervise the clothing encoder separately to improve the accuracy and feature extraction ability of text query.

7. A virtual fitting method according to claim 1, characterized in that: The total loss function L of the MM-VTON model in step S5 enc The calculation formula is: L enc =L dm +L query Where, L dm is the diffusion loss function, L query The query loss function can be calculated by the following formulas: Where c is the text condition associated with the latent image feature z, t is the time step, and z t represents the encoded image ε(I) with random Gaussian noise ∈~N(0,1) added, D is the training set, ⊙ represents element-by-element multiplication, σ represents the sigmoid activation function, F represents clothing features, M represents the target area prior mask corresponding to the key prompt word, and N represents the number of elements in F and M.

8. A virtual fitting method according to claim 1, characterized in that: In step S7, the reasoning method of the MM-VTON model is performed according to the model clothing picture input by the user, the multi-target clothing style pictures to be tried on and the corresponding text prompt words: S71: Encode the text prompt words through the pre-trained text encoder to generate a high-dimensional text feature vector; S72: using an image encoder to extract features from the model clothing image and the reference clothing image respectively, to generate latent space features and a high-dimensional image feature vector; S73: Based on the MFILA module, the multimodal attention mechanism is used to align and fuse text features and image features to generate multimodal conditional embedding features. The perceptual resampler is used to downsample the fused features to balance the embedding length and image feature granularity. S74: Initialize the generated latent space feature into a pure noise feature using the forward process of diffusion as the initial input of the parallel UNet module; S75: Use the parallel UNet module to gradually denoise the input noisy features, and estimate the denoised version of the virtual fitting model image from the pure noise as the final result output.

9. A virtual fitting device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor runs the computer program, the steps of the virtual fitting method based on the MM-VTON model are implemented.

Images