Virtual fitting image generation method and device, equipment and storage medium

Abstract

The invention provides a virtual fitting image generation method and device, equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring a try-on user image of a try-on user, a to-be-try-on target clothing image of the try-on user and text prompt information representing description characteristics of the target clothing image; inputting the try-on user image, the target clothing image and the text prompt information into a try-on image generation network for processing to generate a virtual try-on image; wherein the try-on image generation network is obtained by training based on a user image sample, a clothing image sample and a text prompt information sample; the virtual try-on image represents a virtual image of the try-on user trying on the target garment image. The embodiment of the invention is used for solving the defect of poor authenticity of the generated virtual try-on image in the prior art, the virtual try-on image with high authenticity can be generated through the try-on image generation network, and the generalization is high.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for generating a virtual try-on image. Background Art

[0002] Image-based Virtual Try-On (VTON) is an important computer vision task that aims to demonstrate the visual effects of a given garment worn by an arbitrary person. Generating realistic virtual try-on images is of great interest due to its convenience and ability to provide a personalized shopping experience for e-commerce users. The key challenge in VTON is to adapt the garment to various human poses or gestures while maintaining the integrity of the garment's pattern and texture without distortion. Given a pair of images depicting a target person and a garment, image-based virtual try-on aims to generate the visual effects of the target person wearing the specified garment.

[0003] Existing methods based on generative adversarial networks (GANs) typically first reshape clothing to conform to the human body, then fuse the deformed clothing into the subject image using a generator. While many methods attempt to reduce distortion and mismatch between clothing and the subject, they lack generalization capabilities when handling arbitrary subject images with complex backgrounds or poses. These methods use a separate warping module to reshape clothing to the human body and then fuse the GAN generator with the target subject, resulting in virtual try-on images with poor realism. Summary of the Invention

[0004] The present invention provides a method, device, equipment and storage medium for generating virtual try-on images, which are used to solve the defect of poor realism of virtual try-on images generated in the prior art, and realize that virtual try-on images with high realism and high generalization can be generated through a try-on image generation network.

[0005] In a first aspect, the present invention provides a method for generating a virtual try-on image, comprising the following steps: Acquire a user image of a user trying on clothes, an image of a target garment to be tried on by the user trying on clothes, and textual prompt information representing descriptive features of the target garment image; Inputting the user trying-on image, the target clothing image and the text prompt information into a try-on image generation network for processing to generate a virtual try-on image; The try-on image generation network is trained based on user image samples, clothing image samples and text prompt information samples; the virtual try-on image represents a virtual image of the try-on user trying on the target clothing image.

[0006] Preferably, according to a method for generating a virtual try-on image provided by the present invention, the try-on image generation network comprises at least: a clothing image processing network and a character image processing network; The step of inputting the user trying-on image, the target clothing image, and the text prompt information into a try-on image generation network for processing to generate a virtual try-on image includes: Based on the target clothing image and the text prompt information, processing is performed according to the clothing image processing network to generate low-level clothing image features; Based on the try-on user image, the text prompt information and the low-level clothing image features, processing is performed according to the character image processing network to generate the virtual try-on image.

[0007] Preferably, according to a method for generating a virtual try-on image provided by the present invention, the clothing image processing network comprises at least: a clothing image encoder, a first text encoder, and a clothing self-attention layer; The step of processing the target clothing image and the text prompt information according to the clothing image processing network to generate low-level clothing image features includes: Encoding the target clothing image using the clothing image encoder to obtain clothing coding features; Using a first text encoder to convert the text prompt information to obtain a first text embedding vector; The clothing encoding feature and the first text embedding vector are input into the clothing self-attention layer for processing to generate the low-level clothing image feature.

[0008] Preferably, according to a method for generating a virtual try-on image provided by the present invention, the character image processing network comprises at least: a multimodal conditional encoder, a character self-attention layer; The try-on image generation network further includes: an image prompt adapter; The generating of the virtual try-on image by processing the user try-on image, the text prompt information, and the low-level clothing image features according to the character image processing network includes: Using the multimodal conditional encoder, encoding the user image of the user trying on the clothing to obtain a user image potential value, a clothing mask, a hidden user image potential value, and a user image exposed potential value; Obtaining potential noise of the image of the user trying on the clothing; Inputting the user image potential value, the clothing mask, the hidden user image potential value, the exposed user image potential value, and the potential noise into the character self-attention layer, performing channel alignment processing with the low-level clothing image features input into the character self-attention layer, and generating an initial virtual try-on image; The virtual try-on image is generated based on the high-level clothing image features obtained by the image prompt adapter and the initial virtual try-on image.

[0009] Preferably, according to the method for generating a virtual try-on image provided by the present invention, the character image processing network further comprises at least: a second text encoder, a cross attention layer, and an image generator; The generating of the virtual try-on image based on the initial virtual try-on image and the advanced clothing image features includes: Using the second text encoder to convert the text prompt information to obtain a second text embedding vector; encoding the target clothing image using the image prompt adapter to generate high-level clothing image features; Inputting the high-level clothing image feature, the initial virtual try-on image and the second text embedding vector into the cross attention layer for integration processing to generate a virtual try-on image feature; The image generator is used to perform generation processing on the virtual try-on image features to generate the virtual try-on image.

[0010] Preferably, according to a method for generating a virtual try-on image provided by the present invention, the image prompt adapter comprises at least: a feature projection layer, an image cross attention layer; The step of encoding the target clothing image using the image prompt adapter to generate high-level clothing image features includes: Extracting target clothing features of the target clothing image; The target clothing features are sequentially input into the feature projection layer and the image cross attention layer for processing to determine the high-level clothing image features.

[0011] In a second aspect, the present invention further provides a device for generating a virtual try-on image, comprising: An acquisition module is used to acquire a user image of a user trying on clothes, an image of a target garment to be tried on by the user trying on clothes, and textual prompt information representing descriptive features of the target garment image; A generation module is used to input the try-on user image, the target clothing image and the text prompt information into a try-on image generation network for processing to generate a virtual try-on image; wherein the try-on image generation network is trained based on user image samples, clothing image samples and text prompt information samples; the virtual try-on image represents a virtual image of the try-on user trying on the target clothing image.

[0012] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for generating a virtual try-on image as described above is implemented.

[0013] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for generating a virtual try-on image as described in any one of the above.

[0014] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described methods for generating a virtual try-on image.

[0015] The present invention provides a method, apparatus, device, and storage medium for generating a virtual try-on image. The method comprises obtaining a try-on user image, an image of a target garment to be tried on by the try-on user, and textual prompt information describing the features of the target garment image; inputting the try-on user image, the target garment image, and the textual prompt information into a try-on image generation network for processing to generate a virtual try-on image. The try-on image generation network is trained based on user image samples, garment image samples, and textual prompt information samples; and the virtual try-on image represents a virtual image of the try-on user trying on the target garment image. The method addresses the drawback of the low realism of virtual try-on images generated in the prior art, and enables the try-on image generation network to generate highly realistic virtual try-on images with high generalization capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 This is one of the flow charts of the method for generating a virtual try-on image provided by the present invention.

[0018] Figure 2 This is the second schematic diagram of the method for generating a virtual try-on image provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of the device for generating a virtual try-on image provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0022] In the related art, there are at least the following technical problems: Existing methods based on generative adversarial networks (GANs) typically first reshape clothing to conform to the human body, then fuse the deformed clothing into the subject image using a generator. While many methods attempt to reduce distortion and mismatch between clothing and the subject, they lack generalization capabilities when handling arbitrary subject images with complex backgrounds or poses. These methods use a separate warping module to reshape clothing to the human body and then fuse the GAN generator with the target subject, resulting in virtual try-on images with poor realism.

[0023] Compared with GAN, diffusion models perform better in generating realistic images and show the potential to overcome the above challenges. The diffusion-based VTON method effectively enhances the naturalness of try-on images by leveraging the generative priors of a pre-trained text-to-image (T2I) diffusion model. TryOnDiffusion proposed an architecture based on two parallel UNets. The virtual try-on capability based on the diffusion model was verified by training on a large-scale dataset. Subsequent research converted virtual try-on into a sample-based image restoration problem and fine-tuned the restoration diffusion model on the virtual try-on dataset to generate high-quality virtual try-on images. However, these methods still have shortcomings in preserving clothing details, which limits their application in the real world.

[0024] In some other embodiments, conditional control is added to the diffusion model. Although the text-to-image (T2I) diffusion model can generate high-quality images based on text prompts, it has limited fine-grained control over image synthesis due to the ambiguity of natural language.

[0025] In some embodiments, models based on generative adversarial networks (GANs) attempt to apply example-based inpainting diffusion methods to virtual try-on to improve the naturalness of generated images. However, these methods fail to effectively preserve clothing features. Many GAN-based methods struggle to generate high-quality images and can lead to unexpected distortions of clothing when generalizing to images of diverse human figures. While many methods attempt to mitigate distortions and mismatches between clothing and the human body, their generalization capabilities remain insufficient when processing arbitrary human images with complex backgrounds or poses.

[0026] In other examples, the diffusion-based VTON method has shortcomings in preserving fine details of clothing (such as patterns, textures, shapes, and colors), limiting its real-world application. For example, the TryOnDiffusion model struggles to effectively preserve these details, thus affecting its practical performance.

[0027] In other embodiments, adding conditional controls to the diffusion model can, to some extent, compensate for the lack of fine-grained control of text prompts, but this also introduces additional modules, increasing the overall complexity of the model. However, even with the addition of conditional controls, the model may still have difficulty fully preserving complex details in actual generation, such as fine textures, unique patterns, or subtle shape changes in clothing. In addition, over-reliance on conditional controls may cause the generated images to appear mechanical, lacking natural transitions and rich details, thus affecting visual realism.

[0028] The following combination Figures 1-4 The present invention describes a method, apparatus, device, and storage medium for generating virtual try-on images. These methods address the low fidelity of virtual try-on images generated in existing technologies, enabling the generation of highly realistic virtual try-on images through a try-on image generation network with high generalization capabilities. Based on ControlNet and T2IAdapter, additional modules are introduced to fine-tune spatial information such as edges, depth, and body pose, thereby achieving fine-grained control over the diffusion model generation process. The Image Prompt Adapter (IP Adapter) proposes adjusting the T2I diffusion model using high-level semantic features of reference images, thereby more effectively controlling the image generation process based on text and visual cues.

[0029] Figure 1 This is one of the flow charts of a method for generating a virtual try-on image provided by the present invention, such as Figure 1 As shown, the method may include but is not limited to steps S100 to S200: S100, obtaining a user image of a user trying on clothes, an image of a target garment to be tried on by the user trying on clothes, and textual prompt information representing descriptive features of the target garment image; S200, inputting the try-on user image, the target clothing image and the text prompt information into a try-on image generation network for processing to generate a virtual try-on image; wherein the try-on image generation network is trained based on user image samples, clothing image samples and text prompt information samples; the virtual try-on image represents a virtual image of the try-on user trying on the target clothing image.

[0030] In step S100 of some embodiments, a user image of a user trying on clothes, an image of a target garment to be tried on by the user trying on clothes, and textual prompt information describing features of the target garment image are obtained.

[0031] It is understood that methods for obtaining the user's image during a try-on include real-time capture and local upload. That is, the user's current image can be captured using a device camera (mobile phone / computer), or a frontal or full-body photo uploaded by the user during a try-on can be obtained from a local database.

[0032] The target clothing image can be obtained from a local database or by matching the style selected by the user who is trying on the clothing from the clothing library.

[0033] The purpose of textual hints is to provide semantic descriptions of clothing (such as “red dress”) and to supplement details that cannot be covered by the target clothing image (such as fabric and style).

[0034] In step S200 of some embodiments, the user trying on the garment image, the target garment image, and the text prompt information are input into a try-on image generation network for processing to generate a virtual try-on image; The try-on image generation network is trained based on user image samples, clothing image samples and text prompt information samples; the virtual try-on image represents a virtual image of the try-on user trying on the target clothing image.

[0035] It can be understood that the try-on image generation network at least includes: a clothing image processing network and a person image processing network; The step of inputting the user trying-on image, the target clothing image, and the text prompt information into a try-on image generation network for processing to generate a virtual try-on image includes: Based on the target clothing image and the text prompt information, processing is performed according to the clothing image processing network to generate low-level clothing image features; Based on the try-on user image, the text prompt information and the low-level clothing image features, processing is performed according to the character image processing network to generate the virtual try-on image.

[0036] Furthermore, the fitting image generation network includes at least a clothing image processing network and a person image processing network. The clothing image processing network can be GarmentNet, which is used to generate high-fidelity clothing images, extracting and preserving low-level details (such as texture and pattern) of the clothing, while also encoding high-level semantics (such as style and color).

[0037] The structure of the clothing image processing network includes at least, but is not limited to, a clothing image encoder, a first text encoder, and a clothing self-attention layer. It can also be designed for multi-level feature decoupling based on a UNet. This involves freezing a pre-trained encoder (such as SDXL UNet) to extract multi-scale features, and then combining a cross-attention layer to fuse semantic features from textual cues (e.g., "photo of a short-sleeved round-neck T-shirt") and IP-Adapter (image cues adapter).

[0038] The character image processing network may be TryonNet, which has the function of fitting clothing to character images, processing human body postures, and collecting deformations (such as wrinkles and shadows) to achieve a natural try-on effect.

[0039] The human image processing network can be based on a modified diffusion model, UNet. Its inputs include a latent representation of the person (VAE encoding), a segmentation mask (with original clothing removed), DensePose pose features, clothing features (from GarmentNet), and text embeddings. It also includes spatially adaptive normalization (SPADE), which dynamically adjusts clothing deformation to match the human pose. It also includes a cross-attention mechanism, which fuses person features (such as skin and hair regions) with clothing features, refining details through a self-attention layer.

[0040] Furthermore, in some embodiments of the present invention, the clothing image processing network comprises at least: a clothing image encoder, a first text encoder, and a clothing self-attention layer; The step of processing the target clothing image and the text prompt information according to the clothing image processing network to generate low-level clothing image features includes: Encoding the target clothing image using the clothing image encoder to obtain clothing coding features; Using a first text encoder to convert the text prompt information to obtain a first text embedding vector; The clothing encoding feature and the first text embedding vector are input into the clothing self-attention layer for processing to generate the low-level clothing image feature.

[0041] It can be understood that the input target clothing image is encoded through the clothing image encoder to obtain a feature map, and then the average pooling is used to compress the spatial dimension to obtain the clothing encoding feature.

[0042] You can use the Tokenizer of CLIP's text encoder to convert the text prompt information into an ID sequence, then input it into the CLIP model, and output the first text embedding vector of the default dimension. At this time, you can directly load pre-trained weights, such as fixed CLIP weights, to avoid overfitting.

[0043] The clothing encoding features (2048 dimensions) and text embedding (512 dimensions) are mapped to the same dimension (e.g., 512 dimensions) through a linear layer.

[0044] The clothing encoding features and the first text embedding vector mapped to the same dimension are concatenated and input into a multi-head self-attention layer (number of heads = 8, key / query dimension = 64, value dimension = 64) to generate attention-weighted features. Residual connection is then performed based on the attention-weighted features, and they are normalized by LayerNorm to finally output low-level clothing image features.

[0045] Low-level clothing features refer to the underlying visual information directly related to local pixels in the image, which are usually extracted by shallow neural networks.

[0046] In some embodiments of the present invention, the character image processing network includes at least: a multimodal conditional encoder, a character self-attention layer; The try-on image generation network further includes: an image prompt adapter; The generating of the virtual try-on image by processing the user try-on image, the text prompt information, and the low-level clothing image features according to the character image processing network includes: Using the multimodal conditional encoder, encoding the user image of the user trying on the clothing to obtain a user image potential value, a clothing mask, a hidden user image potential value, and a user image exposed potential value; Obtaining potential noise of the image of the user trying on the clothing; Inputting the user image potential value, the clothing mask, the hidden user image potential value, the exposed user image potential value, and the potential noise into the character self-attention layer, performing channel alignment processing with the low-level clothing image features input into the character self-attention layer, and generating an initial virtual try-on image; The virtual try-on image is generated based on the high-level clothing image features obtained by the image prompt adapter and the initial virtual try-on image.

[0047] It can be understood that the input of the user trying-on image to the multimodal conditional encoder to encode the user trying-on image may be performed in the following specific steps: The latent space is a high-dimensional compressed representation of the original image. Diffusion models (such as Stable Diffusion) rely on variational autoencoders (VAE) to encode the user image to the latent value of the user image, that is, , Generated through semantic segmentation or human body analysis technology, the pre-trained semantic segmentation model (such as HRNet, DeepLab series) is used to perform pixel-level classification on the user's image of the clothing, distinguish the clothing area from other parts of the human body (such as skin and hair), and remove the mask of the clothes on the person image to obtain the clothing mask m. The pre-trained encoder maps the user's image to a low-dimensional space, and combines the conditional generation mechanism of clothing mask to diffusion model to obtain the hidden character image potential value, that is, the hidden user image potential value .

[0048] And through the conditional generation mechanism of human body analysis, mask guidance and diffusion model, the potential value of dense pose exposure of character images, that is, the potential value of user image exposure, is determined. .

[0049] Based on the user’s image, a noise vector (512 dimensions) is sampled from a standard normal distribution as the potential noise.

[0050] In some embodiments, the delay Potential noise With clothing mask m, hidden user image potential value and user image exposure potential value Connect them as the input of UNet.

[0051] Use pre-trained models (such as SCHP and LIP) to segment the user's trying-on image into different regions (skin, hair, clothing, etc.), generate semantic masks (i.e., user image latent value, clothing mask, hidden user image latent value and exposed user image latent value, and potential noise in the user's trying-on image), and extract human key points to guide the spatial deformation of the clothing.

[0052] The person mask and key points are encoded as a spatial conditional map. In the UNet of the diffusion model, person features (such as pose embedding) interact with clothing features through the attention mechanism to dynamically adjust clothing deformation.

[0053] GarmentNet is a specialized module designed for multi-level feature decoupling of clothing images. Its core difference from traditional encoders lies in the explicit separation and targeted processing of low-level and high-level features.

[0054] In some embodiments, the steps of channel alignment are: The user potential value (512 dimensions) and clothing features (768 dimensions) are adjusted to the same channel (e.g., 512 dimensions) through 1×1 convolution.

[0055] The clothing mask (single channel) is upsampled to the same size as the feature map through bilinear interpolation.

[0056] The steps of self-attention fusion are: The user’s exposed potential value, hidden potential value, potential noise and aligned clothing features are concatenated and input into the multi-head self-attention layer (8 heads).

[0057] Self-attention calculates the similarity between user features and clothing features, generates a weight matrix, and weightedly fuses features.

[0058] The steps for generating the initial virtual try-on image are as follows: the fused features are passed through a deconvolution decoder (such as PixelShuffle) to generate the initial virtual try-on image (RGB, 256×256).

[0059] The image hint adapter optimizes details by: Input: an initial virtual try-on image, high-level garment image features obtained based on the image prompt adapter, and a second text embedding vector (with optional refined descriptions such as "slim fit") obtained by converting the text prompt information using the second text encoder.

[0060] In some embodiments of the present invention, the character image processing network further comprises at least: a second text encoder, a cross attention layer, and an image generator; The generating of the virtual try-on image based on the initial virtual try-on image and the advanced clothing image features includes: Using the second text encoder to convert the text prompt information to obtain a second text embedding vector; encoding the target clothing image using the image prompt adapter to generate high-level clothing image features; Inputting the high-level clothing image feature, the initial virtual try-on image and the second text embedding vector into the cross attention layer for integration processing to generate a virtual try-on image feature; The image generator is used to perform generation processing on the virtual try-on image features to generate the virtual try-on image.

[0061] It is understandable that the pre-trained CLIP or BERT model is used to tokenize the text and generate word sequences.

[0062] The word sequence is converted into a second text embedding vector (dimension: 512 dimensions) through a text encoder (such as CLIP's TextEncoder).

[0063] For detailed descriptions (such as "slim fit"), the text embedding weights are dynamically adjusted through the cross-attention layer to enhance the semantic expression of key attributes (such as style and texture).

[0064] The steps of integrating multimodal features in the cross-attention layer include: Input: high-level clothing image features (1024 dimensions), initial virtual try-on image (256×256 RGB), second text embedding vector (512 dimensions).

[0065] Processing steps: Feature alignment and projection: High-level clothing features and text embeddings are mapped to the same dimension (e.g., 512 dimensions) through a linear layer.

[0066] The initial fitting image is extracted with features (512 dimensions) through a CNN encoder (such as VGG16) and concatenated with clothing features and text features.

[0067] Cross attention calculation steps: Use multi-head cross attention (MHA, head number = 8) to calculate the similarity weights of clothing features, text features, and try-on image features.

[0068] formula: Among them, Q (query) is the try-on image feature, and K / V (key / value) is the clothing and text features.

[0069] Residual fusion and normalization: Concatenate the attention output with the original feature residual and normalize it through LayerNorm.

[0070] Output: Virtual try-on image features (512 dimensions, latent representation after multimodal fusion). Furthermore, the step of generating a high-resolution virtual try-on image by the image generator includes: Input: Virtual try-on image features (512 dimensions) Processing steps: Feature upsampling: The potential features are gradually upsampled to the target resolution (such as 512×512) through deconvolution or PixelShuffle.

[0071] Detail enhancement: Use a multi-scale fusion module (such as UNet skip connection) to combine the details of the initial trial image (such as wrinkles, light and shadow).

[0072] Introduce noise conditional generation (such as StyleGAN) to control texture randomness and avoid over-smoothing.

[0073] Adversarial training and perceptual optimization: The discriminator is used to distinguish between generated images and real images and optimize the adversarial loss.

[0074] We use perceptual loss (LPIPS) to align the generated images with the visual distribution of real clothing.

[0075] Output: Final virtual try-on image (512×512 RGB, high resolution and rich in details).

[0076] Introducing diverse text prompts (such as "retro" and "futuristic") to improve the model's adaptability to complex descriptions.

[0077] Through the above steps, the system can efficiently integrate user characteristics, clothing images and text semantics to generate virtual try-on images with realistic details and controllable styles, meeting the real-time and user experience requirements of e-commerce scenarios.

[0078] In some embodiments of the present invention, the image cueing adapter includes at least: a feature projection layer, an image cross attention layer; The step of encoding the target clothing image using the image prompt adapter to generate high-level clothing image features includes: Extracting target clothing features of the target clothing image; The target clothing features are sequentially input into the feature projection layer and the image cross attention layer for processing to determine the high-level clothing image features.

[0079] As can be understood, to adjust the high-level semantics of clothing images, the present invention utilizes an image prompt adapter (IP adapter). To encode clothing images, a frozen CLIP image encoder is used to extract features and fine-tune the feature projection layer and cross-attention layer. These layers are initialized with a pre-trained IP adapter. The feature projection layer and cross-attention layer are components of the image prompt adapter (IP-Adapter).

[0080] The IP-Adapter is used to encode high-level semantic information of clothing images and inject semantic features into the generator of the diffusion model through these layers. The parameters of the feature projection layer and the cross-attention layer are initialized using the IP adapter weights pre-trained on a large image dataset.

[0081] The feature projection layer, through pre-training, learns general visual feature extraction capabilities and quickly adapts to clothing semantic encoding tasks. Ultimately, it acquires the ability to map clothing images into a high-dimensional feature space, providing a foundation for subsequent cross-attention calculations.

[0082] The cross-attention layer uses the attention mechanism learned in pre-training to initialize the feature interaction logic, effectively integrating clothing features with body features. It has the ability to accurately capture the association between clothing and body features in feature space, enabling the generator to produce clothing fitting effects that match human posture.

[0083] The clothing text prompt is converted into a feature vector through the Text Encoder and then fed into the Cross-Attention Layer of the diffusion model. This process enables the generator to combine the semantic information of the text and generate clothing features (such as color, style, pattern, etc.) that are consistent with the prompt content. Through the above process, the clothing text prompt is transmitted. The cross-attention calculation formula is as follows: is the query matrix from the UNet intermediate representation, , is the key and value matrix from the text embedding c, where is the number of samples. The output of the cross attention layer is given by Given, then, the IP adapter computes the key and value matrices based on the image embedding i , .

[0084] The IP adapter freezes the original UNet and fine-tunes image embeddings in stages. and A (linear) projection layer for the key-value matrix of , and a linear projection layer for mapping the CLIP image embedding.

[0085] The CLIP encoder parameters are fixed (frozen), and only the feature projection layer and criss-cross attention layer parameters are fine-tuned (optimized in stages after initialization).

[0086] In the first stage, only IP-Adapter and GarmentNet are fine-tuned to adapt them to the task of clothing semantic feature extraction.

[0087] In the second stage, the cross-attention layer and the self-attention layer are jointly fine-tuned to enhance the fusion ability of clothing features and human posture.

[0088] By adopting a structured and goal-oriented fine-tuning strategy, efficient customization of the diffusion model is achieved through freezing pre-trained modules, staged fine-tuning, and data-driven optimization.

[0089] This method not only retains the generalization ability of the pre-trained model, but also significantly improves the retention and authenticity of clothing details in virtual try-on tasks.

[0090] Figure 2This is the second schematic diagram of the method for generating virtual try-on images provided by the present invention. First, a text prompt is obtained. In order to fully utilize the rich generative priors of the text-to-image diffusion model, the present invention provides a comprehensive title to describe the details of the clothing, such as "short-sleeved round neck T-shirt", and passes it to GarmentNet (garment image processing network) (i.e., "photo of short-sleeved round neck T-shirt") and TryonNet (person image processing network) (i.e., "model wearing short-sleeved round neck T-shirt"). The text prompt is converted into an embedding vector through the CLIP text encoder and input into the diffusion model as a condition. This helps the model encode the high-level semantics of the clothing using natural language and supplements the image-based conditions.

[0091] GarmentNet is responsible for generating or editing images of target garments. Text embeddings are injected into the UNet structure of the diffusion model through a cross-attention layer. The model generates high-fidelity garment images based on the text while maintaining coherence in areas unrelated to the human body (such as the background).

[0092] TryonNet is responsible for fitting the generated clothing (from GarmentNet) to the target person image. Text embeddings are combined with human pose and clothing features to feed into the diffusion model. Spatial Adaptive Normalization (SPADE) or an attention mechanism is used to adjust the clothing's geometric deformations (such as wrinkles and shadows) to ensure a natural fit.

[0093] Input person and clothing images. Image prompt adapter (IP adapter) for clothing images Its core design combines multi-scale feature extraction and cross-modal alignment.

[0094] The multi-scale feature encoder is an encoder based on CNN or Vision Transformer (ViT) to extract low / high-level features.

[0095] The cross-modal fusion module mainly consists of a cross-attention layer and a SPADE layer.

[0096] GarmentNet Encode low-level features.

[0097] Will delay Potential noise With clothing mask m, hidden user image potential value and user image exposure potential value Connect them as the input of UNet.

[0098] Use pre-trained models (such as SCHP, LIP) to segment human images into different regions (skin, hair, clothing, etc.), generate semantic masks, and extract human key points to guide the spatial deformation of clothing.

[0099] Person masks and key points are encoded as spatial conditional maps. In the UNet of the diffusion model, person features (such as pose embeddings) interact with high-level clothing features through an attention mechanism to dynamically adjust clothing deformation.

[0100] GarmentNet is a specialized module designed for multi-level feature decoupling of clothing images. Its core difference from traditional encoders lies in the explicit separation and targeted processing of low-level and high-level features.

[0101] Among them, low-level clothing features refer to the underlying visual information directly related to local pixels in the image, which are usually extracted by shallow neural networks. High-level semantic features reflect the abstract representation of the overall semantics of the image and are usually captured by deep networks or attention mechanisms.

[0102] First, the target clothing image is encoded using the clothing image encoder to obtain clothing encoding features; the text prompt information is converted using a first text encoder to obtain a first text embedding vector; the clothing encoding features and the first text embedding vector are input into the clothing self-attention layer for processing to generate the low-level clothing image features.

[0103] Then, the text prompt information is converted by using the second text encoder to obtain a second text embedding vector; the target clothing image is encoded by using the image prompt adapter to generate high-level clothing image features; The high-level clothing image features, the initial virtual try-on image and the second text embedding vector are input into the cross attention layer for integration processing to generate virtual try-on image features.

[0104] The image generator is used to perform generation processing on the virtual try-on image features to generate the virtual try-on image.

[0105] The present invention uses UNet and attention mechanism to extract and fuse the features of target clothing image and user trying-on image, and then gradually denoises them through conditional diffusion model to finally generate images of people wearing clothing.

[0106] The model designed by the present invention can capture the details of clothing, but when the character image or clothing images It often encounters difficulties when the training distribution is different. Inspired by the text-to-image personalization method, the present invention can fine-tune TryonNet with a pair of clothing and person images.

[0107] The model designed in this paper combines a visual encoder and a parallel Unified Network (UNet) to encode high-level semantics and low-level features of clothing images, respectively, and effectively integrates these features into the base UNet. To adapt to real-world virtual try-on scenarios, the model is customized by fine-tuning the UNet decoder layer when inputting a pair of clothing and person images.

[0108] This invention also supports the use of detailed natural language descriptions of clothing features, further enhancing the realism of generated virtual try-on images. Experimental results demonstrate that the model performs well on multiple datasets, including scenes with complex backgrounds and a variety of human poses. It can significantly preserve clothing details and generate high-fidelity images, significantly outperforming existing technologies.

[0109] The model designed by this invention accurately preserves the original pattern, texture, and color of clothing during the fitting process, effectively avoiding the distortion and distortion of details common in traditional methods. Furthermore, the model can handle various human poses and gestures, allowing clothing to fit naturally to the human body, thereby generating more realistic and believable virtual fitting images.

[0110] The present invention provides a method, apparatus, device, and storage medium for generating a virtual try-on image. The method comprises obtaining a try-on user image, an image of a target garment to be tried on by the try-on user, and textual prompt information describing the features of the target garment image; inputting the try-on user image, the target garment image, and the textual prompt information into a try-on image generation network for processing to generate a virtual try-on image. The try-on image generation network is trained based on user image samples, garment image samples, and textual prompt information samples; and the virtual try-on image represents a virtual image of the try-on user trying on the target garment image. The method addresses the drawback of the low realism of virtual try-on images generated in the prior art, and enables the try-on image generation network to generate highly realistic virtual try-on images with high generalization capabilities.

[0111] The following describes a device for generating a virtual try-on image provided by the present invention. The device for generating a virtual try-on image described below and the method for generating a virtual try-on image described above can refer to each other.

[0112] like Figure 3 FIG. 1 is a schematic diagram of the structure of a device for generating a virtual try-on image provided by the present invention. The device for generating a virtual try-on image includes the following modules: An acquisition module 310 is configured to acquire a user image of a user trying on clothes, an image of a target garment to be tried on by the user trying on clothes, and textual prompt information describing features of the target garment image; The generation module 320 is used to input the try-on user image, the target clothing image and the text prompt information into the try-on image generation network for processing to generate a virtual try-on image; wherein the try-on image generation network is trained based on user image samples, clothing image samples and text prompt information samples; the virtual try-on image represents a virtual image of the try-on user trying on the target clothing image.

[0113] Preferably, the device for generating a virtual try-on image provided by the present invention is further configured so that the try-on image generation network at least comprises: a clothing image processing network and a person image processing network; Based on the target clothing image and the text prompt information, processing is performed according to the clothing image processing network to generate low-level clothing image features; Based on the try-on user image, the text prompt information and the low-level clothing image features, processing is performed according to the character image processing network to generate the virtual try-on image.

[0114] Preferably, the device for generating a virtual try-on image provided by the present invention is further used in that the clothing image processing network at least comprises: a clothing image encoder, a first text encoder, and a clothing self-attention layer; Encoding the target clothing image using the clothing image encoder to obtain clothing coding features; Using a first text encoder to convert the text prompt information to obtain a first text embedding vector; The clothing encoding feature and the first text embedding vector are input into the clothing self-attention layer for processing to generate the low-level clothing image feature.

[0115] Preferably, the device for generating a virtual try-on image provided by the present invention is further used in the character image processing network comprising at least: a multimodal conditional encoder, a character self-attention layer; The try-on image generation network further includes: an image prompt adapter; Using the multimodal conditional encoder, encoding the user image of the user trying on the clothing to obtain a user image potential value, a clothing mask, a hidden user image potential value, and a user image exposed potential value; Obtaining potential noise of the image of the user trying on the clothing; Inputting the user image potential value, the clothing mask, the hidden user image potential value, the exposed user image potential value, and the potential noise into the character self-attention layer, performing channel alignment processing with the low-level clothing image features input into the character self-attention layer, and generating an initial virtual try-on image; The virtual try-on image is generated based on the high-level clothing image features obtained by the image prompt adapter and the initial virtual try-on image.

[0116] Preferably, the device for generating a virtual try-on image provided by the present invention is further used in the character image processing network, which at least further comprises: a second text encoder, a cross attention layer, and an image generator; Using the second text encoder to convert the text prompt information to obtain a second text embedding vector; encoding the target clothing image using the image prompt adapter to generate high-level clothing image features; Inputting the high-level clothing image feature, the initial virtual try-on image and the second text embedding vector into the cross attention layer for integration processing to generate a virtual try-on image feature; The image generator is used to perform generation processing on the virtual try-on image features to generate the virtual try-on image.

[0117] Preferably, the device for generating a virtual try-on image provided by the present invention is further used in the image prompt adapter, which comprises at least: a feature projection layer, an image cross attention layer; Extracting target clothing features of the target clothing image; The target clothing features are sequentially input into the feature projection layer and the image cross attention layer for processing to determine the high-level clothing image features.

[0118] The present invention provides a method, apparatus, device, and storage medium for generating a virtual try-on image. The method comprises obtaining a try-on user image, an image of a target garment to be tried on by the try-on user, and textual prompt information describing the features of the target garment image; inputting the try-on user image, the target garment image, and the textual prompt information into a try-on image generation network for processing to generate a virtual try-on image. The try-on image generation network is trained based on user image samples, garment image samples, and textual prompt information samples; and the virtual try-on image represents a virtual image of the try-on user trying on the target garment image. The method addresses the drawback of the low realism of virtual try-on images generated in the prior art, and enables the try-on image generation network to generate highly realistic virtual try-on images with high generalization capabilities.

[0119] Figure 4 An example of a physical structure diagram of an electronic device is shown below. Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 may call logic instructions in the memory 430 to execute a method for generating a virtual try-on image, which includes: obtaining a user image of a try-on user, an image of a target garment to be tried on by the try-on user, and textual prompt information describing features of the target garment image; inputting the user image, the target garment image, and the textual prompt information into a try-on image generation network for processing to generate a virtual try-on image; wherein the try-on image generation network is trained based on user image samples, garment image samples, and textual prompt information samples; and the virtual try-on image represents a virtual image of the try-on user trying on the target garment image.

[0120] Furthermore, the logic instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0121] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the virtual try-on image generation method provided by the above methods, the method including: obtaining a try-on user image of the try-on user, an image of the target clothing to be tried on by the try-on user, and text prompt information representing the descriptive features of the target clothing image; inputting the try-on user image, the target clothing image, and the text prompt information into a try-on image generation network for processing to generate a virtual try-on image; wherein, the try-on image generation network is trained based on user image samples, clothing image samples, and text prompt information samples; the virtual try-on image represents a virtual image of the try-on user trying on the target clothing image.

[0122] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the method for generating a virtual try-on image provided by the above-mentioned methods, the method comprising: obtaining a try-on user image of a try-on user, an image of a target garment to be tried on by the try-on user, and text prompt information representing descriptive features of the target garment image; inputting the try-on user image, the target garment image, and the text prompt information into a try-on image generation network for processing to generate a virtual try-on image; wherein, the try-on image generation network is trained based on user image samples, garment image samples, and text prompt information samples; the virtual try-on image represents a virtual image of the try-on user trying on the target garment image.

[0123] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0124] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for generating a virtual try-on image, characterized in that: include: Acquire a user image of a user trying on clothes, an image of a target garment to be tried on by the user trying on clothes, and textual prompt information representing descriptive features of the target garment image; Inputting the user trying-on image, the target clothing image and the text prompt information into a try-on image generation network for processing to generate a virtual try-on image; The try-on image generation network is trained based on user image samples, clothing image samples and text prompt information samples; the virtual try-on image represents a virtual image of the try-on user trying on the target clothing image.

2. The method for generating a virtual try-on image according to claim 1, wherein: The try-on image generation network at least includes: a clothing image processing network and a person image processing network; The step of inputting the user trying-on image, the target clothing image, and the text prompt information into a try-on image generation network for processing to generate a virtual try-on image includes: Based on the target clothing image and the text prompt information, processing is performed according to the clothing image processing network to generate low-level clothing image features; Based on the try-on user image, the text prompt information and the low-level clothing image features, processing is performed according to the character image processing network to generate the virtual try-on image.

3. The method for generating a virtual try-on image according to claim 2, wherein: The clothing image processing network includes at least: a clothing image encoder, a first text encoder, and a clothing self-attention layer; The step of processing the target clothing image and the text prompt information according to the clothing image processing network to generate low-level clothing image features includes: Encoding the target clothing image using the clothing image encoder to obtain clothing coding features; Using a first text encoder to convert the text prompt information to obtain a first text embedding vector; The clothing encoding feature and the first text embedding vector are input into the clothing self-attention layer for processing to generate the low-level clothing image feature.

4. The method for generating a virtual try-on image according to claim 2, wherein: The character image processing network at least includes: a multimodal conditional encoder and a character self-attention layer; The try-on image generation network further includes: an image prompt adapter; The generating of the virtual try-on image by processing the user try-on image, the text prompt information, and the low-level clothing image features according to the character image processing network includes: Using the multimodal conditional encoder, encoding the user image of the user trying on the clothing to obtain a user image potential value, a clothing mask, a hidden user image potential value, and a user image exposed potential value; Obtaining potential noise of the image of the user trying on the clothing; Inputting the user image potential value, the clothing mask, the hidden user image potential value, the exposed user image potential value, and the potential noise into the character self-attention layer, performing channel alignment processing with the low-level clothing image features input into the character self-attention layer, and generating an initial virtual try-on image; The virtual try-on image is generated based on the high-level clothing image features obtained by the image prompt adapter and the initial virtual try-on image.

5. The method for generating a virtual try-on image according to claim 4, wherein: The character image processing network also includes at least: a second text encoder, a cross attention layer, and an image generator; The generating of the virtual try-on image based on the initial virtual try-on image and the advanced clothing image features includes: Using the second text encoder to convert the text prompt information to obtain a second text embedding vector; encoding the target clothing image using the image prompt adapter to generate high-level clothing image features; Inputting the high-level clothing image feature, the initial virtual try-on image and the second text embedding vector into the cross attention layer for integration processing to generate a virtual try-on image feature; The image generator is used to perform generation processing on the virtual try-on image features to generate the virtual try-on image.

6. The method for generating a virtual try-on image according to claim 5, wherein: The image prompt adapter includes at least: a feature projection layer, an image cross attention layer; The step of encoding the target clothing image using the image prompt adapter to generate high-level clothing image features includes: Extracting target clothing features of the target clothing image; The target clothing features are sequentially input into the feature projection layer and the image cross attention layer for processing to determine the high-level clothing image features.

7. A device for generating a virtual try-on image, characterized in that: include: An acquisition module is used to acquire a user image of a user trying on clothes, an image of a target garment to be tried on by the user trying on clothes, and textual prompt information representing descriptive features of the target garment image; A generation module is used to input the try-on user image, the target clothing image and the text prompt information into a try-on image generation network for processing to generate a virtual try-on image; wherein the try-on image generation network is trained based on user image samples, clothing image samples and text prompt information samples; the virtual try-on image represents a virtual image of the try-on user trying on the target clothing image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the method for generating a virtual try-on image according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating a virtual try-on image according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating a virtual try-on image according to any one of claims 1 to 6 is implemented.

Images