Virtual fitting method, model training method, electronic device, and storage medium

Abstract

Embodiments of the present application provide a virtual fitting method, a model training method, an electronic device and a storage medium, an image of clothes to be fitted and a first image are obtained; wherein a first character wears a first clothes in the first image; the first clothes in the first image are separated to obtain a binary mask image of the first clothes and a second image in which the first clothes are blocked; posture extraction is performed based on the first image and a character posture analysis model to obtain posture feature information of the first character; the clothes image is input into a clothes detail analysis model and a clothes appearance analysis model respectively to obtain detail feature information and appearance feature information of the clothes; the posture feature information after adding hidden noise is taken as a condition input side of a pre-trained diffusion model, and the binary mask image, the second image, the detail feature information and the appearance feature information are taken as a main input side of the pre-trained diffusion model to obtain a target image of the character wearing the clothes to be fitted. The present application improves the virtual fitting effect.

Description

Technical Field

[0001] This application relates to the field of visual processing technology, and in particular to a virtual try-on method, a model training method, an electronic device, and a storage medium. Background Technology

[0002] In the apparel e-commerce industry, the visual presentation of products is crucial for sales. While traditional flat lay images of clothing are sufficient, images showcased by models are generally more effective at capturing customer attention. However, producing a large number of model-showcase images is both time-consuming and labor-intensive. To address this issue, virtual try-on technology has emerged, which not only enhances the consumer shopping experience but also effectively reduces operating costs for apparel retailers.

[0003] In related technologies, an image of a given person trying on clothing is generated based on a given image of a person and an image of clothing. Since clothing is not a rigid body, it deforms when worn. Therefore, related virtual try-on methods often struggle to handle complex clothing textures or complex human postures, resulting in less realistic images.

[0004] Therefore, how to generate high-quality virtual try-on effects has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of this application provide a virtual try-on method, a model training method, an electronic device, and a storage medium to at least partially solve the above-mentioned problems.

[0006] According to a first aspect of the embodiments of this application, a virtual try-on method is provided, including:

[0007] The process involves acquiring an image of the clothing to be tried on and a first image; wherein a first person is wearing the first clothing in the first image; separating the first clothing in the first image to obtain a binary mask image of the first clothing and a second image of the first clothing after it is occluded; extracting pose based on the first image and a person pose analysis model to obtain the pose feature information of the first person; inputting the image of the clothing to be tried on into a clothing detail analysis model and a clothing appearance analysis model to obtain corresponding detail feature information and appearance feature information; using the pose feature information with added latent noise as the conditional input side of a pre-trained diffusion model, and using the binary mask image, the second image, the detail feature information, and the appearance feature information as the main input side of the pre-trained diffusion model to obtain a target image of the first person wearing the clothing to be tried on.

[0008] According to a second aspect of the embodiments of this application, a model training method is provided, comprising:

[0009] Acquire a clothing sample image and a first image; wherein, in the first image, a first person is wearing the clothing sample;

[0010] The clothing sample in the first image is masked to obtain a binary mask image of the clothing sample and a second image after the clothing sample is masked.

[0011] Based on the first image and the person's pose analysis model, pose extraction is performed to obtain the pose feature information of the first person;

[0012] The clothing sample images are input into the clothing detail analysis model and the clothing appearance analysis model to be trained, respectively, to obtain the clothing detail feature information and the clothing appearance feature information.

[0013] The pose feature information with added latent noise is used as the conditional input of the diffusion model to be trained, and the binary mask image, the second image, the detail feature information, and the appearance feature information are used as the main input of the diffusion model to be trained to determine the noise loss and pixel-level loss; the model is trained using the noise loss and the pixel-level loss until a trained model is obtained.

[0014] According to a third aspect of the embodiments of this application, an electronic device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the first aspect or the second aspect.

[0015] According to a fourth aspect of the embodiments of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method as described in the first aspect or the second aspect.

[0016] According to the solution provided in the embodiments of this application, an image of the clothing to be tried on and a first image are obtained; wherein, in the first image, a first person is wearing the first clothing; the first clothing in the first image is separated to obtain a binary mask image of the first clothing and a second image after the first clothing is occluded; based on the first image and the person's pose analysis model, pose extraction is performed to obtain the pose feature information of the first person; the image of the clothing to be tried on is input into the clothing detail analysis model and the clothing appearance analysis model respectively to obtain the corresponding detail feature information and appearance feature information; the pose feature information after adding latent noise is used as the conditional input side of the pre-trained diffusion model, and the binary mask image, the second image, the detail feature information and the appearance feature information are used as the main input side of the pre-trained diffusion model to obtain the target image of the first person wearing the clothing to be tried on. In this process, acquiring detailed and appearance feature information provides clothing references for the target image. In the second image, specific areas where the first garment is separated are redrawn using the detailed and appearance feature information of the garment to be tried on, along with a binary mask image. This allows the newly drawn area of ​​the garment to be tried on to blend harmoniously with the rest of the original second image, achieving visual unity and a natural feel. Simultaneously, pose feature information is input as a conditional constraint into the diffusion model, enabling the final generated target image to visualize the effect of the garment being tried on the first person while maintaining the overall realism and naturalness of the image. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0018] Figure 1 This application provides an overall architecture diagram of a virtual try-on method.

[0019] Figure 2 A flowchart illustrating a virtual try-on method provided in an embodiment of this application;

[0020] Figure 3 A schematic diagram of a model training process provided in an embodiment of this application;

[0021] Figure 4 A general flowchart of a virtual try-on method provided in an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0024] To adapt to the virtual try-on method provided in the embodiments of this application, the embodiments of this application provide a general architecture diagram of the virtual try-on method, that is, the virtual try-on method of the embodiments of this application can be applied to the virtual try-on system provided in the embodiments of this application. For example... Figure 1 As shown, the virtual try-on system includes an image acquisition module, an image processing module, and a target image generation module. The image acquisition module acquires an image of the clothing to be tried on and a first image. In the first image, a first person (e.g., a model) is wearing the first clothing. During the training phase, the acquired image of the clothing to be tried on is the same as the first person wearing the first clothing; during the inference phase, the acquired image of the clothing to be tried on and the first person wearing the first clothing can be different. The image processing module processes the image of the clothing to be tried on and the first image. Specifically, it uses open-source semantic segmentation techniques, such as the Segment Anything Model (SAM), to separate the first clothing from the first image, thereby obtaining a binary mask image of the first clothing and a second image (i.e., a background image redrawn) after occluding the first clothing. Finally, it uses an open-source human pose extraction model to extract the pose image of the first person from the first image. The target image generation module specifically includes: extracting detailed feature information and appearance feature information from the image of the clothing to be tried on using the clothing detail analysis model and clothing appearance analysis model in the clothing reference model (e.g., contrastive language-image pre-trained image encoder); processing the pose image using an open-source human pose analysis model to obtain pose feature information; and inputting the pose feature information, the extracted detailed feature information, the appearance feature information, the binary mask image, and the second image into the diffusion model for processing to obtain the target image.

[0025] The following is combined Figure 1 and Figure 2 This application describes a virtual try-on method provided by an embodiment. For example... Figure 2 As shown, Figure 2 This is a flowchart illustrating the virtual try-on method provided in the embodiments of this application. The virtual try-on method provided in the embodiments of this application can be executed by an electronic device, such as a computer or a server.

[0026] like Figure 2 As shown, the virtual try-on method includes:

[0027] S101. Obtain an image of the clothing to be tried on and a first image; wherein, in the first image, a first person is wearing the first clothing.

[0028] In the embodiments of this application, the clothing to be tried on in the image of the clothing to be tried on can be clothing that needs to be displayed or sold to users. The first image is an image of a first person wearing the first clothing. The first person can be a real person model displaying the clothing, a simulated person model displaying the clothing, a digital person model displaying the clothing, or a customer purchasing the clothing, etc. The first image can be one or multiple images. When there are multiple first images, the first person in each first image can be different. The same clothing to be tried on can be displayed by multiple first persons with different body shapes, so that users can make a suitable choice according to their own body shape.

[0029] In embodiments of this application, the first garment may be different from the garment to be tried on.

[0030] S102. Separate the first garment in the first image to obtain a binary mask image of the first garment and a second image after the first garment is occluded.

[0031] In embodiments of this application, a binary mask image is used to accurately separate the outline of clothing from the background, allowing a person to "try on" different garments in a digital environment without actually wearing them. The second image is a background image for redrawing the clothing image; that is, the clothing to be tried on can be drawn in the area where the first garment is separated in the second image.

[0032] By using open-source semantic segmentation techniques, such as SAM, the first clothing on the first person in the first image can be separated to obtain a binary mask image of the first clothing and a second image after the first clothing is occluded.

[0033] S103. Based on the first image and the person's posture analysis model, perform posture extraction to obtain the posture feature information of the first person.

[0034] In the first image, after the original first garment of the first person is peeled off, the process of redrawing the garment to be tried on on the area where the first garment was peeled off needs to ensure that the face, hairstyle and body movements of the first person remain unchanged. In this way, the final target image can visualize the effect of the garment to be tried on the first person, while maintaining the realism and naturalness of the overall image. Therefore, it involves the acquisition of the person's posture information.

[0035] Postural features involve the position, orientation, and movement of various body parts, and how these factors combine to express an individual's emotions, intentions, and personality. This includes, but is not limited to, head posture features, trunk features, facial expression features, and micro-expression and micro-movement features.

[0036] In the embodiments of this application, a first image can be input into a person pose extraction model obtained from open source to obtain a pose image. The pose image can then be input into a pose analysis model (e.g., a pose encoder) to obtain the pose feature information of the first person. Subsequently, to enhance data diversity and model robustness, initial latent space noise can be determined based on a defined noise type and magnitude, or initial latent space noise can be randomly generated. This initial latent space noise is then added to the extracted pose feature information, such as adding defined or randomly generated latent space Gaussian noise to the extracted pose feature information to obtain pose feature information with added latent noise. This can then be input into a pre-trained diffusion model.

[0037] S104. Input the images of the clothing to be tried on into the clothing detail analysis model and the clothing appearance analysis model respectively to obtain the corresponding detail feature information and appearance feature information.

[0038] The details of clothing refer to the subtle yet crucial elements that constitute the design, production, and wearing experience of garments. These details not only concern the aesthetic appearance of the clothing but also directly affect its comfort, durability, and overall quality. These details include, but are not limited to, fabric selection and texture, cut and design, decorative elements, functional details, and linings and accessories. The appearance information of clothing refers to all the elements and characteristics that the garment visually presents, including but not limited to color and pattern, style and silhouette, the luster and texture of the material, and the overall effect of the combination.

[0039] In the embodiments of this application, the image of the clothing to be tried on is input into the clothing detail analysis model to obtain detail feature information, and the image of the clothing to be tried on is input into the clothing appearance analysis model to obtain appearance feature information. The appearance feature information and detail feature information can provide a reference for the generation of subsequent target images.

[0040] The clothing appearance analysis model can be a Contrastive Language-Image Pre-training (CLIP) model, while the clothing detail analysis model can be trained based on the StableDiffusion model.

[0041] Stable Diffusion (LDM) is a classic latent diffusion model. LDM employs a variational autoencoder. Given an input image, the encoder maps the image to the latent space, allowing diffusion to occur within this space. During the forward diffusion process, Gaussian noise is gradually added to the latent variables over T iterations. The backward diffusion process aims to progressively recover the image from the previous state, and finally, the image is decoded to obtain the final image.

[0042] S105. The pose feature information after adding latent noise is used as the conditional input side of the pre-trained diffusion model, and the binary mask image, the second image, the detail feature information and the appearance feature information are used as the main input side of the pre-trained diffusion model to obtain the target image of the first person wearing the clothes to be tried on.

[0043] Pre-trained diffusion models allow models to generate data based on additional input information (conditions), which can be category labels, text descriptions, another image, or any form of guiding signal. The purpose is to guide the generation process so that the generated results conform to specific conditions or attributes.

[0044] In this embodiment, the pose feature information after adding latent noise is used as the conditional input of the pre-trained diffusion model, the binary mask image, the second image, the detail feature information and the appearance feature information are used as the main input of the pre-trained diffusion model, and the pose information of the first person is used as an additional input condition to guide the pre-trained diffusion model, so as to obtain the target image of the first person wearing the clothes to be tried on.

[0045] It is understood that in the embodiments of this application, an image of the clothing to be tried on and a first image are acquired; wherein, in the first image, a first person is wearing the first clothing; the first clothing in the first image is separated to obtain a binary mask image of the first clothing and a second image after the first clothing is occluded; pose extraction is performed based on the first image and a person pose analysis model to obtain the pose feature information of the first person; the image of the clothing to be tried on is input into a clothing detail analysis model and a clothing appearance analysis model respectively to obtain corresponding detail feature information and appearance feature information; the pose feature information after adding latent noise is used as the conditional input side of a pre-trained diffusion model, and the binary mask image, the second image, the detail feature information, and the appearance feature information are used as the main input side of the pre-trained diffusion model to obtain a target image of the first person wearing the clothing to be tried on. In this process, detail feature information and appearance feature information can be obtained in a specific area where the first clothing is separated in the second image to provide clothing reference for the target image. By combining the detailed and appearance features of the clothing to be tried on with a binary mask image, a new image of the clothing is drawn in a specific region. This allows the newly drawn image to blend harmoniously with the rest of the original second image, achieving visual unity and a natural feel. Simultaneously, pose feature information is input as a constraint into the diffusion model, ensuring that the final generated target image visualizes the effect of the clothing on a person while maintaining the overall realism and naturalness of the image.

[0046] In some embodiments of this application, the pre-trained diffusion model includes a self-attention layer and a cross-attention layer. In S105, the pose feature information after adding latent noise is used as the conditional input side of the pre-trained diffusion model, and the binary mask image, the second image, the detail feature information and the appearance feature information are used as the main input side of the pre-trained diffusion model. The target image of the first person wearing the clothes to be tried on can be obtained through S1051~S1054, which will be explained through the following steps.

[0047] S1051. Determine the first latent space feature vector after reweighting based on the binary mask image, pose feature information, second image and self-attention layer.

[0048] S1052. Input the first latent space feature vector and appearance feature information into the cross-attention layer to obtain the first query matrix, the first key matrix and the first value matrix corresponding to the first latent space feature vector.

[0049] The query matrix is ​​the starting point for searching or querying in the attention layer. Each input position generates a corresponding query matrix, which is used to calculate similarity with the key matrices of all positions to determine which positions' information should be focused on (or "attentioned") at the current position. The key matrix represents the target being queried or retrieved in the attention layer. Each input position also generates a corresponding key matrix, used to calculate similarity with the query matrix, helping the model understand the relationships between different positions in the sequence. The value matrix is ​​the vector containing the actual information in the attention layer. Each input position also generates a corresponding value matrix. It contains specific information about each position in the sequence, which is weighted and summed based on the attention weights calculated from the query matrix and the key matrix.

[0050] The main function of the cross-attention layer is to handle the dependency relationship between two different input data, thereby enabling the interaction and fusion of data.

[0051] In some embodiments of this application, a first key matrix is ​​determined by appearance feature information and a second preset weight matrix, a first value matrix is ​​determined by a third preset weight matrix and appearance feature information, and a first query matrix is ​​determined by a first latent space feature vector and a first preset weight matrix.

[0052] For example, These are the first query matrix, the first key matrix, and the first value matrix, respectively. The trainingable linear mapping layer consists of a first preset weight matrix, a second preset weight matrix, and a third preset weight matrix. These three preset weight matrices are automatically learned during the training process of the diffusion model. During training, the goal of the diffusion model is to minimize a certain loss function (such as cross-entropy loss), which typically involves adjusting the parameters of the diffusion model, including... Once training is complete, the first, second, and third preset weight matrices will be saved for use during model inference. For appearance feature information, First latent space feature vector.

[0053] S1053. Input the detailed feature information into the cross-attention layer to confirm the second query matrix, the second key matrix, and the second value matrix.

[0054] In some embodiments of this application, detailed feature information is input into the cross-attention layer and mapped to three different vector spaces through three different linear transformation matrices (weight matrices) to obtain the second query matrix, the second key matrix, and the second value matrix.

[0055] S1054. Matrix fusion is performed based on the first key matrix, the first value matrix, the second key matrix, and the second value matrix to determine the target image.

[0056] Matrix fusion aims to integrate results from multiple data sources to obtain more accurate and comprehensive information. Each data source can be viewed as a matrix, and different matrices can be merged by element-wise multiplication or addition. Matrix fusion can improve the accuracy, comprehensiveness, and robustness of information.

[0057] For example, as shown in the following formula (1):

[0058]

[0059] In the above formula (1), Equivalent to The two matrices were merged separately. First query matrix, It is the dimension of the key matrix.

[0060] It is understood that in some embodiments of this application, a reweighted first latent space feature vector is determined based on a binary mask image, pose feature information, a second image, and a self-attention layer. The first latent space feature vector and appearance feature information are then input into a cross-attention layer to obtain a first query matrix, a first key matrix, and a first value matrix corresponding to the first latent space feature vector. This achieves the interaction and fusion of clothing-related information and pose-related information. Detail feature information is input into the cross-attention layer to determine a second query matrix, a second key matrix, and a second value matrix. Matrix fusion is then performed based on the first key matrix, the first value matrix, the second key matrix, and the second value matrix to determine the target image. By fusing different matrices, the accuracy, comprehensiveness, and robustness of the information are improved, thereby enhancing the generation effect of the target image.

[0061] In some embodiments of this application, S1054 can be implemented by S1054A to S1054D, which will be described through the following steps.

[0062] S1054A: Merge the first value matrix and the second value matrix to obtain the target value matrix.

[0063] S1054B: Merge the first bond matrix and the second bond matrix to obtain the target bond matrix.

[0064] In some embodiments of this application, two value matrices or two key matrices contain different information or emphasize different aspects. By fusing matrices, the complementarity of this information can be fully utilized, and the bias or information gaps that may be caused by a single matrix can be reduced.

[0065] For example, in formula (1), The first bond matrix and the second bond matrix are merged, where, Second bond matrix, First bond matrix. It involves merging the first-value matrix and the second-value matrix, where... First-value matrix.

[0066] S1054C. Based on the first query matrix, the target value matrix and the target key matrix are weighted and summed to obtain the reweighted second latent space feature vector.

[0067] In some embodiments of this application, as described in formula (1), after calculating the attention score based on the first query matrix and the target key matrix, the obtained attention score is normalized by the softmax function so that the sum of all attention scores corresponding to each query is 1, thereby obtaining a weight vector. The normalized attention weight is used to perform weighted summation on the target value matrix to obtain the context representation of each query. This context vector contains the weighted information of all elements in the sequence for the current query and is a comprehensive representation of the information at different positions in the input feature vector. Finally, the reweighted second latent space feature vector is obtained.

[0068] S1054D, based on the second latent space feature vector and the preset decoder, obtains the target image.

[0069] The preset decoder can be a decoder for a variational autoencoder (VAE). The role of the decoder in a VAE is to transform the latent vector sampled from the latent space back into the original data space, thereby generating reconstructed data that is as close as possible to the original input data, while preserving the key features of the data.

[0070] In some embodiments of this application, after obtaining the second latent space feature vector, the target image is obtained by decoding and reconstructing it through a preset decoder.

[0071] It is understood that in some embodiments of this application, the first value matrix and the second value matrix are fused to obtain the target value matrix, and the first key matrix and the second key matrix are fused to obtain the target key matrix. Matrix fusion can fully utilize the complementarity of this information and reduce the bias or information loss that may be caused by a single matrix. Based on the second latent space feature vector and the preset decoder, the target image is obtained. This reconstruction capability enables the preset decoder to learn the inherent representation of the data and capture the main features of the data, thereby improving the generation effect of the target image.

[0072] In some embodiments of this application, the determination of the reweighted first latent space feature vector based on the binary mask image, pose feature information, second image and self-attention layer in S1051 can be implemented by S1051A~S1051B, which will be specifically explained through the following steps.

[0073] S1051A: The binary mask image, pose feature information, and second image are concatenated to obtain the concatenated latent space feature vector.

[0074] Channel concatenation allows feature vectors from different levels to be combined, thus utilizing both high-level abstract features and low-level detailed features simultaneously. By increasing the number of channels, the model can learn more feature representations, which helps improve the model's expressive power and discriminative ability. Pose feature information already has noise added; concatenating pose feature information with added latent space noise can enhance the model's generalization ability, robustness, and perform data augmentation.

[0075] In some embodiments of this application, the second image is input into a preset encoder (which may be a VAE encoder) to obtain a 4-channel latent space for redrawing the background. The latent space feature vector of the 4-channel redrawn background, the pose feature information with added latent space noise of 4 channels, and the binary mask image of 1 channel are concatenated along the channels to obtain the input of 9 channels, which is the concatenated latent space feature vector.

[0076] S1051B: The first latent space vector is obtained by weighted summation based on the concatenated latent space feature vector and the self-attention layer.

[0077] In some embodiments of this application, the self-attention layer can enhance the model's understanding and modeling ability of dependencies between features by calculating the correlation between feature vectors. The concatenated latent space feature vector is first processed by other layers of the pre-trained diffusion model, and then input into the self-attention layer. After generating the query matrix, transforming the key matrix and value matrix, calculating the attention score, applying the softmax function, and performing weighted summation, a reweighted first latent space feature vector is obtained.

[0078] Understandably, in some embodiments of this application, the binary mask image, pose feature information, and second image are concatenated through channels to obtain a concatenated latent space feature vector. By increasing the number of channels, the network can learn more feature representations, which helps improve the model's expressive power and discriminative ability. The concatenated latent space feature vector is then input into a self-attention layer for weighted summation to obtain a reweighted first latent space vector. By calculating the correlation between feature vectors, the model's ability to understand and model the dependencies between features is enhanced.

[0079] The following is combined Figure 1 and Figure 3 This application introduces the model training method provided in its embodiments. For example... Figure 3 As shown, Figure 3 This is a schematic diagram of a model training process provided in an embodiment of this application.

[0080] like Figure 3 As shown, the model training methods include:

[0081] S201. Obtain a clothing sample image and a first image; wherein, in the first image, a first person is wearing a clothing sample.

[0082] In the embodiments of this application, during the model training phase, clothing sample images are collected and used to train the model. The clothing samples can be clothing intended for display to users or for sale, or clothing used only for model training and not displayed to users. The first person can be a live model displaying the clothing, a simulated model displaying the clothing, a digital model displaying the clothing, or an image retained by a customer who has purchased clothing, etc. In the first image, the clothing sample worn by the first person is the same as the acquired clothing sample; that is, during the model training phase, a pair of clothing sample images and an image of the first person wearing that clothing sample are required as input. The clothing sample images in the model training phase are equivalent to the clothing images to be tried on in the model inference phase.

[0083] S202. Occlude the clothing sample in the first image to obtain a binary mask image of the clothing sample and a second image after the clothing sample is occluded.

[0084] In embodiments of this application, a binary mask image is used to accurately separate the outline of clothing from the background, allowing a person to "try on" different garments in a digital environment without actually wearing them. The second image is a background image for redrawing the clothing image; that is, the clothing to be tried on can be drawn in the second image.

[0085] By using open-source semantic segmentation techniques, such as SAM, the clothing sample worn by the first person can be extracted from the first image to obtain a binary mask image of the clothing sample and a second image after the clothing sample is occluded.

[0086] S203. Based on the first image and the person's posture analysis model, perform posture extraction to obtain the posture feature information of the first person.

[0087] S204. Input the clothing sample images into the clothing detail analysis model and the clothing appearance analysis model to be trained, respectively, to obtain the clothing detail feature information and the clothing appearance feature information.

[0088] In the embodiments of this application, the implementation methods for obtaining posture feature information in S203 and S103 are similar, and the implementation methods for obtaining detailed feature information and appearance feature information in S204 and S104 are similar, and will not be described in detail here.

[0089] S205. The pose feature information after adding latent noise is used as the conditional input side of the diffusion model to be trained, and the binary mask image, the second image, the detailed feature information and the appearance feature information are used as the main input side of the diffusion model to be trained to determine the noise loss and pixel-level loss; and the model is trained by noise loss and pixel-level loss until a trained model is obtained.

[0090] A loss function is a function used to quantify the difference between a model's prediction and the actual target value. The goal of training a model is to minimize this loss function by adjusting the model's parameters, thereby making the model's predictions as close as possible to the true output.

[0091] In the embodiments of this application, the pose feature information after adding latent space noise is used as the conditional input of the diffusion model to be trained, and the second image, binary mask image, detail feature information, and appearance feature information are used as the main input of the diffusion model to be trained, resulting in predicted noise and predicted image. Based on the predicted noise and predicted image, noise loss and pixel-level loss are determined. Finally, the model is trained using noise loss and pixel-level loss until the training conditions are met, resulting in the trained model.

[0092] During model training, noise loss and pixel-level loss can be used simultaneously to train the clothing detail analysis model, clothing appearance analysis model, and diffusion model.

[0093] It is understood that, in the embodiments of this application, the pose feature information, detail feature information, appearance feature information, second image, and binary mask image of the person are obtained from the acquired clothing sample image and the first image, respectively. Then, the pose feature information is used as a conditional constraint input to the conditional input side of the diffusion model. The second image, binary mask image, detail feature information, and appearance feature information are used as the main input side of the diffusion model to be trained to determine the noise loss and pixel-level loss. Using multiple loss values ​​for training can improve the performance and generalization ability of the model.

[0094] In some embodiments of this application, S205 uses the pose feature information after adding latent noise as the conditional input side of the diffusion model to be trained, and uses the binary mask image, the second image, the detailed feature information and the appearance feature information as the main input side of the diffusion model to be trained. The noise loss and pixel-level loss can be determined through S2051~S2053, which will be explained through the following steps.

[0095] S2051. The pose feature information after adding latent noise, the binary mask image, and the second image are concatenated to determine the latent space feature vector after concatenation.

[0096] For example, the second image is input into a preset encoder (which can be a VAE encoder) to obtain the latent space of the redrawn background in 4 channels. The latent space feature vector of the 4-channel redrawn background, the pose feature information with added latent space noise of 4 channels, and the binary mask image of 1 channel are concatenated along the channels to obtain the input of 9 channels, which is the concatenated latent space feature vector.

[0097] S2052. Determine the predicted image and predicted noise based on the stitched latent space feature vector, detail feature information and appearance feature information.

[0098] S2053. Determine the noise loss based on the predicted noise and the real noise; and determine the pixel-level loss based on the predicted image and the real image.

[0099] In some embodiments of this application, the predicted image and predicted noise are obtained based on the concatenated latent space feature vector, detail feature information, and appearance feature information as the main input to the diffusion model to be trained. Noise loss is determined based on the error between the predicted noise and the actual noise, and pixel-level loss is determined based on the error between the predicted image and the real image. When calculating the noise loss and pixel-level loss, loss functions such as mean squared error, cross-entropy loss, or absolute error loss can be used.

[0100] For example, the noise loss is shown in formula (2):

[0101]

[0102] In the above formula (2), where, These are the pose analysis model, clothing detail analysis model, and diffusion model, which are the trainable parts of the model. This is a pre-trained CLIP image encoder that does not participate in training. Predict noise, Pose images, , .

[0103] The pixel-level loss is calculated as shown in formula (3) below:

[0104]

[0105] In the above formula (3), , , To predict the image.

[0106] It is understood that, in the embodiments of this application, the predicted image and predicted noise are obtained based on the pose feature information after adding latent noise, the binary mask image, and the second image as the main input side of the diffusion model to be trained. The noise loss is determined by minimizing the difference between the predicted noise and the actual added noise, and the pixel-level loss is determined by the predicted image and the actual image. The noise loss and pixel-level loss can accurately reflect the difference between the model prediction and the real target, and can guide the model to develop towards the optimal solution, thus ensuring the effectiveness, efficiency, and ultimate practicality of the model training.

[0107] In some embodiments of this application, S205 performs model training through noise loss and pixel-level processing until a trained model is obtained. This can be achieved through S205A to S205B, as described in the following steps.

[0108] S205A: Summing the noise loss and pixel-level loss yields the total loss.

[0109] S205B: Train the human pose analysis model, clothing detail analysis model, and diffusion model to be trained separately using the total loss until the pre-trained clothing detail analysis model, pre-trained human pose analysis model, and pre-trained diffusion model are obtained respectively.

[0110] For example, the formula for calculating the total loss is shown in (4) below:

[0111]

[0112] In the above formula (4),

[0113] After calculating the total loss, the total loss is used to train the human posture analysis model, the clothing detail analysis model, and the diffusion model to be trained simultaneously until the training conditions are met, thus obtaining the pre-trained clothing detail analysis model, the pre-trained human posture analysis model, and the pre-trained diffusion model.

[0114] It is understood that in some embodiments of this application, the noise loss and pixel-level loss are summed to obtain the total loss. The diffusion model, clothing detail analysis model, and clothing pose analysis model to be trained are then trained using the total loss to obtain the trained model. By summing the losses, different loss functions are defined for each type of data, so that the model can effectively integrate multiple information sources, improving the model's performance and generalization ability.

[0115] In the embodiments of this application, the above steps are combined, such as Figure 4 As shown, Figure 4 This is a general flowchart of a virtual try-on method provided in an embodiment of this application. Figure 4 In this process, the image acquisition module acquires an image of the garment to be tried on and a first image. The first image shows a first person (e.g., a model) wearing the first garment (during training, the first garment worn by the first person is the same as the garment to be tried on; during inference, the first garment worn by the first person is different from the garment to be tried on). The image of the garment to be tried on and the first image are input into the image processing module to obtain a binary mask image of the first garment, a second image of the first garment with the first garment occluded, and a pose image of the first person. First, the image of the garment to be tried on is processed by the VAE encoder and then input into the garment detail analysis model. Simultaneously, the image of the garment to be tried on is input into the garment appearance analysis model (CLIP image encoder). The garment detail analysis model preserves the details of the garment to be tried on. The garment appearance analysis model retains the overall color and texture of the garment to be tried on. Then, the detail feature information and appearance feature information are input into the diffusion model. The acquired pose image is input into the pose analysis model to obtain the pose feature information of the first person, which serves as additional pose control for the first person. The second image is input into the VAE encoder to obtain the latent space feature vector. The latent space feature vector, the pose feature information with added noise, and the binary mask image are concatenated to obtain the concatenated latent space feature vector. The concatenated feature vector is input into the diffusion model, first passing through the self-attention layer to obtain the reweighted first latent space feature vector. The first latent space feature vector and the appearance feature information are input into the cross-attention layer to determine the query matrix, key matrix, and value matrix (as described in detail above). The determined key matrix and value matrix are fused with the value matrix and key matrix corresponding to the detail feature information, respectively, to obtain the reweighted second latent space feature vector. The VAE decoder is used to reconstruct the data from the second latent space feature vector to obtain the target image.

[0116] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method provided in any of the above embodiments.

[0117] This application also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method provided in any of the above embodiments.

[0118] Reference Figure 5 This document illustrates a schematic diagram of an electronic device according to an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0119] like Figure 5As shown, the electronic device may include: a processor 1002, a communications interface 1004, a memory 1006, and a communications bus 1008.

[0120] in:

[0121] The processor 1002, communication interface 1004, and memory 1006 communicate with each other via communication bus 1008.

[0122] Communication interface 1004 is used to communicate with other electronic devices or servers.

[0123] The processor 1002 is used to execute program 1010, specifically the relevant steps in the above method embodiments.

[0124] Specifically, program 1010 may include program code that includes computer operation instructions.

[0125] The processor 1002 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0126] Memory 1006 is used to store program 1010. Memory 1006 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0127] Specifically, program 1010 can be used to cause processor 1002 to execute the operations corresponding to the methods described in the above method embodiments.

[0128] The specific implementation of each step in program 1010 can be found in the corresponding steps and units described in the above method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0129] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0130] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0131] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0132] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A virtual try-on method, characterized in that, include: Acquire an image of the garment to be tried on and a first image; wherein, in the first image, a first person is wearing the first garment; The first garment in the first image is separated to obtain a binary mask image of the first garment and a second image after the first garment is occluded; Based on the first image and the person's pose analysis model, pose extraction is performed to obtain the pose feature information of the first person; The images of the garments to be tried on are input into the garment detail analysis model and the garment appearance analysis model, respectively, to obtain the corresponding detail feature information and appearance feature information; The pose feature information with added latent noise is used as the conditional input of the pre-trained diffusion model, and the binary mask image, the second image, the detail feature information and the appearance feature information are used as the main input of the pre-trained diffusion model to obtain the target image of the first person wearing the clothing to be tried on. The pre-trained diffusion model includes a cross-attention layer and a self-attention layer. The pose feature information with added latent noise is used as the conditional input to the pre-trained diffusion model, and the binary mask image, the second image, the detail feature information, and the appearance feature information are used as the main input to the pre-trained diffusion model to obtain the target image of the first person wearing the clothing to be tried on. Based on the binary mask image, the pose feature information, the second image, and the self-attention layer, a reweighted first latent space feature vector is determined. The first latent space feature vector and the appearance feature information are input into the cross attention layer to obtain the first query matrix, the first key matrix and the first value matrix corresponding to the first latent space feature vector; The detailed feature information is input into the cross-attention layer to determine the second query matrix, the second key matrix, and the second value matrix; The target image is determined by matrix fusion based on the first key matrix, the first value matrix, the second key matrix, and the second value matrix.

2. The method according to claim 1, characterized in that, The step of determining the target image by matrix fusion based on the first key matrix, the first value matrix, the second key matrix, and the second value matrix includes: The first value matrix and the second value matrix are fused to obtain the target value matrix; The first key matrix and the second key matrix are fused to obtain the target key matrix; Based on the first query matrix, the target value matrix and the target key matrix are weighted and summed to obtain a reweighted second latent space feature vector; The target image is obtained based on the second latent space feature vector and the preset decoder.

3. The method according to claim 1, characterized in that, The first query matrix is ​​determined by the first latent space feature vector and the first preset weight matrix; the first key matrix is ​​determined by the appearance feature information and the second preset weight matrix; the first value matrix is ​​determined by the appearance feature information and the third preset matrix.

4. The method according to claim 1, characterized in that, The step of determining the reweighted first latent space feature vector based on the binary mask image, the pose feature information, the second image, and the self-attention layer includes: The binary mask image, the pose feature information, and the second image are concatenated by channels to obtain the concatenated latent space feature vector; The first latent space feature vector is obtained by weighted summation based on the concatenated latent space feature vector and the self-attention layer.

5. A model training method, characterized in that, include: Acquire a clothing sample image and a first image; wherein, in the first image, a first person is wearing the clothing sample; The clothing sample in the first image is masked to obtain a binary mask image of the clothing sample and a second image after the clothing sample is masked. Based on the first image and the person's pose analysis model, pose extraction is performed to obtain the pose feature information of the first person; The clothing sample images are input into the clothing detail analysis model and the clothing appearance analysis model to be trained, respectively, to obtain the clothing detail feature information and the clothing appearance feature information. The pose feature information with added latent noise is used as the conditional input of the diffusion model to be trained, and the binary mask image, the second image, the detail feature information, and the appearance feature information are used as the main input of the diffusion model to be trained, to obtain predicted noise and predicted image. Noise loss and pixel-level loss are determined based on the predicted noise and predicted image; and the model is trained using the noise loss and the pixel-level loss until a trained model is obtained, including: The pose feature information with added latent noise, the binary mask image, and the second image are concatenated to determine the concatenated latent space feature vector; The predicted image and predicted noise are determined based on the stitched latent space feature vector, the detail feature information, and the appearance feature information. The noise loss is determined based on the predicted noise and the actual noise; and the pixel-level loss is determined based on the predicted image and the actual image. The total loss is obtained by summing the noise loss and the pixel-level loss. The total loss is used to train the character pose analysis model, the clothing detail analysis model, and the diffusion model to be trained, respectively, until pre-trained clothing detail analysis models, pre-trained character pose analysis models, and pre-trained diffusion models are obtained.

6. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform an operation corresponding to the method described in any one of claims 1-4 or 5.

7. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-4 or 5.

Images