Virtual fitting method, model training method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a virtual fitting method, a model training method, electronic equipment and a storage medium. The virtual fitting method comprises the following steps: acquiring an image of to-be-fitted clothes, description information of the to-be-fitted clothes and a first image; the first figure holds the first clothes in the first image; separating the first clothes in the first image to obtain a binary mask graph of the first clothes and a second image after the first clothes are shielded; inputting the first image into a figure posture analysis model to obtain posture feature information of the first figure; inputting the image of the to-be-tried-on clothes into a clothes analysis model to obtain clothes feature information of the to-be-tried-on clothes; inputting the description information into a text analysis model to obtain semantic feature information; and taking the posture feature information as a condition input side of a pre-trained diffusion model, taking the second image, the binary mask graph, the clothes feature information and the semantic feature information as a main input side, and obtaining a target image of the to-be-tried clothes of the first person.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of visual processing technology, and in particular to a virtual try-on method, a model training method, an electronic device, and a storage medium. Background Art

[0002] In the e-commerce clothing industry, the visual presentation of goods is crucial to sales. Although traditional flat clothing images are sufficient, images presented by models are usually more attractive to customers. However, producing a large number of model display images is time-consuming and laborious. To solve this problem, virtual try-on technology came into being, which can not only enhance consumers' shopping experience, but also effectively reduce the operating costs of clothing retailers.

[0003] In related technologies, the main way to try on clothes is to formulate a standard style, and then paste the customer's head portrait on it to let consumers feel the effect of trying on the clothes. However, since the actual body shape of the person trying on the clothes is not taken into consideration, and due to the soft nature of the clothes, they will deform to a certain extent when they fit the human body, which makes virtual fitting technology face challenges when dealing with clothes with complex textures or human body dynamics, thus affecting the realism of the final image.

[0004] Therefore, how to generate high-quality virtual try-on effects becomes an urgent problem to be solved. Summary of the invention

[0005] In view of this, embodiments of the present application provide a virtual try-on method, a model training method, an electronic device and a storage medium to at least partially solve the above problems.

[0006] According to a first aspect of an embodiment of the present application, a virtual try-on method is provided, comprising:

[0007] Acquire an image of clothing to be tried on, description information of the clothing to be tried on, and a first image; wherein, in the first image, a first person wears a first clothing; separate the first clothing in the first image to obtain a binary mask image of the first clothing and a second image after the first clothing is blocked; input the first image into a person posture analysis model to obtain posture feature information of the first person; input the image of clothing to be tried on into a clothing analysis model to obtain clothing feature information of the clothing to be tried on; input the description information into a text analysis model to obtain semantic feature information; use the posture feature information as the conditional input side of the pre-trained diffusion model, and use the second image, the binary mask image, the clothing feature information, and the semantic feature information as the main input side of the pre-trained diffusion model to obtain a target image of the first person wearing the clothing to be tried on.

[0008] According to a second aspect of an embodiment of the present application, a model training method is provided, including:

[0009] Acquire a clothing sample image and a first image; wherein, in the first image, a first person wears a clothing sample; shield the clothing sample in the first image to obtain a binary mask image of the clothing sample and a second image after the clothing sample is shielded; input the first image into a person posture analysis model to obtain posture feature information of the first person; input the clothing sample image into the clothing analysis model to obtain clothing feature information; input the acquired description information of the clothing sample into a text analysis model to obtain semantic feature information; use the posture feature information as a conditional input side of a diffusion model to be trained, and use the second image, the binary mask image, the clothing feature information and the semantic feature information as the main input side of the diffusion model to be trained to determine noise loss and pixel-level loss; and train the diffusion model to be trained by the noise loss and the pixel-level loss to obtain a trained diffusion model.

[0010] According to a third aspect of an embodiment of the present application, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in the first aspect or the second aspect.

[0011] According to a fourth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect or the second aspect is implemented.

[0012] According to the scheme provided by the embodiment of the present application, an image of clothing to be tried on, description information of the clothing to be tried on and a first image are obtained; wherein, in the first image, a first person wears a first piece of clothing; the first piece of clothing in the first image is separated to obtain a binary mask map of the first piece of clothing and a second image after the first piece of clothing is blocked; the first image is input into a character posture analysis model to obtain posture feature information of the first person; the image of clothing to be tried on is input into a clothing analysis model to obtain clothing feature information of the clothing to be tried on; the description information is input into a text analysis model to obtain semantic feature information; the posture feature information is used as the conditional input side of a pre-trained diffusion model, and the second image, the binary mask map, the clothing feature information and the semantic feature information are used as the main input side of the pre-trained diffusion model to obtain a target image of the first person wearing the clothing to be tried on. In this process, the specific area where the clothing is separated in the second image can be redrawn in combination with the description information of the clothing and the binary mask map, so that the newly drawn image area of ​​the clothing to be tried on can be harmoniously integrated with other parts of the original second image to achieve visual unity and naturalness. The posture feature information is input as a conditional constraint into the conditional input side of the diffusion model, so that the target image finally generated can visualize the effect of the clothes to be tried on the person, while maintaining the authenticity and naturalness of the overall image. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0014] Figure 1 A general architecture diagram of a virtual try-on method provided in an embodiment of the present application;

[0015] Figure 2 It is a flowchart of a virtual try-on method provided in an embodiment of the present application;

[0016] Figure 3 It is a flowchart of a model training method provided in an embodiment of the present application;

[0017] Figure 4 A general flow chart of a virtual try-on method provided in an embodiment of the present application;

[0018] Figure 5 The present invention is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0020] In order to adapt the virtual try-on method provided in the embodiment of the present application, the embodiment of the present application provides a general architecture diagram of the virtual try-on method, that is, the virtual try-on method of the embodiment of the present application can be applied to the virtual try-on system provided in the embodiment of the present application. Figure 1 As shown, the virtual try-on system includes an image acquisition module, an image processing module and a target image generation module. The image acquisition module is used to acquire a clothing image and a first image, in which a first person (such as a model) wears a first clothing, wherein in the training stage, the acquired clothing image is the same as the first clothing image worn by the first person; in the inference stage, the acquired clothing image and the first clothing image worn by the first person may be different. The image processing module is used to process the clothing image and the first image, specifically by using an open source semantic segmentation technology, such as the SegmentAnything Model (SAM), the first clothing worn by the first person can be image separated in the first image, and then a binary mask image of the clothing and a second image after the clothing is blocked, that is, a background image of the redrawn image, and an open source model for extracting human posture is used to extract human posture from the first image. The description information of the clothing image is saved, and for clothing without description information, a large model can be used to generate description information corresponding to the clothing image. The target image generation module specifically includes: extracting features of the clothing image through a clothing image prompt adapter to obtain clothing feature information. The description information is feature extracted through the text prompt module to obtain semantic feature information, and the posture image is feature extracted through the posture prompt model (which can be a ControlNet model) to obtain posture feature information. The acquired posture feature information, semantic feature information, clothing feature information, the second image of the occluded clothing, and the binary mask image of the clothing are input into the diffusion model to obtain the target image.

[0021] Combine the following Figure 1 and Figure 2 The virtual try-on method provided by the embodiment of the present application is introduced. Figure 2 As shown, Figure 2 This is a flow chart of the virtual try-on method provided in the embodiment of the present application. Figure 1 The virtual try-on method provided in the embodiment of the present application can be executed by an electronic device, and the electronic device can be, for example, a computer, a server, etc.

[0022] like Figure 2 As shown, the virtual try-on method comprises:

[0023] S101. Acquire an image of clothing to be tried on, description information of the clothing to be tried on, and a first image; wherein in the first image, a first person wears a first clothing.

[0024] In an embodiment of the present application, the image of clothing to be tried on may be clothing that needs to be displayed or sold to the user, and the descriptive information is information that explains the appearance, details or wearing scene of the clothing; the first image is an image of a first person wearing a first clothing, wherein the first person may be a real-life model for displaying clothing, or a simulated model for displaying clothing, or a digital model for displaying clothing, or a customer who purchases clothing, etc.; the first clothing may be different from the clothing to be tried on.

[0025] Exemplarily, an image of clothing to be tried on is obtained, and the descriptive information of the clothing to be tried on can be the style of the clothing (design style and details, such as round neck, V-neck, straight tube, slim fit, etc.), color, size (size range from small to large, such as S, M, L, XL, etc.), material (fabric composition, such as cotton, silk, polyester, etc.) and category, etc.

[0026] S102: Separate the first clothing in the first image to obtain a binary mask image of the first clothing and a second image after the first clothing is blocked.

[0027] In the embodiment of the present application, the binary mask image is used to accurately separate the outline of the clothing from the background, so that the character can "try on" different clothing in a digital environment without actually wearing them. The second image is the background image for redrawing the clothing image, that is, the clothing to be tried on can be drawn in the second image.

[0028] By using open source semantic segmentation technology, such as SAM, the first clothing can be accurately separated from the image of the first person wearing the clothing, thereby obtaining a binary mask image of the clothing and a second image after the first clothing is blocked.

[0029] S103: Input the first image into a character posture analysis model to obtain posture feature information of the first character.

[0030] After the first clothing of the first person is stripped off, when the clothing to be tried on is redrawn on the image of the first person, it is necessary to ensure that the face, hairstyle and body movements of the first person remain unchanged in their original state. In this way, the target image finally generated can visualize the effect of the clothing to be tried on the first person, while maintaining the authenticity and naturalness of the overall image, so it involves the acquisition of the character's posture information.

[0031] Posture feature information involves the position, direction and movement of various parts of the body, and how these factors are combined to express the individual's emotions, intentions and personality. It includes but is not limited to head posture feature information, torso feature information, facial expression feature information, and micro-expression and micro-movement feature information.

[0032] In an embodiment of the present application, the character posture analysis model may include a posture prompt ControlNet model, inputting the first image into an encoder ε of a variational autoencoder to obtain a latent space encoding c of the posture p =ε(I pose ), input the latent space code into the posture hint ControlNet, that is, the encoding block and intermediate block copy of the diffusion model, after the zero convolution layer, add the encoding block and intermediate block of the posture hint ControlNet to the decoding block and intermediate block of the diffusion model through the residual connection, and input the posture feature information into the diffusion model. Among them, the ControlNet model branch creates a trainable diffusion model U-Net encoding block and intermediate block copy, and adds an additional zero convolution layer. The encoding block (Downblocks) and intermediate block of ControlNet are added to the decoding block (Upblocks) and intermediate block of the diffusion model.

[0033] S104: inputting the image of the clothing to be tried on into a clothing analysis model to obtain clothing feature information of the clothing to be tried on.

[0034] In the embodiments of the present application, the characteristic information of clothing is multifaceted, not only related to practicality and comfort, but also reflects cultural and fashion trends, including but not limited to color characteristic information, material characteristic information, pattern and print characteristic information, and accessory characteristic information.

[0035] The clothing analysis model may include a contrastive language-image pre-training encoder (CLIP), a linear layer, and a layer normalization layer. The clothing analysis model is used to process the image of the clothing to be tried on to obtain feature information of the clothing to be tried on.

[0036] S105: Input the description information into a text analysis model to obtain semantic feature information.

[0037] In an embodiment of the present application, the descriptive information is information that explains the appearance, details or wearing scenarios of the clothing. When generating a target image, the descriptive information of the clothing is added. The semantic information can provide rich guidance information for the generation of the target image, thereby generating a more realistic and delicate clothing display effect.

[0038] The text analysis model can be a CLIP text encoder, which inputs the description information into the text analysis model to obtain semantic feature information and provide text reference for the generation of the target image.

[0039] S106. Using the posture feature information as the conditional input side of the pre-trained diffusion model, and using the second image, the binary mask image, the clothing feature information and the semantic feature information as the main input side of the pre-trained diffusion model, to obtain a target image of the first person wearing the clothing to be tried on.

[0040] Pre-trained diffusion models allow the model to generate data based on additional input information (conditions), which can be a category label, a text description, another image, or any form of guiding signal, with the goal of guiding the generation process so that the generated results meet specific conditions or properties.

[0041] In an embodiment of the present application, the second image, the binary mask map, the clothing feature information and the semantic feature information are used as the main input side of the pre-trained diffusion model, and the posture feature information is used as the conditional input side of the pre-trained diffusion model and as an additional input condition of the pre-trained diffusion model for guidance to obtain a target image of the first person wearing the clothing to be tried on.

[0042] It can be understood that in an embodiment of the present application, an image of clothing to be tried on, description information of the clothing to be tried on, and a first image are obtained; wherein, in the first image, a first person wears a first clothing; the first clothing in the first image is separated to obtain a binary mask image of the first clothing and a second image after the first clothing is blocked; the first image is input into a character posture analysis model to obtain posture feature information of the first person; the image of the clothing to be tried on is input into a clothing analysis model to obtain clothing feature information of the clothing to be tried on; the description information is input into a text analysis model to obtain semantic feature information; the posture feature information is used as the conditional input side of a pre-trained diffusion model, and the second image, the binary mask image, the clothing feature information, and the semantic feature information are used as the main input side of the pre-trained diffusion model to obtain a target image of the first person wearing the clothing to be tried on. In this process, the specific area where the clothes are separated in the second image can be combined with the description information of the clothes and the binary mask map to redraw the image of the clothes to be tried on, so that the newly drawn image area of ​​the clothes to be tried on can be harmoniously integrated with the other parts of the original second image to achieve visual unity and naturalness. And the posture feature information is input as a conditional constraint to the conditional input side of the diffusion model, so that the target image finally generated can visualize the effect of the clothes to be tried on the person, while maintaining the authenticity and naturalness of the overall image.

[0043] In some embodiments of the present application, S106 uses posture feature information as the conditional input side of a pre-trained diffusion model, and uses the second image, binary mask map, clothing feature information and semantic feature information as the main input side of the pre-trained diffusion model. Obtaining the target image of the first person wearing the clothes to be tried on can be achieved through S1061 to S1066, which is explained in the following steps.

[0044] S1061. Input the second image into the pre-trained encoder to obtain a first latent space feature vector.

[0045] The pre-trained encoder can be an encoder for a variational autoencoder. The encoder takes input data (such as images, audio, or text) and converts it into a set of compact, low-dimensional feature representations. This process usually involves multiple layers of neural networks, each of which gradually abstracts the input data and extracts higher-level features. Unlike traditional autoencoders, the encoder of a VAE does not just reconstruct the input data. It also attempts to learn the distribution of a latent space, where points in this space correspond to samples in the original data set. This means that the encoder not only compresses the data, but also estimates the probability distribution of each data point in the latent space.

[0046] In some embodiments of the present application, the second image is input into a pre-trained encoder, and the encoder extracts key features of the second image through a series of neural network layers, such as convolutional layers, fully connected layers, or other types of layers. The VAE encoder outputs the parameters of the latent variables, that is, the first latent space feature vector is obtained.

[0047] S1062: Concatenate the first latent space feature vector, the latent space noise, and the binary mask image according to a preset channel to obtain a concatenated latent space feature vector.

[0048] Channel concatenation allows feature vectors of different levels to be combined, so that both high-level abstract features and low-level detailed features can be utilized. By increasing the number of channels, the network can learn more feature representations, which helps improve the expressiveness and discrimination of the model. By concatenating latent space noise, the generalization ability and robustness of the model can be enhanced, as well as data enhancement can be performed.

[0049] In some embodiments of the present application, the second image is input into the pre-trained encoder to obtain the first latent space feature vector of 4 channels, and the latent space random Gaussian noise is randomly generated. The first latent space feature vector of 4 channels, the generated latent space noise and the binary mask map of 1 channel are spliced ​​along the channel, and finally a latent space feature vector of 9 channels is obtained, that is, the spliced ​​latent space feature vector.

[0050] S1063. Perform feature extraction based on the concatenated latent space feature vector to obtain a second latent space feature vector.

[0051] S1064. Determine a third latent space feature vector based on the second latent space feature vector, semantic feature information, and text cross-attention layer.

[0052] In some embodiments of the present application, key features are extracted from the spliced ​​latent space feature vector to obtain a second latent space feature vector. The second latent space feature vector represents the image features, and the text cross-attention layer can focus on the key information in the semantic information. The second latent space feature vector and the semantic feature information are input into the text cross-attention layer to establish a connection between the two different features, enhance the understanding between the two features, and obtain a third latent space feature vector that fuses the image feature information and the semantic feature information.

[0053] S1065. Determine a fourth latent space feature vector based on the second latent space feature vector, clothing feature information, and the image cross-attention layer.

[0054] In some embodiments of the present application, the key features of the spliced ​​latent space feature vector are extracted to obtain a second latent space feature vector. The second latent space feature vector represents the image features, and the image cross-attention layer can focus on the key information in the image information. The second latent space feature vector and the clothing feature information are input into the image cross-attention layer to establish a connection between the two different features, enhance the understanding between the two features, and obtain a fourth latent space feature vector that fuses the image feature information and the clothing feature information.

[0055] S1066. Determine the target person image based on the third latent space feature vector, the fourth latent space feature vector and the posture feature information.

[0056] In some embodiments of the present application, the pre-trained diffusion model strips the first clothing worn by the first person in the first image, and then redraws the clothing to be tried on. This process needs to ensure that other parts of the model, such as the face, hairstyle, and background, remain in their original state. In this way, the final generated image can visualize the effect of the clothing to be tried on the person, while maintaining the authenticity and naturalness of the overall image. Therefore, a more realistic and natural target image is determined by combining the third latent space vector and the fourth latent space vector with the posture feature information.

[0057] It can be understood that in some embodiments of the present application, the second character image is input into the pre-trained encoder to obtain the first latent space feature vector, so that higher-level features are extracted. The first latent space feature vector, latent space noise and binary mask map are spliced ​​according to the preset channel to obtain the spliced ​​latent space feature vector, improve the model's expression ability and discrimination ability and the generalization ability of the model, etc.; feature extraction is performed based on the spliced ​​latent space feature vector to obtain the second latent space feature vector; based on the second latent space feature vector, semantic feature information and text cross-attention layer, the third latent space feature vector is determined; based on the second latent space feature vector, clothing feature information and image cross-attention layer, the fourth latent space feature vector is determined, and through the text cross-force mechanism and the image cross-force mechanism, the model can establish a connection between two different features and enhance the understanding between the two features; based on the third latent space feature vector, the fourth latent space feature vector and the posture feature information, the target character image is determined, and the posture feature information is introduced to determine a more realistic and natural target character image.

[0058] In some embodiments of the present application, S1064 determines that the third latent space feature vector is based on the second latent space feature vector, semantic feature information and text cross-attention layer, which can be implemented through S1064A and S1064B, which is specifically explained through the following steps.

[0059] S1064A, input the semantic feature information and the second latent space feature vector into the text cross attention layer to determine the first query matrix.

[0060] S1064B. Perform weighted summation based on the first query matrix to obtain a third latent space eigenvector.

[0061] Exemplarily, the semantic feature information and the second latent space feature vector are input into the text cross attention layer, and the first query matrix and the third latent space feature vector are calculated by the following formula (1):

[0062]

[0063] Where Q = ZW q , K = c t W k , V = c t W v are the query, key, and value matrices of text features, respectively, and W q , W k , W v is a trainable linear mapping layer, Z is the second latent space feature vector, Q is the first query matrix, and Z′ is the third latent space feature vector.

[0064] It can be understood that in some embodiments of the present application, the semantic feature information and the second latent space feature vector are input into the text cross-attention layer, the first query matrix is ​​determined, and a weighted sum is performed based on the first query matrix to obtain the third latent space feature vector, so that the model can establish a connection between the two different features of the semantic feature information and the second latent space feature vector, thereby enhancing the understanding between the two features.

[0065] In some embodiments of the present application, S1065 determines that the fourth latent space feature vector is based on the second latent space feature vector, clothing feature information and image cross-attention layer, which can be implemented through S1065A and S1065B, which is specifically explained through the following steps.

[0066] S1065A, input the clothing feature information and the second latent space feature vector into the image cross-attention layer to determine the second query matrix.

[0067] S1065B. Perform weighted summation based on the second query matrix to obtain a fourth latent space eigenvector.

[0068] Exemplarily, the clothing feature information and the second latent space feature vector are input into the image cross attention layer, and the first query matrix and the third latent space feature vector are calculated by the following formula (2):

[0069]

[0070] Where Q = ZW q , K′=c i W′ k , V′=c i W′ v are the query, key, and value matrices of image features respectively. k and W′ v is a trainable linear mapping layer, Z is the second latent space feature vector, Q is the second query matrix, and Z″ is the fourth latent space feature vector.

[0071] It can be understood that in some embodiments of the present application, the clothing feature information and the second latent space feature vector are input into the image cross-attention layer, the second query matrix is ​​determined, and a weighted sum is performed based on the second query matrix to obtain the fourth latent space feature vector, so that the model can establish a connection between the two different features of the clothing feature information and the second latent space feature vector, thereby enhancing the understanding of the two features.

[0072] In some embodiments of the present application, determining the target person image based on the third latent space feature vector, the fourth latent space feature vector and the posture feature information in S1066 can be implemented through S1066A to S1066C, which is explained through the following steps.

[0073] S1066A. Perform an AND operation on the third latent space eigenvector and the fourth latent space eigenvector to determine a fifth latent space eigenvector.

[0074] In some embodiments of the present application, the third latent space feature vector and the fourth latent space feature vector are ANDed to create a comprehensive representation to obtain a fifth latent space vector.

[0075] S1066B. Process the fifth latent space feature vector and posture feature information based on the pre-trained diffusion model to obtain a sixth latent space vector.

[0076] S1066C. Decode the sixth latent space feature vector to obtain the target person image.

[0077] In some embodiments of the present application, after obtaining the fifth latent space vector, the fifth latent space vector and posture feature information are subjected to traditional module processing to obtain a combined latent space vector, i.e., the sixth latent space vector, and finally the sixth latent space vector is decoded to obtain the target person image.

[0078] It can be understood that in some embodiments of the present invention, the third latent space feature vector and the fourth latent space feature vector are summed to determine the fifth latent space feature vector, the fifth latent space feature vector and the posture feature information are processed based on the pre-trained diffusion model to obtain the sixth latent space vector, and the sixth latent space feature vector is decoded to obtain the target image. In this process, the fifth latent space feature vector has integrated the image feature information and the text feature information, introduced the posture feature information, and used the pre-trained diffusion model to process the posture feature information and the fifth latent space feature vector, and finally decoded the sixth latent space feature vector to obtain the target image, which not only enables the specific area where the clothes are separated in the second image to redraw the image of the clothes to be tried on in combination with the description information of the clothes and the binary mask map, but also can obtain a more realistic and natural target character map.

[0079] Combine the following Figure 1 and Figure 3 The model training method provided in the embodiment of the present application is introduced. Figure 3 As shown, Figure 3 It is a flowchart of a model training provided in an embodiment of the present application.

[0080] like Figure 3 As shown, the model training method includes:

[0081] S201. Acquire a clothing sample image and a first image; wherein, in the first image, a first person wears the clothing sample.

[0082] In an embodiment of the present application, during the model training phase, clothing sample images are collected and the model is trained using the clothing sample images. The clothing samples may be clothing that needs to be displayed to users or sold, or clothing that is only used to train the model and is not displayed to users. The first person may be a real-life model for displaying clothing, or a simulated model for displaying clothing, or a digital model for displaying clothing, or a retained image of a customer who has purchased clothing, etc.; the clothing sample worn by the first person is the same as the acquired clothing sample, that is, during the model training phase, it is necessary to input a pair of clothing sample images and an image of the first person wearing the clothing sample. The clothing sample images in the model training phase are equivalent to the images of clothing to be tried on in the model reasoning phase.

[0083] S202: Mask the clothing samples in the first image to obtain a binary mask image of the clothing samples and a second image after the clothing samples are masked.

[0084] In the embodiment of the present application, the binary mask image is used to accurately separate the outline of the clothing from the background, so that the character can "try on" different clothing in a digital environment without actually wearing them. The second image is the background image for redrawing the clothing image, that is, the clothing to be tried on can be drawn in the second image.

[0085] By using open source semantic segmentation technology, such as SAM, the clothing sample worn by the first person in the first image can be stripped to obtain a binary mask image of the clothing sample and a second image after the clothing sample is blocked.

[0086] S203: Input the first image into a character posture analysis model to obtain posture feature information of the first character.

[0087] S204: Input the first clothing sample image into the clothing analysis model to obtain clothing feature information.

[0088] S205: Input the obtained description information of the clothing sample into the text analysis model to obtain semantic feature information.

[0089] In the embodiment of the present application, S203 is similar to the implementation method of S103 regarding obtaining posture feature information, S204 is similar to the implementation method of S104 regarding obtaining clothing feature information, and S205 is similar to the implementation method of S105 regarding obtaining semantic feature information, which will not be described in detail here.

[0090] During the training phase, for clothing samples without description information, a large model can be used to process the clothing samples to obtain description information of the clothing samples.

[0091] S206. Taking the posture feature information as the conditional input side of the diffusion model to be trained, and taking the second image, the binary mask map, the clothing feature information and the semantic feature information as the main input side of the diffusion model to be trained to determine the noise loss and the pixel level loss; and training the diffusion model to be trained by the noise loss and the pixel level loss to obtain the trained diffusion model.

[0092] The loss function is a function used to quantify the difference between the model's prediction and the actual target value. The training goal of the model is to minimize this loss function by adjusting the model's parameters so that the model's prediction is as close to the actual output as possible.

[0093] In an embodiment of the present application, the posture feature information is used as the conditional input side of the diffusion model to be trained, and the second image, the binary mask map, the clothing feature information and the semantic feature information are used as the main input side of the diffusion model to be trained, and the predicted noise and the predicted image are obtained. The noise loss and the pixel-level loss are determined based on the predicted noise and the predicted image. Finally, the diffusion model to be trained is trained by the noise loss and the pixel-level loss to obtain the trained diffusion model.

[0094] It can be understood that in an embodiment of the present application, based on obtaining the posture feature information, clothing feature information and semantic feature information of the person from the acquired clothing sample image and the first image respectively, the posture feature information is then input as a conditional constraint into the conditional input side of the diffusion model, and the second image, the binary mask map, the clothing feature information and the semantic feature information are used as the main input side of the diffusion model to be trained to determine the noise loss and pixel-level loss, and using multiple loss values ​​for training can improve the performance and generalization ability of the model.

[0095] In some embodiments of the present application, S206 can be implemented through S2061 to S2063, which is specifically described through the following steps.

[0096] S2061. Add noise to the binary mask image and the two images to obtain an image after noise addition.

[0097] In some embodiments of the present application, in order to enhance the generalization ability of the model, improve robustness and perform data enhancement, noise is added to the binary mask image and the second image to obtain an image after noise addition. When adding noise, random latent space noise is generated, the second image is processed using a pre-trained encoder to obtain a latent space feature vector, and the latent space feature vector, the binary mask image and the randomly generated noise can be spliced ​​to obtain an image after noise addition.

[0098] S2062: obtaining a predicted image and predicted noise based on the image after noise addition, clothing feature information, and semantic feature information as the main input side of the diffusion model to be trained.

[0099] S2063. Determine noise loss based on predicted noise and real noise; and determine pixel-level loss based on the predicted image and the real image.

[0100] In some embodiments of the present application, the image after adding noise, clothing feature information and semantic feature information are used as the main input side of the diffusion model to be trained to obtain the predicted image and predicted noise, the noise loss is determined according to the error between the predicted noise and the actual noise, and the pixel-level loss is determined according to the error between the predicted image and the real image. When calculating the noise loss and the pixel-level loss, the loss function such as mean square error, cross entropy loss or absolute error loss can be used to calculate the loss.

[0101] Exemplarily, the noise loss is shown in formula (3):

[0102]

[0103] When the posture prompt model is the posture prompt ControlNet, in the above formula (3), τ θ ,∈ θ They are posture prompt ControlNet and diffusion model, c p is the latent space encoding of posture, I bg is the second image, c t is the semantic feature, c i is the image feature, is the prediction noise, t is the denoising time step, z t is the latent space encoding at time t. The text cross-attention module in the diffusion model is fixed and does not participate in training, and the rest is the trainable part. The weight of the posture prompt ControlNet model is fixed and does not participate in training. If necessary, the rest of the model can be fixed and only the posture prompt ControlNet model can be trained.

[0104] The calculation of pixel-level loss is shown in the following formula (4):

[0105]

[0106] In the above formula (4), is the pixel-level loss, I gt is the real image, I gen is the predicted image.

[0107] It can be understood that in the embodiments of the present application, the predicted image and predicted noise are obtained based on the image after noise addition, clothing feature information and semantic feature information as the main input side of the diffusion model to be trained, the noise loss is determined by minimizing the difference between the predicted noise and the actual added noise, and the pixel-level loss is determined by the predicted image and the actual image. The noise loss and pixel-level loss can accurately reflect the difference between the model prediction and the true target, and can guide the model to develop in the direction of the optimal solution, which can ensure the effectiveness, efficiency and ultimate practicality of the model training.

[0108] In some embodiments of the present application, S2063 can be implemented through S2063A to S2063B, which will be specifically explained through the following steps.

[0109] S2063A. Sum the noise loss and the pixel-level loss to obtain the total loss.

[0110] S2063B. Train the diffusion model to be trained by using the total loss to obtain a trained diffusion model.

[0111] For example, the total loss is calculated as follows (5):

[0112]

[0113] In the above formula (5), is the total loss and λ is a hyperparameter.

[0114] After calculating the total loss, the parameters of the diffusion model are adjusted using the total loss until the training conditions are met, thereby obtaining a trained diffusion model.

[0115] It is understood that in some embodiments of the present application, the noise loss and the pixel-level loss are summed to obtain the total loss, and the diffusion model to be trained is trained by the total loss to obtain the trained diffusion model. By adding the losses, a different loss function is defined for each type of data, so that the model can effectively integrate multiple information sources and improve the performance and generalization ability of the model.

[0116] In the embodiments of the present application, in combination with the above steps, Figure 4 As shown, Figure 4 This is a general flow chart of a virtual try-on method provided in an embodiment of the present application. Figure 4In the method, the image acquisition module is used to acquire an image of clothes to be tried on and a first image, in which a first person (such as a model) wears a first piece of clothing (in the training stage, the first piece of clothing worn by the first person is the same as the clothes to be tried on, and in the reasoning stage, the first piece of clothing worn by the first person is different from the clothes to be tried on), and the clothing image and the first image are input into the image processing module to obtain the clothing image, a binary mask image of the clothing, a second image in which the clothing is blocked, and a posture image of the first person. The picture and text prompts use a decoupled cross-attention mechanism. The clothing image is input into the image encoder for feature extraction, and the extracted features are input into the image cross-attention layer of the diffusion model (including the text cross-attention layer and the image cross-attention layer) after the linear layer; the second image after the clothing is occluded is processed by the VAE encoder to obtain the latent space feature vector, and the latent space feature vector, the binary mask map of the clothing and the noise are spliced ​​and input into the diffusion model; the description information of the clothing, that is, the text prompt words, is input into the text encoder for feature extraction, and the extracted features are input into the text cross-attention layer in the diffusion model; the posture image is input into the VAE encoder for feature extraction, and the extracted human posture features are input into the diffusion model through the residual connection, and the input data is processed in the diffusion model to obtain the target image.

[0117] Reference Figure 5 , shows a schematic diagram of the structure of an electronic device according to an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device.

[0118] like Figure 5 As shown, the electronic device may include: a processor (processor) 502, a communication interface (Communications Interface 504, a memory (memory) 506, and a communication bus 508.

[0119] in:

[0120] The processor 502 , the communication interface 504 , and the memory 506 communicate with each other via a communication bus 508 .

[0121] The communication interface 504 is used to communicate with other electronic devices or servers.

[0122] The processor 502 is used to execute the program 510, and specifically can execute the relevant steps in the above method embodiment.

[0123] Specifically, the program 510 may include program codes, which include computer operation instructions.

[0124] The processor 502 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0125] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0126] The program 510 may be specifically used to enable the processor 502 to execute operations corresponding to the methods described in the above method embodiments.

[0127] The specific implementation of each step in program 510 can refer to the corresponding description of the corresponding steps and units in the above method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above method embodiment, which will not be repeated here.

[0128] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0129] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.

[0130] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0131] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. A virtual try-on method, It is characterized in that include: Acquire an image of clothing to be tried on, description information of the clothing to be tried on, and a first image; wherein in the first image, a first person wears a first clothing; Separating the first clothing in the first image to obtain a binary mask image of the first clothing and a second image after the first clothing is blocked; Inputting the first image into a character posture analysis model to obtain posture feature information of the first character; Inputting the image of the clothing to be tried on into a clothing analysis model to obtain clothing feature information of the clothing to be tried on; Inputting the description information into a text analysis model to obtain semantic feature information; The posture feature information is used as the conditional input side of the pre-trained diffusion model, and the second image, the binary mask map, the clothing feature information and the semantic feature information are used as the main input side of the pre-trained diffusion model to obtain a target image of the first person wearing the clothing to be tried on.

2. The method according to claim 1, It is characterized in that The pre-trained diffusion model includes a text cross-attention layer and an image cross-attention layer; The step of using the posture feature information as a conditional input of the pre-trained diffusion model, and using the second image, the binary mask image, the clothing feature information, and the semantic feature information as a main input of the pre-trained diffusion model to obtain a target image of the first person wearing the clothing to be tried on, comprises: Inputting the second image into a pre-trained encoder to obtain a first latent space feature vector; splicing the first latent space feature vector, the latent space noise and the binary mask image according to a preset channel to obtain a spliced ​​latent space feature vector; Perform feature extraction based on the concatenated latent space feature vector to obtain a second latent space feature vector; Determining a third latent space feature vector based on the second latent space feature vector, the semantic feature information and the text cross attention layer; Determining a fourth latent space feature vector based on the second latent space feature vector, the clothing feature information, and the image cross attention layer; The target image is determined based on the third latent space feature vector, the fourth latent space feature vector and the posture feature information.

3. The method according to claim 2, It is characterized in that The determining of a third latent space feature vector based on the second latent space feature vector, the semantic feature information and the text cross attention layer comprises: Inputting the semantic feature information and the second latent space feature vector into the text cross attention layer to determine a first query matrix; A weighted sum is performed based on the first query matrix to obtain the third latent space feature vector.

4. The method according to claim 2, It is characterized in that The determining of a fourth latent space feature vector based on the second latent space feature vector, the clothing feature information and the image cross attention layer comprises: Inputting the clothing feature information and the second latent space feature vector into the image cross attention layer to determine a second query matrix; A weighted sum is performed based on the second query matrix to obtain the fourth latent space eigenvector.

5. The method according to claim 2, It is characterized in that The determining the target image based on the third latent space feature vector, the fourth latent space feature vector and the posture feature information comprises: Performing an AND operation on the third latent space eigenvector and the fourth latent space eigenvector to determine a fifth latent space eigenvector; Processing the fifth latent space feature vector and the posture feature information based on the pre-trained diffusion model to obtain a sixth latent space vector; The sixth latent space feature vector is decoded to obtain the target image.

6. A model training method, It is characterized in that include: Acquire a clothing sample image and a first image; wherein, in the first image, a first person wears the clothing sample; Masking the clothing sample in the first image to obtain a binary mask image of the clothing sample and a second image after the clothing sample is masked; Inputting the first image into a character posture analysis model to obtain posture feature information of the first character; Inputting the clothing sample image into a clothing analysis model to obtain clothing feature information; Inputting the obtained description information of the clothing sample into a text analysis model to obtain semantic feature information; The posture feature information is used as the conditional input side of the diffusion model to be trained, and the second image, the binary mask map, the clothing feature information and the semantic feature information are used as the main input side of the diffusion model to be trained to determine the noise loss and the pixel level loss; and the diffusion model to be trained is trained by the noise loss and the pixel level loss to obtain a trained diffusion model.

7. The method according to claim 6, It is characterized in that The step of using the second image, the binary mask image, the clothing feature information, and the semantic feature information as the main input side of the diffusion model to be trained to determine the noise loss and the pixel level loss includes: Adding noise to the binary mask image and the second image to obtain an image after the noise is added; Obtaining a predicted image and predicted noise based on the image after the noise is added, the clothing feature information and the semantic feature information as the main input side of the diffusion model to be trained; The noise loss is determined based on the predicted noise and the real noise; and the pixel level loss is determined according to the predicted image and the real image.

8. The method according to claim 6, It is characterized in that The step of training the diffusion model to be trained by using the noise loss and the pixel-level loss to obtain the trained diffusion model includes: The noise loss and the pixel level loss are summed to obtain a total loss; The diffusion model to be trained is trained using the total loss to obtain a trained diffusion model.

9. An electronic device, include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method as described in any one of claims 1-5 or any one of claims 6-8.

10. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 5 or any one of claims 6 to 8 is implemented.

Images