Virtual fitting method, model training method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a virtual fitting method, a model training method, electronic equipment and a storage medium. The virtual fitting method comprises the following steps: acquiring an image of to-be-fitted clothes and a first image; wherein in the first image, the first person holds the first clothes; separating the first clothes in the first image to obtain a binary mask graph of the first clothes and a second image after the first clothes are shielded; performing posture extraction based on the first image and a figure posture analysis model to obtain posture feature information of the first figure; inputting the clothes image into a clothes detail analysis model and a clothes appearance analysis model respectively to obtain detail feature information and appearance feature information of the clothes; and taking the attitude feature information added with the hidden noise as a condition input side of a pre-trained diffusion model, and taking the binary mask graph, the second image, the detail feature information and the appearance feature information as a main input side of the pre-trained diffusion model, thereby obtaining a target image of the to-be-tried clothes. According to the invention, the virtual fitting effect is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of visual processing technology, and in particular to a virtual try-on method, a model training method, an electronic device, and a storage medium. Background Art

[0002] In the e-commerce clothing industry, the visual presentation of goods is crucial to sales. Although traditional flat clothing images are sufficient, images presented by models are usually more attractive to customers. However, producing a large number of model display images is time-consuming and laborious. To solve this problem, virtual try-on technology came into being, which can not only enhance consumers' shopping experience, but also effectively reduce the operating costs of clothing retailers.

[0003] In the related art, an image of a given person trying on the clothing is generated based on a given person image and clothing image. Since clothing is not a rigid body, it will deform when worn on a person. Therefore, the related virtual try-on methods often have difficulty in processing complex clothing textures or complex human postures, resulting in the generated image not being realistic enough.

[0004] Therefore, how to generate high-quality virtual try-on effects becomes an urgent problem to be solved. Summary of the invention

[0005] In view of this, embodiments of the present application provide a virtual try-on method, a model training method, an electronic device and a storage medium to at least partially solve the above problems.

[0006] According to a first aspect of an embodiment of the present application, a virtual try-on method is provided, comprising:

[0007] Acquire an image of clothing to be tried on and a first image; wherein, in the first image, a first person wears a first clothing; separate the first clothing in the first image to obtain a binary mask image of the first clothing and a second image after the first clothing is blocked; perform posture extraction based on the first image and a person posture analysis model to obtain posture feature information of the first person; input the image of the clothing to be tried on into a clothing detail analysis model and a clothing appearance analysis model respectively to obtain corresponding detail feature information and appearance feature information; use the posture feature information after adding hidden noise as the conditional input side of a pre-trained diffusion model, and use the binary mask image, the second image, the detail feature information and the appearance feature information as the main input side of the pre-trained diffusion model to obtain a target image of the first person wearing the clothing to be tried on.

[0008] According to a second aspect of an embodiment of the present application, a model training method is provided, comprising:

[0009] Acquire a clothing sample image and a first image; wherein, in the first image, a first person wears the clothing sample;

[0010] Masking the clothing sample in the first image to obtain a binary mask image of the clothing sample and a second image after the clothing sample is masked;

[0011] Performing posture extraction based on the first image and the character posture analysis model to obtain posture feature information of the first character;

[0012] Inputting the clothing sample images into a clothing detail analysis model to be trained and a clothing appearance analysis model to be trained respectively, to obtain clothing detail feature information and clothing appearance feature information;

[0013] The posture feature information after adding hidden noise is used as the conditional input side of the diffusion model to be trained, and the binary mask map, the second image, the detail feature information and the appearance feature information are used as the main input side of the diffusion model to be trained to determine the noise loss and the pixel level loss; and the model is trained by using the noise loss and the pixel level loss until a trained model is obtained.

[0014] According to a third aspect of an embodiment of the present application, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in the first aspect or the second aspect.

[0015] According to a fourth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect or the second aspect is implemented.

[0016] According to the scheme provided in the embodiment of the present application, an image of clothing to be tried on and a first image are obtained; wherein, in the first image, a first person wears a first clothing; the first clothing in the first image is separated to obtain a binary mask image of the first clothing and a second image after the first clothing is blocked; posture extraction is performed based on the first image and a character posture analysis model to obtain posture feature information of the first person; the image of the clothing to be tried on is respectively input into a clothing detail analysis model and a clothing appearance analysis model to obtain corresponding detail feature information and appearance feature information; the posture feature information after adding hidden noise is used as the conditional input side of a pre-trained diffusion model, and the binary mask image, the second image, the detail feature information and the appearance feature information are used as the main input side of the pre-trained diffusion model to obtain a target image of the first person wearing the clothing to be tried on. In this process, obtaining detail feature information and appearance feature information can provide clothing reference for the target image. In the specific area where the first clothing is separated in the second image, the image of the clothing to be tried is redrawn in the specific area in combination with the detail feature information, appearance feature information and binary mask map of the clothing to be tried, so that the area of ​​the newly drawn clothing image to be tried can be harmoniously integrated with the other parts of the original second image to achieve visual unity and naturalness. At the same time, the posture feature information is input as a conditional constraint to the conditional input side of the diffusion model, so that the target image finally generated can visualize the effect of the clothing to be tried on the first person, while maintaining the authenticity and naturalness of the overall image. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0018] Figure 1 A general architecture diagram of a virtual try-on method provided in an embodiment of the present application;

[0019] Figure 2 A flowchart of a virtual try-on method provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of a model training process provided in an embodiment of the present application;

[0021] Figure 4 A general flow chart of a virtual try-on method provided in an embodiment of the present application;

[0022] Figure 5 The present invention is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0024] In order to adapt the virtual try-on method provided in the embodiment of the present application, the embodiment of the present application provides a general architecture diagram of the virtual try-on method, that is, the virtual try-on method of the embodiment of the present application can be applied to the virtual try-on system provided in the embodiment of the present application. Figure 1 As shown, the virtual try-on system includes an image acquisition module, an image processing module and a target image generation module. The image acquisition module is used to acquire an image of the clothes to be tried on and a first image, in which a first person (such as a model) wears a first piece of clothing, wherein in the training stage, the acquired image of the clothes to be tried on is the same as the first piece of clothing worn by the first person; in the reasoning stage, the acquired image of the clothes to be tried on and the first piece of clothing worn by the first person may be different. The image processing module is used to process the image of the clothes to be tried on and the first image, specifically by using open source semantic segmentation technology, such as the Segment Anything Model (SAM), the first piece of clothing worn by the first person can be separated in the first image, and then a binary mask image of the first piece of clothing and a second image after the first piece of clothing is blocked, i.e., the background image of the redrawn image are obtained, and an open source human posture extraction model is used to extract the posture image of the first person from the first image. The target image generation module specifically includes: extracting detail feature information and appearance feature information of the image of the clothing to be tried on respectively through the clothing detail analysis model and the clothing appearance analysis model in the clothing reference model (for example, comparing the language-image pre-trained image encoder); using an open source human posture analysis model to process the posture image to obtain posture feature information, and inputting the posture feature information, the extracted detail feature information, the appearance feature information, the binary mask map and the second image into the diffusion model for processing to obtain the target image.

[0025] Combine the following Figure 1 and Figure 2 The virtual try-on method provided by the embodiment of the present application is introduced. Figure 2 As shown, Figure 2 This is a flow chart of a virtual try-on method provided in an embodiment of the present application. The virtual try-on method provided in an embodiment of the present application can be executed by an electronic device, and the electronic device can be, for example, a computer, a server, etc.

[0026] like Figure 2 As shown, the virtual try-on method comprises:

[0027] S101, obtaining an image of clothing to be tried on and a first image; wherein, in the first image, a first person wears a first clothing.

[0028] In an embodiment of the present application, the clothes to be tried on in the image of clothes to be tried on may be clothes that need to be displayed or sold to the user, and the first image is an image of a first person wearing the first clothes, wherein the first person may be a real-life model for displaying clothes, or a simulated model for displaying clothes, or a digital model for displaying clothes, or a customer who buys clothes, etc.; the first image may be one or multiple images. When there are multiple first images, the first persons in the first images may be different, and the same clothes to be tried on may be tried on by multiple first persons with different body shapes, so that the user can make an appropriate choice according to his or her body shape.

[0029] In an embodiment of the present application, the first clothing may be different from the clothing to be tried on.

[0030] S102: Separate the first clothing in the first image to obtain a binary mask image of the first clothing and a second image after the first clothing is blocked.

[0031] In the embodiment of the present application, the binary mask image is used to accurately separate the outline of the clothing from the background, so that the character can "try on" different clothing in a digital environment without actually wearing them. The second image is the background image for redrawing the clothing image, that is, the clothing to be tried on can be drawn in the area where the first clothing is separated in the second image.

[0032] By using open source semantic segmentation technology, such as SAM, the first clothing on the first person can be separated in the first image, thereby obtaining a binary mask image of the first clothing and a second image after the first clothing is blocked.

[0033] S103: Perform posture extraction based on the first image and the character posture analysis model to obtain posture feature information of the first character.

[0034] In the first image, after the original first clothing of the first person is stripped off, when the clothing to be tried on is re-drawn on the area where the first clothing is stripped off, it is necessary to ensure that the face, hairstyle and body movements of the first person remain in their original state. In this way, the target image finally generated can visualize the effect of the clothing to be tried on the first person, while maintaining the authenticity and naturalness of the overall image, so it involves the acquisition of the person's posture information.

[0035] Posture feature information involves the position, direction and movement of various parts of the body, and how these factors are combined to express the individual's emotions, intentions and personality. It includes but is not limited to head posture feature information, torso feature information, facial expression feature information, and micro-expression and micro-movement feature information.

[0036] In an embodiment of the present application, the first image can be input into a character posture extraction model obtained from open source to obtain a posture image, and the posture image can be further input into a posture analysis model (e.g., a posture encoder) to obtain posture feature information of the first character. In order to enhance the diversity of data and the robustness of the model, the initial latent space noise can be determined according to the determined noise type and noise size, or the initial latent space noise can be randomly generated, and the initial latent space noise can be added to the extracted posture feature information, such as adding a determined or randomly generated latent space Gaussian noise to the extracted posture feature information, to obtain the posture feature information after adding the latent noise, which can be subsequently input into the pre-trained diffusion model.

[0037] S104: Input the image of the clothing to be tried on into the clothing detail analysis model and the clothing appearance analysis model respectively to obtain corresponding detail feature information and appearance feature information.

[0038] Clothes’ detailed information refers to the subtle but important elements that make up the design, production and wearing experience of clothing. These details are not only related to the appearance of the clothing, but also directly affect the comfort, durability and overall quality of wearing, including but not limited to fabric selection and texture, tailoring and design, decorative elements, functional details, lining and accessories. Clothes’ appearance information refers to all elements and features that the clothing visually presents, including but not limited to color and pattern, style and outline, material gloss and texture, matching and overall effect, etc.

[0039] In an embodiment of the present application, an image of clothing to be tried on is input into a clothing detail analysis model to obtain detail feature information, and at the same time, the image of clothing to be tried on is input into a clothing appearance analysis model to obtain appearance feature information. The appearance feature information and detail feature information can provide a reference for the generation of subsequent target images.

[0040] The clothing appearance analysis model may be a contrastive language-image pre-training model (CLIP), and the clothing detail analysis model may be obtained by training based on a latent space diffusion model StableDiffusion.

[0041] Stable Diffusion is a classic latent diffusion model (LDM). LDM uses a variational autoencoder. Given an input image, the encoder first maps the image to the latent space, so that diffusion is performed in the latent space. In the forward process of diffusion, Gaussian noise is gradually added to the latent variable in T iterations. The reverse diffusion process is to gradually recover from to, and finally pass through the decoder to get the final image.

[0042] S105. Using the posture feature information after adding hidden noise as the conditional input side of the pre-trained diffusion model, and using the binary mask image, the second image, the detail feature information and the appearance feature information as the main input side of the pre-trained diffusion model, to obtain a target image of the first person wearing the clothes to be tried on.

[0043] Pre-trained diffusion models allow the model to generate data based on additional input information (conditions), which can be a category label, a text description, another image, or any form of guiding signal, with the goal of guiding the generation process so that the generated results meet specific conditions or properties.

[0044] In an embodiment of the present application, the posture feature information after adding hidden noise is used as the conditional input side of the pre-trained diffusion model, the binary mask map, the second image, the detail feature information and the appearance feature information are used as the main input side of the pre-trained diffusion model, and the posture information of the first person is used as an additional input condition of the pre-trained diffusion model for guidance, so as to obtain a target image of the first person wearing the clothes to be tried on.

[0045] It can be understood that in the embodiment of the present application, an image of the clothing to be tried on and a first image are obtained; wherein, in the first image, the first person wears the first clothing; the first clothing in the first image is separated to obtain a binary mask image of the first clothing and a second image after the first clothing is blocked; posture extraction is performed based on the first image and the character posture analysis model to obtain the posture feature information of the first person; the image of the clothing to be tried on is input into the clothing detail analysis model and the clothing appearance analysis model respectively to obtain the corresponding detail feature information and appearance feature information; the posture feature information after adding hidden noise is used as the conditional input side of the pre-trained diffusion model, and the binary mask image, the second image, the detail feature information and the appearance feature information are used as the main input side of the pre-trained diffusion model to obtain the target image of the first person wearing the clothing to be tried on. In this process, the detail feature information and the appearance feature information can be obtained in the specific area where the first clothing is separated in the second image to provide clothing reference for the target image. Combined with the detail feature information, appearance feature information and binary mask map of the clothes to be tried on, the clothes to be tried on image is redrawn in a specific area, so that the newly drawn clothes to be tried on image area can be harmoniously integrated with other parts of the original second image to achieve visual unity and naturalness. At the same time, the posture feature information is input as a conditional constraint to the conditional input side of the diffusion model, so that the target image finally generated can visualize the effect of the clothes to be tried on the person, while maintaining the authenticity and naturalness of the overall image.

[0046] In some embodiments of the present application, the pre-trained diffusion model includes a self-attention layer and a cross-attention layer. In S105, the posture feature information after adding hidden noise is used as the conditional input side of the pre-trained diffusion model, and the binary mask map, the second image, the detail feature information and the appearance feature information are used as the main input side of the pre-trained diffusion model. Obtaining the target image of the first person wearing the clothes to be tried on can be achieved through S1051 to S1054, which will be explained through the following steps.

[0047] S1051. Determine a re-weighted first latent space feature vector based on the binary mask image, the posture feature information, the second image, and the self-attention layer.

[0048] S1052. Input the first latent space feature vector and the appearance feature information into the cross attention layer to obtain a first query matrix, a first key matrix and a first value matrix corresponding to the first latent space feature vector.

[0049] The query matrix is ​​the starting point of the search or query in the attention layer. The input at each position will generate a corresponding query matrix, which is used to calculate the similarity with the key matrix of all positions to determine which information at the current position should focus on (or "pay attention to"). The key matrix is ​​the target to be queried or retrieved in the attention layer. The input at each position will also generate a corresponding key matrix, which is used to calculate the similarity with the query matrix to help the model understand the relationship between different positions in the sequence. The value matrix is ​​the vector that actually contains the information in the attention layer. The input at each position will also generate a corresponding value matrix. It contains specific information for each position in the sequence, which will be weighted and summed according to the attention weights calculated by the query matrix and the key matrix.

[0050] The main function of the cross attention layer is to process the dependency between two different input data, thereby achieving interaction and fusion between data.

[0051] In some embodiments of the present application, the first key matrix is ​​determined by appearance feature information and a second preset weight matrix, the first value matrix is ​​determined by a third preset weight matrix and appearance feature information, and the first query matrix is ​​determined by a first latent space feature vector and the first preset weight matrix.

[0052] Exemplarily, Q=ZQ q ,K=c t W k ,V=c t W v are the first query matrix, the first key matrix, and the first value matrix, respectively, where W q ,W k ,W v is a trainable linear mapping layer, which is respectively a first preset weight matrix, a second preset weight matrix and a third preset weight matrix, that is, the first preset weight matrix, the second preset weight matrix and the third preset weight matrix are automatically learned through the training process of the diffusion model. During the training process, the goal of the diffusion model is to minimize a certain loss function (such as cross entropy loss), which usually involves adjusting the diffusion model parameters, including W q ,W k ,W v When the training is completed, the first preset weight matrix, the second preset weight matrix and the third preset weight matrix will be saved for use in model inference. t is the appearance feature information, and Z is the first latent space feature vector.

[0053] S1053. Input the detail feature information into the cross attention layer to determine the second query matrix, the second key matrix and the second value matrix.

[0054] In some embodiments of the present application, the detail feature information is input into the cross-attention layer and mapped to three different vector spaces through three different linear transformation matrices (weight matrices) to obtain a second query matrix, a second key matrix and a second value matrix.

[0055] S1054. Determine a target image by performing matrix fusion based on the first key matrix, the first value matrix, the second key matrix, and the second value matrix.

[0056] Matrix fusion is to integrate the results of multiple data sources to obtain more accurate and comprehensive information. Each data source can be regarded as a matrix, and different matrices can be fused by element multiplication or element addition. Matrix fusion can improve the accuracy, comprehensiveness and robustness of information.

[0057] For example, as shown in the following formula (1):

[0058]

[0059] In the above formula (1), cat(V θ ,V g ) or cat(K θ ,K g ) is equivalent to V θ ,V g or K θ ,K g The two matrices are fused separately, Q θ is the first query matrix, d k is the dimension of the key matrix.

[0060] It can be understood that in some embodiments of the present application, the re-weighted first latent space feature vector is determined based on the binary mask map, posture feature information, the second image and the self-attention layer, and the first latent space feature vector and the appearance feature information are input into the cross-attention layer to obtain the first query matrix, the first key matrix and the first value matrix corresponding to the first latent space feature vector, so as to achieve the interaction and fusion of the relevant information of the clothing and the relevant information of the posture. The detail feature information is input into the cross-attention layer to determine the second query matrix, the second key matrix and the second value matrix, and the target image is determined by matrix fusion based on the first key matrix, the first value matrix, the second key matrix and the second value matrix. By fusing different matrices, the accuracy, comprehensiveness and robustness of the information are improved, and the generation effect of the target image is improved.

[0061] In some embodiments of the present application, S1054 may be implemented through S1054A to S1054D, which will be explained through the following steps.

[0062] S1054A. Merge the first value matrix and the second value matrix to obtain a target value matrix.

[0063] S1054B, merge the first bond matrix and the second bond matrix to obtain a target bond matrix.

[0064] In some embodiments of the present application, two value matrices or two key matrices contain different information or emphasize different aspects. Through matrix fusion, the complementarity of this information can be fully utilized and the bias or information loss that may be caused by a single matrix can be reduced.

[0065] For example, in formula (1), cat(V θ ,V g ) is to fuse the first bond matrix and the second bond matrix, where V V is the second bond matrix, V θ is the first bond matrix. cat(K θ ,K g ) is to fuse the first value matrix and the second value matrix, where K g is the second value matrix, V θ is the first value matrix.

[0066] S1054C. Based on the first query matrix, the target value matrix and the target key matrix are weighted and summed to obtain a re-weighted second latent space feature vector.

[0067] In some embodiments of the present application, as described in formula (1), after calculating the attention score based on the first query matrix and the target key matrix, the obtained attention score is normalized by the softmax function so that the sum of all attention scores corresponding to each query is 1, thereby obtaining a weight vector, and using the normalized attention weights to perform weighted summation on the target value matrix to obtain a context representation of each query. This context vector contains the weighted information of all elements in the sequence for the current query, and is a comprehensive representation of the information at different positions in the input feature vector. Finally, the re-weighted second latent space feature vector is obtained.

[0068] S1054D. Based on the second latent space feature vector and the preset decoder, obtain the target image.

[0069] The preset decoder can be a decoder of a variational autoencoder (VAE). The role of the decoder in the VAE is to convert the latent vector sampled from the latent space back to the original data space, thereby generating reconstructed data that is as close as possible to the original input data while maintaining the key features of the data.

[0070] In some embodiments of the present application, after the second latent space feature vector is obtained, it is decoded and reconstructed by a preset decoder to obtain a target image.

[0071] It is understandable that in some embodiments of the present application, the first value matrix and the second value matrix are fused to obtain a target value matrix, and the first key matrix and the second key matrix are fused to obtain a target key matrix. Through matrix fusion, the complementarity of this information can be fully utilized to reduce the bias or information loss that may be caused by a single matrix. Based on the second latent space feature vector and the preset decoder, the target image is obtained. This reconstruction capability enables the preset decoder to learn the intrinsic representation of the data and capture the main features of the data, thereby improving the generation effect of the target image.

[0072] In some embodiments of the present application, determining the re-weighted first latent space feature vector based on the binary mask map, posture feature information, the second image and the self-attention layer in S1051 can be implemented through S1051A~S1051B, which will be specifically explained through the following steps.

[0073] S1051A, performing channel splicing on the binary mask image, the posture feature information, and the second image to obtain a spliced ​​latent space feature vector.

[0074] Channel concatenation allows feature vectors of different levels to be combined, so that both high-level abstract features and low-level detailed features can be utilized. By increasing the number of channels, the model can learn more feature representations, which helps improve the model's expressiveness and discrimination. The pose feature information has been added with noise, and by concatenating the pose feature information with latent space noise, the model's generalization ability, robustness, and data enhancement can be enhanced.

[0075] In some embodiments of the present application, the second image is input into a preset encoder (which may be a VAE encoder) to obtain a latent space feature vector of a 4-channel redrawn background. The latent space feature vector of the 4-channel redrawn background, the posture feature information with the 4-channel latent space noise added, and the 1-channel binary mask map are spliced ​​along the channel to obtain a 9-channel input, that is, to obtain a spliced ​​latent space feature vector.

[0076] S1051B. Perform weighted summation based on the concatenated latent space feature vector and the self-attention layer to obtain a re-weighted first latent space vector.

[0077] In some embodiments of the present application, the self-attention layer can enhance the model's ability to understand and model the dependencies between features by calculating the correlation between feature vectors. The concatenated latent space feature vector is first processed by other layers of the pre-trained diffusion model, and then input into the self-attention layer, and then the query matrix, key matrix, and value matrix are generated, the attention score is calculated, the softmax function is applied, and the weighted summation is performed to obtain the re-weighted first latent space feature vector.

[0078] It is understandable that in some embodiments of the present application, the binary mask map, the posture feature information, and the second image are channel-joined to obtain the joint latent space feature vector. By increasing the number of channels, the network can learn more feature representations, which helps to improve the expressiveness and discrimination of the model. Based on the joint latent space feature vector input to the self-attention layer for weighted summation, the re-weighted first latent space vector is obtained. By calculating the correlation between the feature vectors, the model's understanding and modeling capabilities of the dependency relationship between features are enhanced.

[0079] Combine the following Figure 1 and Figure 3 The model training method provided in the embodiment of the present application is introduced. Figure 3 As shown, Figure 3 A schematic diagram of a model training process provided in an embodiment of the present application.

[0080] like Figure 3 As shown, the model training method includes:

[0081] S201. Acquire a clothing sample image and a first image; wherein, in the first image, a first person wears the clothing sample.

[0082] In an embodiment of the present application, in the model training stage, clothing sample images are collected and the model is trained using the clothing sample images. The clothing samples may be clothing that needs to be displayed to users or sold, or clothing that is only used to train the model and is not displayed to users. The first person may be a real-life model for displaying clothing, or a simulated model for displaying clothing, or a digital model for displaying clothing, or a retained image of a customer who has purchased clothing, etc.; in the first image, the clothing sample worn by the first person is the same as the acquired clothing sample, that is, in the model training stage, it is necessary to input a pair of clothing sample images and an image of the first person wearing the clothing sample. The clothing sample images in the model training stage are equivalent to the images of clothing to be tried on in the model reasoning stage.

[0083] S202: Mask the clothing samples in the first image to obtain a binary mask image of the clothing samples and a second image after the clothing samples are masked.

[0084] In the embodiment of the present application, the binary mask image is used to accurately separate the outline of the clothing from the background, so that the character can "try on" different clothing in a digital environment without actually wearing them. The second image is the background image for redrawing the clothing image, that is, the clothing to be tried on can be drawn in the second image.

[0085] By using open source semantic segmentation technology, such as SAM, the clothing sample worn by the first person in the first image can be stripped to obtain a binary mask image of the clothing sample and a second image after the clothing sample is blocked.

[0086] S203: Perform posture extraction based on the first image and the character posture analysis model to obtain posture feature information of the first character.

[0087] S204: inputting the clothing sample images into the clothing detail analysis model to be trained and the clothing appearance analysis model to be trained respectively, to obtain clothing detail feature information and clothing appearance feature information.

[0088] In the embodiment of the present application, S203 is similar to S103 in terms of the implementation of obtaining posture feature information, and S204 is similar to S104 in terms of the implementation of obtaining detail feature information and appearance feature information, which will not be described in detail here.

[0089] S205. Using the posture feature information after adding hidden noise as the conditional input side of the diffusion model to be trained, and using the binary mask image, the second image, the detail feature information and the appearance feature information as the main input side of the diffusion model to be trained to determine the noise loss and the pixel-level loss; and performing model training through the noise loss and the pixel-level loss until a trained model is obtained.

[0090] The loss function is a function used to quantify the difference between the model's prediction and the actual target value. The training goal of the model is to minimize this loss function by adjusting the model's parameters so that the model's prediction is as close to the actual output as possible.

[0091] In an embodiment of the present application, the posture feature information after adding latent space noise is used as the conditional input side of the diffusion model to be trained, and the second image, the binary mask map, the detail feature information and the appearance feature information are used as the main input side of the diffusion model to be trained, and the predicted noise and the predicted image are obtained, and the noise loss and the pixel-level loss are determined based on the predicted noise and the predicted image. Finally, the model is trained by the noise loss and the pixel-level loss until the training conditions are met to obtain the trained model.

[0092] Among them, when training the model, noise loss and pixel-level loss can be used simultaneously to train the clothing detail analysis model, clothing appearance analysis model and diffusion model to be trained.

[0093] It can be understood that in the embodiments of the present application, based on respectively obtaining the posture feature information, detail feature information, appearance feature information, the second image and the binary mask map of the person from the acquired clothing sample image and the first image, the posture feature information is then input as a conditional constraint into the conditional input side of the diffusion model, and the second image, the binary mask map, the detail feature information and the appearance feature information are used as the main input side of the diffusion model to be trained to determine the noise loss and the pixel-level loss, and using multiple loss values ​​for training can improve the performance and generalization ability of the model.

[0094] In some embodiments of the present application, S205 uses the posture feature information after adding hidden noise as the conditional input side of the diffusion model to be trained, and uses the binary mask map, the second image, the detail feature information and the appearance feature information as the main input side of the diffusion model to be trained. Determining the noise loss and pixel-level loss can be achieved through S2051 to S2053, which is explained through the following steps.

[0095] S2051, splicing the posture feature information after adding hidden noise, the binary mask image and the second image to determine the spliced ​​hidden feature vector.

[0096] Exemplarily, the second image is input into a preset encoder (which may be a VAE encoder) to obtain a latent space feature vector of a 4-channel redrawn background. The latent space feature vector of the 4-channel redrawn background, the posture feature information with the 4-channel latent space noise added, and the 1-channel binary mask map are spliced ​​along the channel to obtain a 9-channel input, that is, the spliced ​​latent space feature vector.

[0097] S2052: Determine a predicted image and predicted noise based on the concatenated latent space feature vector, detail feature information, and appearance feature information.

[0098] S2053, determining noise loss based on predicted noise and real noise; and determining pixel-level loss based on predicted image and real image.

[0099] In some embodiments of the present application, a predicted image and predicted noise are obtained based on the spliced ​​latent space feature vector, detail feature information, and appearance feature information as the main input side of the diffusion model to be trained, and the noise loss is determined according to the error between the predicted noise and the actual noise, and the pixel-level loss is determined according to the error between the predicted image and the real image. When calculating the noise loss and the pixel-level loss, the loss function such as mean square error, cross entropy loss, or absolute error loss can be used to calculate the loss.

[0100] Exemplarily, the noise loss is shown in formula (2):

[0101]

[0102] In the above formula (2), τ p ,τ g ,∈ θ They are the posture analysis model, clothing detail analysis model and diffusion model, which are the trainable parts of the model. θ It is a pre-trained CLIP image encoder and does not participate in training. is the prediction noise, t is the denoising time step, z t is the latent space code at time t, I pose is the pose image, I g For the clothes to be tried on, I m is a binary mask image, and g represents the clothing detail analysis model.

[0103] The calculation of pixel-level loss is shown in the following formula (3):

[0104]

[0105] In the above formula (3), is the pixel-level loss, I gt is the real image, I gen is the predicted image.

[0106] It can be understood that in the embodiments of the present application, based on the posture feature information after adding hidden noise, the binary mask map and the second image as the main input side of the diffusion model to be trained, the predicted image and predicted noise are obtained, the noise loss is determined by minimizing the difference between the predicted noise and the actual added noise, and the pixel-level loss is determined by the predicted image and the actual image. The noise loss and pixel-level loss can accurately reflect the difference between the model prediction and the real target, and can guide the model to develop in the direction of the optimal solution, which can ensure the effectiveness, efficiency and ultimate practicality of the model training.

[0107] In some embodiments of the present application, S205 performs model training through noise loss and pixel level until a trained model is obtained, which can be implemented through S205A to S205B, as illustrated by the following steps.

[0108] S205A, sum the noise loss and the pixel level loss to obtain the total loss.

[0109] S205B, respectively training the character posture analysis model to be trained, the clothing detail analysis model to be trained, and the diffusion model to be trained by using the total loss, until a pre-trained clothing detail analysis model, a pre-trained character posture analysis model, and a pre-trained diffusion model are respectively obtained.

[0110] For example, the total loss is calculated as follows (4):

[0111]

[0112] In the above formula (4), is the total loss and λ is a hyperparameter.

[0113] After calculating the total loss, the total loss is used to train the character posture analysis model to be trained, the clothing detail analysis model to be trained, and the diffusion model to be trained simultaneously until the training conditions are met, thereby obtaining the pre-trained clothing detail analysis model, the pre-trained character posture analysis model, and the pre-trained diffusion model.

[0114] It is understandable that in some embodiments of the present application, the noise loss and the pixel-level loss are summed to obtain the total loss, and the diffusion model to be trained, the clothing detail analysis model to be trained, and the clothing posture analysis model to be trained are trained by the total loss to obtain the trained model. By adding the losses, different loss functions are defined for each type of data, so that the model can effectively integrate multiple information sources and improve the performance and generalization ability of the model.

[0115] In the embodiments of the present application, in combination with the above steps, Figure 4 As shown, Figure 4 This is a general flow chart of a virtual try-on method provided in an embodiment of the present application. Figure 4In the method, the image acquisition module is used to acquire an image of the clothes to be tried on and a first image, in which a first person (such as a model) wears a first piece of clothing (in the training stage, the first piece of clothing on the first person is the same as the clothes to be tried on, and in the inference stage, the first piece of clothing on the first person is different from the clothes to be tried on). The image of the clothes to be tried on and the first image are input into the image processing module to obtain a binary mask image of the first piece of clothing, a second image of the first piece of clothing being blocked, and a posture image of the first person. First, the image of the clothes to be tried on is processed by the VAE encoder and input into the clothing detail analysis model. At the same time, the image of the clothes to be tried on is input into the clothing appearance analysis model (CLIP image encoder). The clothing detail analysis model can keep the details of the clothes to be tried on. The clothing appearance analysis model can retain the overall color and texture of the clothes to be tried on, and then the detail feature information and appearance feature information are input into the diffusion model. The acquired posture image is input into the posture analysis model to obtain the posture feature information of the first person as an additional posture control of the first person. The second image is input into the VAE encoder to obtain the latent space feature vector. The latent space feature vector, the posture feature information with added noise and the binary mask map are spliced ​​to obtain the spliced ​​latent space feature vector. The spliced ​​feature vector is input into the diffusion model, first passes through the self-attention layer to obtain the re-weighted first latent space feature vector, the first latent space feature vector and the appearance feature information are input into the cross-attention layer to determine the query matrix, key matrix and value matrix (described in detail above), the determined key matrix and value matrix are respectively fused with the value matrix and key matrix corresponding to the detail feature information, and finally the re-weighted second latent space feature vector is obtained. The VAE decoder is used to reconstruct the data of the second latent space feature vector to obtain the target image.

[0116] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method provided in any of the above embodiments when executing the program.

[0117] The present application also provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the method provided in any of the above embodiments is implemented.

[0118] Reference Figure 5 , shows a schematic diagram of the structure of an electronic device according to an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device.

[0119] like Figure 5As shown, the electronic device may include: a processor (processor) 1002 , a communication interface (Communications Interface) 1004 , a memory (memory) 1006 , and a communication bus 1008 .

[0120] in:

[0121] The processor 1002 , the communication interface 1004 , and the memory 1006 communicate with each other via a communication bus 1008 .

[0122] The communication interface 1004 is used to communicate with other electronic devices or servers.

[0123] The processor 1002 is used to execute the program 1010, and specifically can execute the relevant steps in the above method embodiment.

[0124] Specifically, the program 1010 may include program codes, which include computer operation instructions.

[0125] The processor 1002 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0126] The memory 1006 is used to store the program 1010. The memory 1006 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0127] The program 1010 may be specifically used to enable the processor 1002 to execute operations corresponding to the methods described in the above method embodiments.

[0128] The specific implementation of each step in program 1010 can refer to the corresponding description of the corresponding steps and units in the above method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above method embodiment, which will not be repeated here.

[0129] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0130] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.

[0131] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0132] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. A virtual try-on method, characterized in that: include: Acquire an image of clothing to be tried on and a first image; wherein in the first image, a first person wears a first clothing; Separating the first clothing in the first image to obtain a binary mask image of the first clothing and a second image after the first clothing is blocked; Performing posture extraction based on the first image and a character posture analysis model to obtain posture feature information of the first character; Inputting the image of the clothing to be tried on into a clothing detail analysis model and a clothing appearance analysis model respectively to obtain corresponding detail feature information and appearance feature information; The posture feature information after adding hidden noise is used as the conditional input side of the pre-trained diffusion model, and the binary mask map, the second image, the detail feature information and the appearance feature information are used as the main input side of the pre-trained diffusion model to obtain a target image of the first person wearing the clothes to be tried on.

2. The method according to claim 1, characterized in that: The pre-trained diffusion model includes a cross-attention layer and a self-attention layer; The method uses the posture feature information after adding hidden noise as the conditional input side of the pre-trained diffusion model, and uses the binary mask image, the second image, the detail feature information and the appearance feature information as the main input side of the pre-trained diffusion model to obtain a target image of the first person wearing the clothing to be tried on, including: Determine a reweighted first latent space feature vector based on the binary mask map, the posture feature information, the second image, and the self-attention layer; Inputting the first latent space feature vector and the appearance feature information into the cross attention layer to obtain a first query matrix, a first key matrix and a first value matrix corresponding to the first latent space feature vector; Inputting the detail feature information into the cross attention layer to determine a second query matrix, a second key matrix, and a second value matrix; The target image is determined by performing matrix fusion based on the first key matrix, the first value matrix, the second key matrix and the second value matrix.

3. The method according to claim 2, characterized in that The determining the target image by performing matrix fusion based on the first key matrix, the first value matrix, the second key matrix and the second value matrix includes: Merging the first value matrix and the second value matrix to obtain a target value matrix; Merging the first bond matrix and the second bond matrix to obtain a target bond matrix; Based on the first query matrix, the target value matrix and the target key matrix are weighted and summed to obtain a re-weighted second latent space feature vector; Based on the second latent space feature vector and a preset decoder, the target image is obtained.

4. The method according to claim 2, characterized in that: The first query matrix is ​​determined by the first latent space feature vector and a first preset weight matrix; the first key matrix is ​​determined by the appearance feature information and a second preset weight matrix; and the first value matrix is ​​determined by the appearance feature information and a third preset matrix.

5. The method according to claim 2, characterized in that: The step of determining a re-weighted first latent space feature vector based on the binary mask image, the posture feature information, the second image, and the self-attention layer includes: Perform channel splicing on the binary mask image, the posture feature information, and the second image to obtain a spliced ​​latent space feature vector; A weighted sum is performed based on the concatenated latent space feature vector and the self-attention layer to obtain the re-weighted first latent space feature vector.

6. A model training method, characterized in that: include: Acquire a clothing sample image and a first image; wherein, in the first image, a first person wears the clothing sample; Masking the clothing sample in the first image to obtain a binary mask image of the clothing sample and a second image after the clothing sample is masked; Performing posture extraction based on the first image and a character posture analysis model to obtain posture feature information of the first character; Inputting the clothing sample images into a clothing detail analysis model to be trained and a clothing appearance analysis model to be trained respectively, to obtain clothing detail feature information and clothing appearance feature information; The posture feature information after adding hidden noise is used as the conditional input side of the diffusion model to be trained, and the binary mask map, the second image, the detail feature information and the appearance feature information are used as the main input side of the diffusion model to be trained to determine the noise loss and the pixel level loss; and the model is trained by using the noise loss and the pixel level loss until a trained model is obtained.

7. The method according to claim 6, characterized in that The method of using the posture feature information after adding hidden noise as the conditional input side of the diffusion model to be trained, and using the binary mask map, the second image, the detail feature information and the appearance feature information as the main input side of the diffusion model to be trained to determine the noise loss and the pixel level loss includes: splicing the posture feature information after adding hidden noise, the binary mask image and the second image to determine a spliced ​​hidden feature vector; Determine a predicted image and predicted noise based on the spliced ​​latent space feature vector, the detail feature information and the appearance feature information; The noise loss is determined based on the predicted noise and the real noise; and the pixel level loss is determined according to the predicted image and the real image.

8. The method according to claim 6, characterized in that The performing model training by using the noise loss and the pixel level until a trained model is obtained comprises: The noise loss and the pixel level loss are summed to obtain a total loss; The character posture analysis model to be trained, the clothing detail analysis model to be trained and the diffusion model to be trained are trained respectively by using the total loss, until a pre-trained clothing detail analysis model, a pre-trained character posture analysis model and a pre-trained diffusion model are obtained respectively.

9. An electronic device, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method as described in any one of claims 1-5 or any one of claims 6-8.

10. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 5 or any one of claims 6 to 8 is implemented.

Images