Virtual fitting method and system for generating portrait fitting effects based on image reverse generation

Abstract

The present invention relates to the field of virtual fitting technology, and discloses a virtual fitting method and system for reversely generating a portrait fitting effect based on an image, comprising: constructing a virtual fitting model including a diffusion model, an appearance flow deformation module, three encoders, and a discriminator, wherein the encoder extracts features of clothing images, character images, and clothing style data respectively, and the appearance flow deformation module generates predicted deformed clothing; using clothing images and annotated character images as training sets to train the virtual fitting model, adding noise to the clothing images and inputting them into a diffusion model and a discriminator to generate images, performing edge softening processing on the generated images, detecting the occlusion information of the characters and the clothing therein and feeding it back to the corresponding images; obtaining the clothing to be virtualized and the character images input into the trained model to obtain the virtual fitting result. The present invention can fully extract the detailed features of clothing, adjust the image boundaries, consider the occlusion relationship of body parts, and enhance the fidelity and detail expression of the result image.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual fitting, and in particular to a virtual fitting method and system for reversely generating a portrait fitting effect based on an image. Background Art

[0002] Virtual fitting technology is a technology that uses computer technology to achieve virtual clothing fitting. From the early image splicing and image editing technology to the current deep generative model based on the Generative Adversarial Network (GAN) and encoder-decoder structure, virtual fitting technology has undergone tremendous changes, realizing the transition from simple grafting to real and delicate image translation.

[0003] In the existing virtual fitting technology based on deep learning, the mainstream method is to generate virtual clothing effects from portrait images. However, such methods often find it difficult to effectively retain clothing texture, details and texture information while ensuring the overall effect, resulting in blurred or distorted clothing details in the generated image. There are also methods that focus on covering the target person's clothes and then synthesizing the target clothes based on the covering. However, in actual application effects, the capabilities shown by many models always reach an upper limit and cannot be broken through. This is because many clothing details, such as patterns, wrinkles, and styles, are very variable, and the model often loses details during the synthesis process; and there are infinite combinations of different human bodies and different clothes, so how to smoothly generate high-quality renderings is extremely challenging. At the same time, compared with the generation mode from person to clothes, the generation from clothes to person requires more pixels to be processed, and it will consume more computing power to process the smooth transition between person, person background, and person and background, but the existing methods cannot efficiently extract detailed features from clothes; and, although a large number of details of the clothes are retained, the occluded part of the person and the clothes needs to be considered in the final synthesis, which requires processing the limb boundaries and the pixel coverage of the clothes, but the existing methods do not effectively process the limb boundaries, resulting in the overall unnatural effect of the final virtual try-on. Summary of the invention

[0004] To this end, the technical problem to be solved by the present invention is to overcome the deficiencies in the prior art and provide a virtual fitting method and system for reversely generating a portrait fitting effect based on an image, which can fully extract the detail features in the clothing image, adjust the image boundary, consider the limb occlusion relationship, and enhance the realism and detail expression of the result image.

[0005] In order to solve the above technical problems, the present invention provides a virtual fitting method for reversely generating a portrait fitting effect based on an image, comprising:

[0006] Acquire a clothing image and corresponding clothing data, wherein the clothing image includes an image of the clothing in its original state and an image of the clothing in its deformed state, and pre-process the clothing image; acquire a person image and corresponding person data, wherein the person data includes a person's posture and body structure, and annotate the person image according to the clothing data;

[0007] Constructing a virtual try-on model, the model includes a diffusion model, a generator, three encoders, and a discriminator; the three encoders respectively extract features of input clothing images, character images, and clothing style data; the generator includes an appearance flow deformation module, and the appearance flow deformation module is used to generate predicted deformed clothing;

[0008] The preprocessed clothing images and the annotated person images are used as a training set, and the training set is used to train the virtual try-on model to obtain a trained virtual try-on model;

[0009] When training the virtual fitting model, the preprocessed clothing image is denoised and input into the diffusion model and the discriminator to generate an image, wherein the generated image is an image containing a person and a transformed target clothing; the image containing the person and the transformed target clothing is subjected to edge softening processing, the occlusion information of the person and the clothing in the image containing the person and the transformed target clothing is detected, and the occlusion information is fed back to a corresponding position of the corresponding image;

[0010] The images of clothes and people that need to be virtually tried on are obtained and input into the trained virtual try-on model to obtain the virtual try-on results.

[0011] Furthermore, the overall structure of the virtual try-on model is a UNet network, the encoder for extracting features of a person image is ResNet50, the encoder for extracting features of a clothing image is ResNet34, and the encoder for extracting features of clothing style data is Bert.

[0012] Furthermore, the step of adding noise to the preprocessed clothing image and inputting the noise into the diffusion model and the discriminator to generate an image includes:

[0013] The area other than the clothes in the preprocessed original state image of the clothes is marked and noise is added, and the image of the preprocessed clothes in the deformed state remains unchanged;

[0014] The image with added noise is input into the diffusion model to start denoising and generate an image. After each denoising step, the current reconstruction loss is calculated, and the loss of clothing style data is calculated at the same time, and the generation result is guided step by step;

[0015] The denoised image is input into the discriminator to calculate the adversarial loss. The total loss in the denoising image generation process is obtained by combining the reconstruction loss, the loss of clothing style data, and the adversarial loss.

[0016] Furthermore, detecting occlusion information of the person and the clothes in the image containing the person and the transformed target clothes includes:

[0017] The occlusion mask is calculated as:

[0018] M occlusion =G σ *(I limb ⊙I cloth )+M pose ,

[0019] Among them, M occlusion is the occlusion mask, G σ is a Gaussian kernel, * indicates a convolution operation, I limb This is a picture of the body structure of a person. cloth is the binary image corresponding to the clothing image, ⊙ represents element-by-element multiplication, M pose is the occlusion mask predicted by the human pose estimation model.

[0020] Furthermore, when training the virtual fitting model, the overall loss is:

[0021]

[0022] Among them, σ() is a supervision function, which is used to ensure that the smaller the loss of the previous item, the larger the weight coefficient of the subsequent loss; L deform is the deformation constraint loss, L consist is the clothing consistency loss, L semantic is the local semantic loss, L perceptual is the loss of perceived fidelity.

[0023] Furthermore, the calculation method of the deformation constraint loss is:

[0024] L deform =||I cloth -I cloth ′||2,

[0025] Among them, I cloth is the original state image of the clothes, I cloth ′ is the image of the deformed state of the clothes, and ||||2 represents the quadratic norm calculation.

[0026] Furthermore, the calculation method of the clothing consistency loss is:

[0027]

[0028] Among them, I gen represents the generated image containing the person and the transformed target clothes, M′ rrepresents the deformed binary mask corresponding to the key area r, and ||||2 represents the quadratic norm calculation.

[0029] Furthermore, the calculation method of the local semantic loss is:

[0030]

[0031] Among them, ω r is the weight given to region r, φ i represents the i-th layer feature extractor using the attention mechanism, M r Represents the binary mask of the key region r, I gen represents the generated image containing the person and the transformed target clothes, I target The target image contains the person and the transformed target clothes. * indicates element-wise multiplication, and ||||1 indicates a single norm calculation.

[0032] Furthermore, the calculation method of the perceived fidelity loss is:

[0033] L perceptual =D(I gen , I target ),

[0034] Among them, I gen represents the generated image containing the person and the transformed target clothes, I target represents the target image containing the person and the transformed target clothes, and D represents the discriminator.

[0035] The present invention also provides a virtual fitting system for generating a portrait fitting effect based on an image inversely, comprising:

[0036] A data acquisition module is used to acquire clothing images and corresponding clothing data, wherein the clothing images include images of the clothing in its original state and images of the clothing in its deformed state; and to acquire character images and corresponding character data, wherein the character data includes the character's posture and body structure.

[0037] A data processing module is used to pre-process clothing images and annotate character images according to clothing data;

[0038] A virtual try-on model construction module is used to construct a virtual try-on model including a diffusion model, a generator, three encoders, and a discriminator. The three encoders respectively extract features of input clothing images, character images, and clothing style data. The generator includes an appearance flow deformation module, and the appearance flow deformation module is used to generate predicted deformed clothing.

[0039] The model training module is used to use the preprocessed clothing image and the annotated person image as a training set, and use the training set to train the virtual fitting model to obtain a trained virtual fitting model; when training the virtual fitting model, the preprocessed clothing image is denoised and input into the diffusion model and the discriminator to generate an image, and the generated image is an image containing the person and the transformed target clothing; the image containing the person and the transformed target clothing is subjected to edge softening processing, and the occlusion information of the person and the clothing in the image containing the person and the transformed target clothing is detected, and the occlusion information is fed back to the corresponding position of the corresponding image;

[0040] The virtual try-on module is used to use the data acquisition module to obtain the clothing images and person images that need to be virtually tried on, use the data processing module to process the clothing images and person images that need to be virtually tried on and input the trained virtual try-on model to obtain the virtual try-on results.

[0041] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0042] The present invention extracts image semantic information through three types of encoders, fully extracts the detail features in clothing images, and makes image translation more accurate and efficient; by extracting and fusing the semantic information of portrait images and clothing images, powerful multi-angle semantic guidance is provided to the generation network, and the effect of conversion from clothing to portrait is achieved; in combination with the biological constraints of the human body such as the posture and limb structure of the character, edge softening processing and anti-occlusion capability post-processing are performed, and the image boundary and limb occlusion relationship are adjusted while ensuring the overall effect of the generated result image, which significantly enhances the realism and detail expression of the result image. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0044] Figure 1 Flow chart of the method in the preferred embodiment of the present invention.

[0045] Figure 2 It is a schematic diagram of processing clothing images and person images in a preferred embodiment of the present invention.

[0046] Figure 3 In the preferred embodiment of the present invention, Figure 2 Schematic diagram of virtual try-on after image processing. DETAILED DESCRIPTION

[0047] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0048] Embodiment 1

[0049] Reference Figure 1-Figure 3 As shown, the present invention discloses a virtual fitting method for generating a portrait fitting effect based on an image reversely, comprising the following steps:

[0050] S1: Acquire a clothing image and corresponding clothing data, wherein the clothing image includes an image of the clothing in its original state and an image of the clothing in its deformed state, and pre-process the clothing image; acquire a person image and corresponding person data, wherein the person data includes a person's posture and body structure, and annotate the person image according to the clothing data.

[0051] In this embodiment, two types of data need to be collected: one is the image of clothes in the original state and the deformed state. The preprocessing of clothes images includes: binarization and cropping according to the clothes data, and finally obtaining the original state image of clothes, the binary image of the original state of clothes, the image of clothes in the deformed state, and the binary image of clothes in the deformed state. The second type is the character image and the corresponding character data. The character data includes the posture and body structure pictures of the character. When annotating the character image, it needs to correspond to the clothing data one by one.

[0052] S2: Construct a virtual try-on model, which includes a diffusion model (Denoising Diffusion Probabilistic Models, DDPM), a generator, three encoders and decoders, and a discriminator.

[0053] The diffusion model decomposes the input image into a change process and only needs to learn the inverse process; it can simplify learning and generate better results. DDPM can be used in combination with networks such as UNet and ResNet and has a flexible structure. DDPM has a fast training speed and an ideal sample generation speed. The algorithm has a low time complexity and good algorithm stability and does not rely on complex hyperparameter adjustments or network operations.

[0054] The three encoders extract the features of the input clothing image, person image, and clothing style data respectively, and the three decoders gradually restore the details corresponding to the features; two images, clothing image and person image, are randomly selected, and the clothing style data is text data. If there is no style, the default style data is used. The clothing image, person image, and clothing style data are input into the three encoders respectively, and finally the virtual fitting model realizes the appearance of the person wearing the given clothing image.

[0055] The generator includes an appearance flow warping module (AFWM), which is used to generate predicted deformed clothes. Before entering AFWM, the clothes data in the character image needs to be marked, which requires locating the clothes data through image data that are respectively marked with the character, environment, and clothes in the data set, and performing optical flow prediction on it and the given clothes data to obtain optical flow data. After that, the optical flow data is calculated with the given clothes to generate predicted deformed clothes. At this time, the clothes fit the character posture, and the model is continuously trained using the loss feedback optical flow prediction process.

[0056] The overall structure of the virtual try-on model is a UNet network, which can capture the contextual information of the input image and is conducive to generating and reconstructing the target structure. In addition, UNet does not rely on complex training techniques, has a simple structure, and is easy to implement. Different data sets and tasks can be adapted by changing hyperparameters such as network width and depth. The encoder for extracting the features of the person image is ResNet50, the encoder for extracting the features of the clothing image is ResNet34, and the encoder for extracting the features of the clothing style data is Bert.

[0057] S3: taking the preprocessed clothing images and the annotated person images as a training set, and using the training set to train the virtual try-on model to obtain a trained virtual try-on model.

[0058] When training the virtual fitting model, the pre-processed clothing image is denoised and input into the diffusion model and discriminator for denoising to generate an image, and the generated image is an image containing a person and a transformed target clothing; at this time, the boundary transitions of the person, background, and clothing in the image containing the person and the transformed target clothing may have defects and need to be further modified slightly. The image containing the person and the transformed target clothing is subjected to edge softening processing, and the image containing the person and the transformed target clothing is subjected to edge softening processing using a Gaussian function. Next, the occlusion information of the person and clothing in the image containing the person and the transformed target clothing is detected, and the data at the occlusion is fed back to the corresponding position of the corresponding image to improve the anti-occlusion capability of the virtual fitting model, specifically: the data at the occlusion is input into the post-processing part of the virtual fitting model and copied to the corresponding position of the corresponding image.

[0059] The preprocessed clothing image is denoised and input into the diffusion model and discriminator for denoising and generating an image, including: marking the areas other than the clothing in the preprocessed clothing original state image and adding noise, and the preprocessed clothing deformed state image is not added with noise and remains unchanged; the preprocessed clothing original state image with noise added is input into the diffusion model to start denoising and generating an image, and the current reconstruction loss is calculated after each denoising step, and the loss of clothing style data is calculated at the same time, and the generation result is guided step by step; the denoised image is input into the discriminator to calculate the adversarial loss, and the total loss in the denoising image generation process is obtained by combining the reconstruction loss, the loss of clothing style data, and the adversarial loss. In this embodiment, the total loss in the denoising image generation process is multiplied and summed according to the ratio set by the hyperparameters, specifically:

[0060] L total =αL denoise +β Lstyle +λL adv ,

[0061] Among them, L total is the total loss, L denoise is the reconstruction loss, L style is the style loss, L adv is the adversarial loss; α, β, and λ are hyperparameters used to control the weights of the three losses, and their values ​​are adjusted according to actual conditions. The final image generated after guided generation includes the person and the transformed target clothes.

[0062] Detecting the occlusion information of the person and the clothes in the image containing the person and the transformed target clothes includes: the occlusion information needs to be calculated from the limb structure, posture, and clothing binary image. The specific calculation includes multiplying the limb structure image with the clothing binary image, combining the human body posture estimation model to predict the occlusion area, and improving the accuracy. The specific calculation of the occlusion mask is:

[0063] M occlusion =G σ *(I limb ⊙I cloth )+M pose ,

[0064] Among them, M occlusion is the occlusion mask, G σ is the Gaussian kernel, σ is the Gaussian kernel parameter, * indicates the convolution operation, I limb This is a picture of the body structure of a person. cloth is the binary image corresponding to the clothing image, ⊙ represents element-by-element multiplication, M pose is the occlusion mask predicted by the human pose estimation model. The human pose estimation model used in this embodiment may be a model such as OpenPose.

[0065] If occlusion occurs, the arm information will be cleared. At this time, the calculated occlusion mask will be compared with the limb structure image, and the overlapping content of the limb part will be used as a new data and binarized, and finally multiplied with the original image. The calculation formula is:

[0066] I occlusion =M occlusion ⊙I limb ,

[0067] Among them, I occlusion It represents the data extracted from the occluded area. Finally, only the data from the occluded area is left. This data is sent to the post-processing part of the model and copied to the corresponding position of the image to provide the model with anti-occlusion capability.

[0068] When training the virtual try-on model, the overall loss is:

[0069]

[0070] Among them, σ() is a supervision function, which is used to ensure that the smaller the loss of the previous item, the larger the weight coefficient of the subsequent loss; L deform is the deformation constraint loss, L consist is the clothing consistency loss, L semantic is the local semantic loss, I perceptual is the loss of perceived fidelity.

[0071] The calculation method of the deformation constraint loss is:

[0072] L deform =||I cloth -I cloth ′||2,

[0073] Among them, I cloth is the original state image of the clothes, I cloth ′ is the image of the deformed state of the clothes, and ||||2 represents the quadratic norm calculation. This loss is used to limit the deformation of the clothes to not be too large.

[0074] The clothing consistency loss is a multi-region consistency loss, and the calculation method of the clothing consistency loss is:

[0075]

[0076] Among them, I gen represents the generated image containing the person and the transformed target clothes, M′ r represents the deformed binary mask corresponding to the key region r, and ||||2 represents the quadratic norm calculation. The key region r can represent the upper body, lower body, etc. This loss requires that the key region in the generated image is consistent with the corresponding deformed region.

[0077] The calculation method of the local semantic loss is:

[0078]

[0079] Among them, ω r is the weight given to region r, φ i represents the i-th layer feature extractor using the attention mechanism, M r Represents the binary mask of the key region r, I gen represents the generated image containing the person and the transformed target clothes, I target The target image contains the person and the transformed target clothes. * indicates element-wise multiplication, and ||||1 indicates a single norm calculation. This loss requires that the semantic features of each key area in the generated image are consistent with the target image.

[0080] The perceptual fidelity loss is calculated as:

[0081] L perceptual =D(I gen , I target ),

[0082] Among them, I gen represents the generated image containing the person and the transformed target clothes, I target represents the target image containing the person and the transformed target clothes, D represents the discriminator, and D() can be the realism score given by the adversarial discriminator or the human eye.

[0083] In this embodiment, when training the virtual fitting model, the overall loss starts from the model, that is, first deforming the clothes, then fixing the clothes, adding noise to other parts, and then continuously denoising to generate limbs and background. Therefore, it is necessary to first set the deformation constraint loss to guide the generation, followed by the clothing generation consistency loss to finally judge whether the generated clothing deformation is successful, the local semantic loss to judge whether the semantic features of the limb area in the generated image are consistent with the target image, and the denoising process loss is used to guide the generation of each step. cloth It is the original clothing, I cloth ′ is the deformed clothing. These two images can be used to calculate the deformation constraint loss L deform I gen represents the generated complete image, M′ r Represents the deformed binary mask corresponding to the key area r. By dividing the image into multiple areas, including upper body, lower body, etc., the quadratic norm is calculated for each area, and the final sum is obtained to obtain the clothing consistency loss L consist , which can also be called multi-region consistency loss. r Represents the binary mask of the key region r, I genrepresents the generated complete image, I target Represents the complete target image. These three images can be used to calculate the local semantic loss. Here, the attention mechanism is used, and φ is recorded. i represents the i-th layer feature extractor using the attention mechanism. r is the weight given to region r, and the final weighted summation results in a loss of L semantic ;I gen represents the generated complete image, I target Represents the complete image of the target, uses these two images for identification, introduces the idea of ​​adversarial network, uses discriminator or human eye scoring, and uses discriminator in the model to obtain the perceived fidelity loss L perceptual ; Finally, the overall loss is obtained, and the idea of ​​cascade loss is introduced on the overall loss. Note that σ() is a supervision function, which can ensure that the smaller the previous loss is, the larger the weight coefficient of the subsequent loss is, so as to ensure that more attention is paid to the previous loss in the early stage of training; for example, L deform , so that the image deformation effect is guaranteed first. Under the condition that the effect of the previous loss is better, the subsequent losses, such as semantic consistency and realism, are gradually paid attention to, and the training process is smoother and more controllable; the previous loss guides the subsequent loss, and finally forms a positive cycle enhancement; compared with simple linear weighting, a more dynamic and orderly loss adjustment can be achieved.

[0084] S4: Obtain the clothing images and person images that need to be virtually tried on and input them into the trained virtual try-on model to obtain the virtual try-on results.

[0085] Embodiment 2

[0086] The present invention also discloses a virtual fitting system for generating a portrait fitting effect based on an image inversely, comprising:

[0087] A data acquisition module is used to acquire clothing images and corresponding clothing data, wherein the clothing images include images of the clothing in its original state and images of the clothing in its deformed state; and to acquire character images and corresponding character data, wherein the character data includes the character's posture and body structure.

[0088] A data processing module is used to pre-process clothing images and annotate character images according to clothing data;

[0089] A virtual try-on model construction module is used to construct a virtual try-on model including a diffusion model, a generator, three encoders, and a discriminator. The three encoders respectively extract features of input clothing images, character images, and clothing style data. The generator includes an appearance flow deformation module, and the appearance flow deformation module is used to generate predicted deformed clothing.

[0090] The model training module is used to use the preprocessed clothing image and the annotated person image as a training set, and use the training set to train the virtual fitting model to obtain a trained virtual fitting model; when training the virtual fitting model, the preprocessed clothing image is denoised and input into the diffusion model and the discriminator to generate an image, and the generated image is an image containing the person and the transformed target clothing; the image containing the person and the transformed target clothing is subjected to edge softening processing, and the occlusion information of the person and the clothing in the image containing the person and the transformed target clothing is detected, and the occlusion information is fed back to the corresponding position of the corresponding image;

[0091] The virtual try-on module is used to use the data acquisition module to obtain the clothing images and person images that need to be virtually tried on, use the data processing module to process the clothing images and person images that need to be virtually tried on and input the trained virtual try-on model to obtain the virtual try-on results.

[0092] Embodiment 3

[0093] The present invention also discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the virtual fitting method for generating a portrait fitting effect based on reverse image generation in the first embodiment is implemented.

[0094] Embodiment 4

[0095] The present invention also discloses a device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the virtual fitting method for reversely generating a portrait fitting effect based on an image in the first embodiment is implemented.

[0096] The present invention extracts image semantic information through three types of encoders, fully extracts the detail features in clothing images, and makes image translation more accurate and efficient. At the same time, by extracting and fusing the semantic information of portrait images and clothing images, powerful multi-angle semantic guidance is provided to the generation network, thus achieving the effect of conversion from clothing to portrait.

[0097] The present invention combines the biological constraints of the human body, such as the posture and limb structure of the character, and is the first attempt to break through the introduction of the biological structure of the human limbs into image generation and adjustment; on this basis, edge softening processing and anti-occlusion capability post-processing are performed, and the image boundary and limb occlusion relationship are adjusted while ensuring the overall effect of the generated result image, which significantly enhances the realism and detail expression of the result image.

[0098] The present invention realizes the reverse generation process from clothes to portraits, solves the technical problem that the current mainstream methods cannot effectively retain the details of clothes, greatly enriches the implementation path of virtual fitting, breaks through the limitations of previous methods on the retention of clothing details, and achieves the purpose of generating high-quality and highly realistic portrait images from clothes.

[0099] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0100] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0101] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0102] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0103] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A virtual fitting method for generating a portrait fitting effect based on an image reversely, characterized in that: include: Acquire a clothing image and corresponding clothing data, wherein the clothing image includes an image of the clothing in its original state and an image of the clothing in its deformed state, and pre-process the clothing image; acquire a person image and corresponding person data, wherein the person data includes a person's posture and body structure, and annotate the person image according to the clothing data; Constructing a virtual try-on model, the model includes a diffusion model, a generator, three encoders, and a discriminator; the three encoders respectively extract features of input clothing images, character images, and clothing style data, the generator includes an appearance flow deformation module, and the appearance flow deformation module is used to generate a predicted deformed clothing image; The preprocessed clothing images and the annotated person images are used as a training set, and the training set is used to train the virtual try-on model to obtain a trained virtual try-on model; When training the virtual fitting model, the preprocessed clothing image is denoised and input into the diffusion model and the discriminator to generate an image, wherein the generated image is an image containing a person and a transformed target clothing; the image containing the person and the transformed target clothing is subjected to edge softening processing, the occlusion information of the person and the clothing in the image containing the person and the transformed target clothing is detected, and the occlusion information is fed back to a corresponding position of the corresponding image; Obtain the clothing images and person images that need to be virtually tried on and input them into the trained virtual try-on model to obtain the virtual try-on results; The method of adding noise to the preprocessed clothing image and inputting the noise into the diffusion model and the discriminator to generate an image includes: marking the area other than the clothing in the preprocessed original state image of the clothing and adding noise, and the image of the preprocessed clothing in the deformed state remains unchanged; inputting the image with the noise added into the diffusion model to start denoising and generate an image, calculating the current reconstruction loss after each denoising step, and calculating the loss of the clothing style data at the same time, and gradually guiding the generation result; inputting the denoised image into the discriminator to calculate the adversarial loss, and the total loss in the process of denoising and generating the image is obtained by combining the reconstruction loss, the loss of the clothing style data, and the adversarial loss; Detecting occlusion information of the person and the clothes in the image containing the person and the transformed target clothes, including: The occlusion mask is calculated as: M occlusion =G σ *(I limb ⊙I cloth )+M pose , Among them, M occlusion is the occlusion mask, G σ is a Gaussian kernel, * indicates a convolution operation, I limb This is a picture of the body structure of a person. cloth is the binary image corresponding to the clothing image, ⊙ represents element-by-element multiplication, M pose is the occlusion mask predicted by the human pose estimation model.

2. The virtual fitting method for generating a portrait fitting effect based on an image reversely according to claim 1, characterized in that: The overall structure of the virtual try-on model is a UNet network, the encoder for extracting features of character images is ResNet50, the encoder for extracting features of clothing images is ResNet34, and the encoder for extracting features of clothing style data is Bert.

3. The virtual fitting method for generating a portrait fitting effect based on an image reversely according to claim 1 or 2, characterized in that: When training the virtual try-on model, the overall loss is: Among them, σ() is a supervision function, which is used to ensure that the smaller the loss of the previous item, the larger the weight coefficient of the subsequent loss; L deform is the deformation constraint loss, L consist is the clothing consistency loss, L semantic is the local semantic loss, L perceptual is the loss of perceived fidelity.

4. The virtual fitting method for generating a portrait fitting effect based on an image reversely according to claim 3, characterized in that: The calculation method of the deformation constraint loss is: L deform =||I cloth -I cloth ′||2, Among them, I cloth is the original state image of the clothes, I cloth ′ is the image of the deformed state of the clothes, and || ||2 represents the quadratic norm calculation.

5. The virtual fitting method for generating a portrait fitting effect based on an image reversely according to claim 3, characterized in that: The calculation method of the clothing consistency loss is: Among them, I gen represents the generated image containing the person and the transformed target clothes, M′ r represents the deformed binary mask corresponding to the key area r, and || ||2 represents the quadratic norm calculation.

6. The virtual fitting method for generating a portrait fitting effect based on an image reversely according to claim 3, characterized in that: The calculation method of the local semantic loss is: Among them, ω r is the weight given to region r, φ i represents the i-th layer feature extractor using the attention mechanism, M r Represents the binary mask of the key region r, I gen represents the generated image containing the person and the transformed target clothes, I target Represents the target image containing the person and the transformed target clothes, * represents element-wise multiplication, and || ||1 represents a single norm calculation.

7. The virtual fitting method for generating a portrait fitting effect based on an image reversely according to claim 3, characterized in that: The perceptual fidelity loss is calculated as: L perceptual =D(I gen ,I target ), Among them, I gen represents the generated image containing the person and the transformed target clothes, I target represents the target image containing the person and the transformed target clothes, and D represents the discriminator.

8. A virtual fitting system for generating a portrait fitting effect based on an image, characterized in that: include: A data acquisition module is used to acquire clothing images and corresponding clothing data, wherein the clothing images include images of the clothing in its original state and images of the clothing in its deformed state; and to acquire character images and corresponding character data, wherein the character data includes the character's posture and body structure. A data processing module is used to pre-process clothing images and annotate character images according to clothing data; A virtual try-on model construction module is used to construct a virtual try-on model including a diffusion model, a generator, three encoders, and a discriminator. The three encoders respectively extract features of input clothing images, character images, and clothing style data. The generator includes an appearance flow deformation module. The appearance flow deformation module is used to generate a predicted deformed clothing image. The model training module is used to use the preprocessed clothing image and the annotated person image as a training set, and use the training set to train the virtual fitting model to obtain a trained virtual fitting model; when training the virtual fitting model, the preprocessed clothing image is denoised and input into the diffusion model and the discriminator to generate an image, and the generated image is an image containing the person and the transformed target clothing; the image containing the person and the transformed target clothing is subjected to edge softening processing, and the occlusion information of the person and the clothing in the image containing the person and the transformed target clothing is detected, and the occlusion information is fed back to the corresponding position of the corresponding image; The method of adding noise to the preprocessed clothing image and inputting the noise into the diffusion model and the discriminator to generate an image includes: marking the area other than the clothing in the preprocessed original state image of the clothing and adding noise, and the image of the preprocessed clothing in the deformed state remains unchanged; inputting the image with the noise added into the diffusion model to start denoising and generate an image, calculating the current reconstruction loss after each denoising step, and calculating the loss of clothing style data at the same time, and gradually guiding the generation result; inputting the denoised image into the discriminator to calculate the adversarial loss, and the total loss in the process of denoising and generating the image is obtained by combining the reconstruction loss, the loss of clothing style data, and the adversarial loss; Detecting occlusion information of a person and clothes in the image containing the person and the transformed target clothes, including: calculating an occlusion mask as: M occlusion =G σ *(I limb ⊙I cloth )+M pose , Among them, M occlusion is the occlusion mask, G σ is a Gaussian kernel, * indicates a convolution operation, I limb This is a picture of the body structure of a person. cloth is the binary image corresponding to the clothing image, ⊙ represents element-by-element multiplication, M pose is the occlusion mask predicted by the human pose estimation model; The virtual try-on module is used to use the data acquisition module to obtain the clothing images and person images that need to be virtually tried on, use the data processing module to process the clothing images and person images that need to be virtually tried on and input the trained virtual try-on model to obtain the virtual try-on results.

Images