Virtual fitting system and method for keeping local details based on diffusion model

Abstract

The invention discloses a virtual fitting system and method for keeping local details based on a diffusion model, and the system obtains the image features of a fitting model through a figure visual feature model, controllably achieves the step-by-step generation of a fitting effect from Gaussian noise through the generation capability of the diffusion model, improves the local details through a double-flow structure, and improves the fitting efficiency. According to the method, the fidelity of wrinkles, textures and fine design elements of the clothes is ensured, the method comprises the steps of data preprocessing, diffusion model training, image generation, post-processing and the like, compared with the prior art, the vivid virtual fitting effect can be generated, the textures, structures and detail information of different parts of the clothes can be kept, and the real-time performance of the clothes is improved. The method is suitable for the fields of e-commerce platforms, virtual shopping applications and the like, and has a good application prospect.

Description

Technical Field

[0001] The present invention relates to a virtual fitting system and method, and more specifically, to a virtual fitting system and method for maintaining local details based on a diffusion model, and belongs to the field of computer vision and artificial intelligence. Background Art

[0002] Traditional online shopping methods cannot provide users with a real try-on experience, and users often find it difficult to judge the style, size, and wearing effect of clothing. In order to solve this problem, virtual fitting technology came into being. Virtual fitting is a specific image generation function. Given a fitting model, it can be a picture of the user uploaded by the user, or a character model designated by the merchant for product display. At the same time, given the clothing to be tried on, the effect of the fitting model wearing the given clothing is directly generated, without the need for actual changing of clothes, but directly using artificial intelligence-generated image generation technology. By virtualizing the user and digitizing the clothing, simulating the real try-on scene, users can see the effect of the clothing on the body more intuitively, avoiding the merchant's product display needs to take pictures of the model wearing the clothing effect, which is usually time-consuming and laborious.

[0003] However, the existing virtual fitting system is based on 3D modeling and computer graphics technology. Although it can simulate the dynamics and wearing effects of clothing to a certain extent, it has the following major problems: First, there is a large gap or deformation between the clothing and the human body, the upper body effect is not natural enough, and the details and materials of the clothing are not fully expressed; second, due to the diversity of human body shape, the support for personalized needs is limited, and it cannot meet the user's personalized needs for specific styles or sizes, resulting in poor user experience; finally, the modeling process is complex and time-consuming, and it is impossible to generate high-quality virtual fitting effects in real time. Summary of the invention

[0004] In order to solve the above-mentioned problems of the prior art, the present invention provides a virtual fitting system and method based on a diffusion model that maintains local details, which can generate high-quality virtual fitting effects and provide personalized clothing recommendation and fitting functions, so as to solve the current problems of insufficient realism of virtual fitting, poor personalized experience, and complex and time-consuming modeling process.

[0005] In order to achieve the above object, the present invention is implemented by the following technical solutions: A virtual fitting system based on a diffusion model that retains local details, comprising a module for acquiring clothing and user images to be fitted, a feature processing module for semantic segmentation of model images and posture morphology, a diffusion model generation module for generating virtual fitting effect images, and an output module for feeding back the generated virtual fitting effect images to the user; The diffusion model generation module generates a high-fidelity fitting effect image from a noisy image through a pre-trained diffusion model; the local detail enhancement module is based on a dual-stream structure and different weight loss functions to improve the detail fidelity of the clothing area; the feature processing module can obtain the features required by the diffusion model based on the model image to be tried on and the clothing white background image input by the user; the output module can feed back the generation results to the user terminal in real time.

[0006] Preferably, the diffusion model is used to generate a virtual fitting effect diagram, and gradually removes noise to generate a realistic image of a model and clothing; the workflow of the diffusion model includes a forward diffusion process and a reverse generation process; The forward diffusion process: The diffusion model gradually adds noise to the image until the image is completely converted into Gaussian noise. The forward diffusion process is described by the following formula: ; in: is at the time step Image representation of; is a hyperparameter that controls the noise intensity at each step; Indicates that from the image Generate the next step image Gaussian distribution; through multiple iterations, the image Finally, it is converted into a pure noise image ,This process can be regarded as a data destruction process; The reverse generation process: the noise image is gradually denoised to generate the target image, and the parameterized model is learned , the diffusion model can be used to infer the original image from any noisy image: ; in, is the mean function in the reverse process, is the covariance matrix, , The parameters are obtained through learning of neural networks; through multiple iterations, the diffusion model gradually generates a realistic target image from Gaussian noise, and the output process is exactly the opposite of the forward diffusion process.

[0007] Preferably, the training goal of the diffusion model during training is to minimize the denoising error between each time step, which is achieved through an L2 loss function: ; in: is an image of real data; is the noise function predicted by the model; is the time step, It is the noise sampled from the standard Gaussian distribution; during the training process, the diffusion model continuously adjusts its parameters so that the noise can be removed as accurately as possible during the reverse generation process and a high-quality image matching the original image can be restored.

[0008] Preferably, the reverse generation process of the diffusion model is used to generate a virtual fitting effect, and specifically includes: The initial input is the concatenation of Gaussian noise, clothing region mask, and model retention region; In the generation process, control information is superimposed, including human body shape and posture as well as clothing feature maps; Obtained through the clothing diffusion model, the intermediate feature map at a specific position is retained after the clothing image is input; Through the reverse generation process of the diffusion model, the model gradually removes noise and generates realistic fitting effect images under the guidance of control information.

[0009] The present invention provides a virtual fitting method for maintaining local details based on a diffusion model, the method comprising the following steps: Step 1: Construct a virtual fitting training set, including model images and their corresponding clothing white background images; Step 2: Preprocess the image of the clothing white background image and the model image: extract the semantic segmentation and morphological posture features of the character through the character visual feature model; according to the result of semantic segmentation, obtain the mask that eliminates the original clothing area, and the area outside the mask is the model reserved area (image preprocessing includes: obtaining the clothing area mask according to the semantic segmentation of the fitting model, indicating the controllable range of the area to be worn, and it is necessary to leave redundancy to be compatible with different styles of clothing; the remaining part after removing the clothing area is the model reserved area, so that it is consistent with the original after the fitting is completed; the model reserved area, clothing area mask, and human body morphological posture are all used as model input); (according to semantic segmentation, the clothing area mask and reserved area can be obtained. The clothing area mask is not a strict clothing area, but an area where clothing may be worn. For example, if the model wears short sleeves, the clothing area mask needs to include the arms); Step 3: Perform virtual fitting effect generation training based on the diffusion model, and obtain feature maps of clothing in different stages of the diffusion model through the clothing diffusion model; Step 4: Generate clothing areas with high detail preservation by splicing corresponding clothing feature maps at the same stage through the virtual fitting model; in the generation process, apply the two-stream structure to ensure the true presentation of clothing texture, wrinkles and structural features in the generated image (the local detail preservation mechanism includes: two-stream structure, the clothing feature model is a diffusion model with the same structure as the virtual fitting generation diffusion model; input the clothing white background image into the clothing feature model, retain the intermediate feature map of the model for training and reasoning of the virtual fitting generation model, this part of the intermediate feature map can significantly improve the detail effect of the clothing in the generated virtual fitting image; based on the key areas of clothing, apply different loss function weights from the background to ensure high detail presentation of these areas); Step 5: Post-process the virtual fitting effect image to improve the final fitting effect; Step 6: Feedback the post-processed fitting effect image to the user through the network.

[0010] Preferably, step 2 specifically includes: obtaining a semantic segmentation map of the model through a mask2former network, selecting a number of a specific clothing semantic to obtain a specific mask, and expanding the mask according to the clothing type and the mask area to be compatible with different styles of clothing; For upper garments, the clothing mask area needs to include the entire torso and the arm area excluding the palms, and expand outward by the set pixel value to completely cover the possible range after the garment is put on; For bottoms, the clothing mask area needs to include the leg area except for the feet or shoes, and expand outward to cover the range if different pants or skirts are worn; For full-body costumes, the costume mask area needs to include all possible areas except the hands, feet, and head; Preferably, a densepose network model is used to obtain a densepose image of the model, which is used to reflect the postures of different areas of the human body; and a dwpose network model is used to obtain a skeleton image of the model, which is used to reflect the overall structure of the human body.

[0011] Preferably, the training of the diffusion model includes: adding set noise to the real model image after fitting, and the amplitude of the noise addition is pre-set by the diffusion model; splicing the noisy clothing, clothing area mask and model reserved area together and inputting them into the diffusion model for training; adding the human body shape and posture to the above input after passing through a multi-layer convolutional network for training together; training together with the clothing feature model; the loss function of the diffusion model includes but is not limited to predicting the mean square error between the added noise and the real noise, so that the model can learn the denoising process and can gradually remove noise from the initial completely random noise to achieve the effect of controllable image generation.

[0012] Preferably, the dual-stream structure is implemented by adopting the DWPOSE dual-stream weight sharing human posture estimation method.

[0013] Preferably, the generated virtual fitting effect image in step 5 needs to be post-processed to improve the image quality. Specifically, the generated image is alpha-fused with the original image, and the fused mask is the clothing area mask, so that the reserved area can be closer to the reserved area of ​​the original image, while retaining the generated clothing area; The stable diffusion series models are used, including stable diffusion 1.5, stable diffusion XL and stable diffusion 3. The input image is the latent image after VAE processing, and the final raw image effect is displayed after passing through the VAE decoder.

[0014] Beneficial effects: The present invention can generate high-quality virtual fitting effects, maintain the texture, structure and detail information of clothes in different parts, effectively improve the user experience of virtual fitting, and provide personalized clothing recommendation and try-on functions, solving the current problems of insufficient realism of virtual fitting, poor personalized experience, and complex and time-consuming modeling process. The system of the present invention is suitable for e-commerce platforms, virtual shopping applications and other fields, and has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the process of the present invention.

[0016] Figure 2 It is a virtual fitting flow chart of the present invention.

[0017] Figure 3 It is a schematic diagram of the connection between the system terminal and the server in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following is in conjunction with the instruction manual Figure 1-3 , the present invention is further illustrated, but the present invention is not limited to the following embodiments.

[0019] The present invention obtains the image features of the fitting model through the character visual feature model, uses the generation ability of the diffusion model to controllably realize the gradual generation of the fitting effect from Gaussian noise, and adopts a dual-stream structure to enhance local details to ensure the fidelity of the wrinkles, textures and subtle design elements of the clothes. The method mainly includes the steps of data preprocessing, diffusion model training, image generation and post-processing.

[0020] The present invention discloses a virtual fitting system for maintaining local details based on a diffusion model, comprising a module for acquiring clothing and user images to be fitted, a feature processing module for semantic segmentation and posture morphology of model images, a diffusion model generation module for generating virtual fitting effect pictures, and an output module for feeding back the generated virtual fitting effect pictures to the user; the diffusion model generation module generates a high-fidelity fitting effect picture from a noise image through a pre-trained diffusion model; the local detail enhancement module is based on a dual-stream structure and different weight loss functions to improve the detail fidelity of the clothing area; the feature processing module can obtain the features required by the diffusion model based on the model picture to be fitted and the clothing white background picture input by the user; the output module can feed back the generated results to the user terminal in real time. Virtual fitting based on the diffusion model can provide rich functions, support different clothing, different actions, different tasks, and different backgrounds. With the popularity of e-commerce and online shopping, online shopping, especially clothing products, the popularity and acceptance of virtual fitting are gradually increasing, and the demand for virtual fitting effects is also gradually increasing.

[0021] A specific system application: The system collects user person images through user terminal devices (such as mobile phones, tablets or computers), for example, users take a full-body photo through a camera. Users upload a clothing product link from an online shopping website according to their preferences. After receiving the user's needs, the server obtains the white background image of the clothing product through crawler technology. The server obtains the corresponding semantic segmentation and morphological features through the human visual model for use by the diffusion model. The server obtains the corresponding clothing area and the corresponding task reserved area in the user person image semantic segmentation map according to the clothing type of the white background image. The server generates the user's virtual fitting effect map through the clothing diffusion model and the virtual fitting generation model, and performs post-processing. The server feeds back the results to the user terminal device. Users can repeatedly upload different clothing links or person model images or choose to directly upload specific clothing white background images, and select the corresponding product according to the fitting effect. Users can save the fitting effect as pictures or videos and share it with friends or social media.

[0022] In a preferred embodiment, the diffusion model is used to generate a virtual fitting effect diagram, and gradually removes noise to generate a realistic image of a model and clothing; the workflow of the diffusion model includes a forward diffusion process and a reverse generation process; The forward diffusion process: The diffusion model gradually adds noise to the image until the image is completely converted into Gaussian noise. The forward diffusion process is described by the following formula: ; in: is at the time step Image representation of; is a hyperparameter that controls the noise intensity at each step; Indicates that from the image Generate the next step image Gaussian distribution; through multiple iterations, the image Finally, it is converted into a pure noise image ,This process can be regarded as a data destruction process; The reverse generation process: the noise image is gradually denoised to generate the target image, and the parameterized model is learned , the diffusion model can be used to infer the original image from any noisy image: ; in, is the mean function in the reverse process, is the covariance matrix, , The parameters are obtained through learning of neural networks; through multiple iterations, the diffusion model gradually generates a realistic target image from Gaussian noise, and the output process is exactly the opposite of the forward diffusion process.

[0023] The training goal of the diffusion model during training is to minimize the denoising error between each time step, which is achieved through an L2 loss function: ; in: is an image of real data; is the noise function predicted by the model; is the time step, It is the noise sampled from the standard Gaussian distribution; during the training process, the diffusion model continuously adjusts its parameters so that the noise can be removed as accurately as possible during the reverse generation process and a high-quality image matching the original image can be restored.

[0024] The reverse generation process of the diffusion model is used to generate a virtual fitting effect, and specifically includes: The initial input is the concatenation of Gaussian noise, clothing region mask, and model retention region; In the generation process, control information is superimposed, including human body shape and posture as well as clothing feature maps; Obtained through the clothing diffusion model, the intermediate feature map at a specific position is retained after the clothing image is input; Through the reverse generation process of the diffusion model, the model gradually removes noise and generates realistic fitting effect images under the guidance of control information.

[0025] like Figure 1 As shown, a flowchart of a virtual fitting method based on a diffusion model to maintain local details of the present invention is provided, and the method comprises the following steps: Step 1: Obtain the fitting model and the clothes to be worn required by the user, and build a virtual fitting training set, including clothes and pictures of models wearing the clothes; Step 2: Extract the semantic segmentation and morphological posture features of the character through the character visual feature model; Step 3: According to the result of semantic segmentation, obtain the mask to eliminate the original clothing area, and the area outside the mask is the model reserved area; Step 4: Input the clothing image into the clothing feature model to obtain the feature image of the clothing diffusion model; Step 5: Splice Gaussian noise, clothing area mask and model reserved area, input the virtual fitting generation model, and input the morphological posture and clothing diffusion model feature map; gradually iterate the denoising to generate the virtual fitting effect map, and perform post-processing to achieve virtual fitting effect generation training based on the diffusion model; in the generation process, apply the two-stream structure to ensure the real presentation of the texture, wrinkles and structural features of the clothing in the generated image; Step 6: Show the fitting effect to the user in real time and provide feedback to the user.

[0026] It can be seen from the above steps that the technical solution of the present invention includes the steps of user input to obtain a model image or clothing to be worn, data (image) preprocessing, diffusion model training, image generation (local detail preservation) post-processing, and effect feedback, which are specifically as follows: 1) User input acquisition: Receive user input of model images and specific clothing to be worn through network services. Model images include but are not limited to user's own pictures, current photos or album selections, or other model images, as well as model images for which merchants have the right to use their portraits. Clothes to be worn can be obtained by crawling white-background images from network links, including but not limited to clothing white-background images or image links uploaded by users themselves.

[0027] The acquisition of training data sets can use open source data sets or self-collected data sets without violating copyright and usage rights. The training of the model depends on the quality of the construction of the training set. The constructed training set needs to include different clothing types, model postures, backgrounds, etc. as much as possible to cover different needs. Clothing and models need to be strictly corresponding. One piece of clothing can correspond to multiple models wearing the clothing. A model can correspond to tops or bottoms due to the difference between the upper and lower garments; the white background image of the clothing needs to be the clothing facing up, and the background is white except for the clothing area. The posture of the model should be the front or the side with a certain angle, including the model in the reverse posture. In this case, the corresponding white background image needs to be the clothing facing up. During training, the model image can be enhanced by random cropping, stretching, contrast saturation, etc., but the cropped reserved area must include the given clothing. At the same time, the image enhancement in color needs to be applied synchronously to the clothing white background image to maintain color stability. The image size finally input to the model needs to be normalized and adjusted to a uniform size; 2) Image preprocessing: Preprocess the clothing white background image and model image. For example, obtain the semantic segmentation map of the model through the mask2former network, select the number of the specific clothing semantics to obtain a specific mask, and expand it according to the clothing type and mask area to be compatible with different styles of clothing. For tops, the clothing mask area needs to include the entire torso and the arm area except the palms, and expand outward by a certain pixel value to completely cover the possible range after wearing the clothing; for bottoms, the clothing mask area needs to include the leg area except the feet or shoes, and expand outward to cover the possible range if wearing different pants or skirts. For full-body clothing, the clothing mask area needs to include all possible areas except the hands, feet and head.

[0028] The densepose network is used to obtain the model's densepose map, which is used to reflect the postures of different areas of the human body. The skeleton diagram of the model is obtained through the dwpose network to reflect the overall structure of the human body.

[0029] 3) Diffusion model training: The training process of the diffusion model is a process of continuously adding noise and gradually denoising. The input of the clothing diffusion model is clothing. After one run, the feature maps of different positions of the diffusion model are retained. The feature map represents the intermediate state of the model and contains detailed information of different scales of clothing. In a specific embodiment, the input feature map of the self-attention layer is retained. The input of the virtual fitting generation model is the real image plus Gaussian noise, the clothing area mask and the splicing of the model reserved area. The noise intensity added to the real image is related to the predefined noise control of the diffusion model. Different time steps correspond to different noise intensities, and generally the noise intensity increases with the increase of the time step. In a specific embodiment, the noise sequence of DDPM is used. When the noise is the largest, the real image is completely transformed into Gaussian noise. After the preprocessed posture map passes through the convolutional network, it has the same size as the input space of the virtual fitting generation model, and can be directly added to control the generated model to maintain a specific posture. The loss function of the model is the mean square error between the predicted noise and the actual added noise. The purpose of training is to enable the virtual fitting diffusion model to gradually denoise and generate virtual fitting images under the guidance of posture control and clothing feature maps when inputting complete Gaussian noise, clothing area masks, and model retention areas. This process requires multiple iterations.

[0030] 4) Local detail preservation: The clothing feature map retains the multi-scale detailed information of the clothing. In the process of generating the virtual fitting diffusion model, splicing the clothing feature map in the same position area and performing self-attention calculation can enable the virtual fitting diffusion model to learn the local details of the clothing. When calculating the mean square error between the predicted noise and the actual added noise, increasing the clothing area, especially the boundary, can improve the generation effect in terms of maintaining local details.

[0031] 5) Effect feedback: The generated virtual fitting effect image needs to be post-processed to improve the image quality. For example, the generated image is alpha-fused with the original image, and the fused mask is the clothing area mask, so that the reserved area can be closer to the reserved area of ​​the original image, while retaining the generated clothing area. In a specific embodiment, a stable diffusion series model example can be used, such as stable diffusion 1.5, stable diffusion XL and stable diffusion 3. The input image is a latent image processed by VAE. When the final raw image effect is displayed, it needs to pass through the VAE decoder. This process will cause the loss of image information and distortion of local details. The final effect is fed back to the user terminal through the network service.

[0032] In a preferred embodiment, step 2 specifically includes: obtaining a semantic segmentation map of the model through a mask2former network, selecting a number of a specific clothing semantic to obtain a specific mask, and expanding the mask according to the clothing type and the mask area to be compatible with different styles of clothing; For upper garments, the clothing mask area needs to include the entire torso and the arm area excluding the palms, and expand outward by the set pixel value to completely cover the possible range after the garment is put on; For bottoms, the clothing mask area needs to include the leg area except for the feet or shoes, and expand outward to cover the range if different pants or skirts are worn; For full-body costumes, the costume mask area needs to include all possible areas except the hands, feet, and head; The densepose network model is used to obtain the model's densepose image, which is used to reflect the postures of different areas of the human body; the dwpose network model is used to obtain the model's skeleton image, which is used to reflect the overall structure of the human body.

[0033] The following is an introduction to the dwpose network model (abbreviated as dwpose) and the densepose network model (abbreviated as densepose): DWPOSE (Dual-stream Weight-shared Pose Estimation) is a dual-stream weight-shared human pose estimation method based on deep learning, which is used to accurately predict the 3D joint positions of the human body from 2D images. The model effectively improves the accuracy of pose estimation by jointly learning 2D and 3D pose information. DWPOSE is particularly suitable for high-precision 3D reconstruction of human poses and can achieve robust pose estimation in various complex backgrounds.

[0034] Working principle: The core architecture of the DWPOSE (network) model consists of two parallel streams: 2D stream: used to extract the joint key point positions of the human body from the input two-dimensional image.

[0035] 3D flow: Based on the output of the 2D flow, further predict the joint positions in the three-dimensional space.

[0036] The two streams transfer information through shared convolutional layers and feature representations. This shared weight mechanism can simultaneously utilize two-dimensional and three-dimensional information to enhance the performance of the model. The prediction process of DWPOSE is divided into the following steps: Feature extraction: Input a two-dimensional human body image and extract multi-scale image features through a deep convolutional neural network (usually using ResNet or HRNet as the backbone network).

[0037] 2D joint point detection: The 2D flow regresses each pixel of the image and outputs a 2D heatmap of human joints. The position of each joint point in two-dimensional space is determined by the peak value of the heatmap.

[0038] 3D pose estimation: The 3D flow predicts the coordinates of each joint point in 3D space based on the 2D heat map and extracted image features. By using multi-view 3D pose data for supervised learning, the model is able to accurately predict the 3D joint point positions.

[0039] DWPOSE model advantages: The main advantages of DWPOSE are: Dual-stream architecture: By simultaneously learning 2D and 3D pose information, DWPOSE is able to handle both 2D and 3D pose estimation tasks in a single architecture, and the model’s shared weight mechanism improves computational efficiency and prediction accuracy.

[0040] Robustness: DWPOSE performs well in scenes with complex backgrounds, occlusions, or extreme changes in human poses, and is suitable for practical applications such as virtual fitting, human tracking, and motion capture.

[0041] Efficiency: The model is lightweight and can run efficiently on mobile devices or real-time systems.

[0042] Application scenarios: DWPOSE is widely used in fields that require high-precision pose estimation, especially for: Virtual Fitting: By accurately estimating the user’s 3D posture, DWPOSE can precisely fit virtual clothing to the user’s body, achieving a realistic fitting effect.

[0043] Motion Capture: In motion capture systems, DWPOSE can be used to track human postures in real time and is widely used in gaming, filmmaking, sports analysis and other fields.

[0044] Augmented Reality (AR): In AR applications, DWPOSE can achieve real-time human posture tracking and provide technical support for augmented reality experience.

[0045] Among them, DensePose is a deep learning model developed by the Facebook AI research team for dense human pose estimation. Its task is to map each pixel of a 2D RGB image to a 3D surface model of the human body. Unlike traditional pose estimation models that only detect joints, DensePose can accurately map the entire human body surface, so it can provide more detailed pose and human surface information. This technology is widely used in virtual fitting, virtual reality, augmented reality, and motion analysis.

[0046] Working principle: The DensePose workflow consists of the following steps: Feature extraction: First, DensePose uses a deep convolutional neural network (such as ResNet) to extract features from the input 2DRGB image and generate a high-level feature representation of the image.

[0047] Human region detection: Next, DensePose uses the R-CNN (Region-based Convolutional Neural Network) architecture to identify the human region in the image and segment the human body.

[0048] UV coordinate prediction: After identifying the human body area, DensePose predicts the UV coordinates (i.e., a two-dimensional parametric coordinate system) of each pixel in the image on a standard 3D human body model. This coordinate is used to represent the corresponding position of each pixel on the human body surface on the three-dimensional model. UV coordinates are a standardized parametric grid that can accurately represent the texture and geometry of the human body surface.

[0049] Pose mapping: By mapping each 2D pixel point to the corresponding part of the 3D human body model, DensePose achieves dense 3D pose mapping of the entire surface of the human body, rather than just estimating key points.

[0050] DensePose has the following significant advantages over traditional pose estimation methods: Full surface mapping: DensePose not only estimates the position of the human body's joints, but also accurately maps each pixel in the image to the 3D human body surface, providing richer posture information.

[0051] Detailed 3D surface reconstruction: Through UV coordinates, DensePose can generate a complete 3D human body surface model, which makes it have great application potential in virtual fitting, motion capture and human body reconstruction.

[0052] Multi-pose and occlusion processing: DensePose performs well in handling complex human pose changes and partial occlusion, and can handle complex body shapes and poses.

[0053] DensePose has a wide range of applications, especially in the following areas: Virtual fitting: By mapping each pixel to the 3D human body surface, DensePose can generate accurate virtual fitting effects, simulating the real fit of clothing to the human body.

[0054] Augmented Reality (AR): In AR applications, DensePose can be used to track the user's posture in real time, generate accurate 3D human models, and provide interactive experiences in AR environments.

[0055] Motion capture: DensePose can accurately reconstruct the three-dimensional surface of the human body and is used to capture complex posture changes of the human body in real time. It has important applications in film production and game development.

[0056] One of the preferred neural networks in this application is Self Attention. The following is an introduction to Self Attention: Self-attention is a common mechanism in neural networks that is used to calculate the correlation between elements in an input sequence. This mechanism dynamically assigns attention weights to emphasize important parts of the input sequence and capture long-distance dependencies. Self-attention was first used in the Transformer model in natural language processing tasks, but it is also applicable to computer vision, image generation and other tasks. How it works The key to self-attention is to make each element (such as words, pixels, etc.) in the input sequence interact with other elements in the sequence to generate an attention map that represents the correlation between the element and other elements. Its core steps include the following parts: 1. Input representation: Assume that the input is a sequence , Self Attention generates query (Query), key (Key), and value (Value) vectors for each element in the sequence.

[0057]

[0058] in, , and are the weight matrices used to generate queries, keys, and values, respectively.

[0059] 2. Correlation calculation: By calculating the dot product of the query and the key, the correlation between the elements within the sequence can be obtained: ; in, is a scaling factor used to prevent the dot product value from being too large.

[0060] 3. Weighting: Use the softmax function to convert the dot product results into attention weights, which are then used to weight the value vector VVV to generate a new input representation.

[0061] Through this process, Self Attention can flexibly assign different weights to each element in the input sequence and capture the dependencies in the sequence. This mechanism is very useful in sequential data (such as text, time series) and pixel-level feature extraction in image processing.

[0062] Advantages: Long-distance dependency capture: Self Attention allows the model to capture the relationship between elements at any position in the sequence without relying on a fixed window size, which is particularly suitable for tasks that handle long-distance dependencies.

[0063] Parallelizable: Unlike recurrent neural networks (RNNs), Self Attention can calculate the attention between all elements in parallel, greatly improving computational efficiency.

[0064] Wide application: Self Attention is not only suitable for natural language processing, but can also be applied to tasks such as image generation, image segmentation, motion capture, etc. to capture the relationship between distant pixels in the image.

[0065] Application scenarios: Self Attention has been widely used in many fields. The following are several typical application scenarios: Natural Language Processing: Self Attention is the core mechanism of the Transformer model and is used for tasks such as machine translation and text classification.

[0066] Image processing: Self Attention is used in Visual Transformer (ViT) to capture the relationship between long-distance pixels in an image, thereby improving the accuracy of image generation and recognition.

[0067] Virtual Fitting: In virtual fitting, Self Attention can be used to capture the relationship between the various parts of the garment, thereby generating a more realistic fitting effect.

[0068] One of the preferred neural networks in this application is Cross Attention Map. The following is an introduction to Cross Attention Map: Cross Attention Map is a commonly used mechanism in deep learning, mainly used to process the relationship between two or more different information sources. In image generation, natural language processing, and multimodal processing tasks (such as converting text descriptions into image generation, or in portrait and clothing fitting systems), Cross Attention can effectively establish the correspondence between different modal information. It calculates the attention weights between different modalities and captures the most relevant information for the current task in the context, thereby helping the model to better understand and generate high-quality results.

[0069] Working principle: The working principle of Cross Attention Map is derived from the attention mechanism, which dynamically adjusts the information flow by learning the dependencies between different inputs. In Cross Attention, the model calculates the correlation of each element between two different input sequences (such as two image feature maps, images and text, human bodies and clothing, etc.) to generate an attention map.

[0070] The basic elements of Cross Attention include query, key, and value, which is similar to the self-attention mechanism, but in Cross Attention, the query comes from one input sequence, while the key and value come from another input sequence. Specifically: Query (Q): represents the information of the target input, such as key points in a human feature map or keywords in a text description.

[0071] Key (K): represents the information of the source input, such as the texture of clothing or the features of important areas in an image.

[0072] Value (V): represents the actual content of the source input, which will be weighted and combined based on the relevance of the key to the query.

[0073] The calculation process of Cross Attention Map includes the following steps: 1. Linear mapping of query, key, and value: Map two input sequences into query, key, and value vectors through linear transformation. Assuming one input is X and the other input is Y, the model maps them to Q, K, and V respectively through learning parameters:

[0074] in , and is the weight matrix of the linear mapping.

[0075] 2. Attention weight calculation: The correlation between the query and the key is calculated by the dot product, and the result is converted into attention weight through the softmax function:

[0076] in, The dimension of the key, used to scale the dot product to avoid too large values.

[0077] 3. Weighted combination of attention weighted values: The model uses the attention weights obtained above to perform a weighted combination of the values ​​V to produce information output related to the query Q. This weighted combination process ensures that each target input can obtain the most relevant source input information.

[0078] Through this process, Cross Attention can generate a cross-attention map for the target input, capturing the fine-grained correspondence between the two inputs. This plays an important role in the integration and information fusion of multimodal inputs.

[0079] Cross Attention Map has many advantages when processing multimodal information: Strong information relevance: Cross Attention Map can dynamically calculate the relevance of different input modalities (such as images and text, human bodies and clothing), and accurately locate the most relevant part of the target input with the source input through the attention mechanism.

[0080] Multimodal fusion: This mechanism allows the effective integration of input information from different sources. For example, in virtual fitting, it can accurately capture the user's body shape characteristics and the adaptability of clothing styles.

[0081] Capturing local details: Cross Attention can refine the input information, especially in image generation or enhancement tasks. The model can focus on specific areas or keywords in the image or text to generate more realistic details.

[0082] Efficiency: Cross Attention Map provides an efficient way to significantly reduce the number of parameters that need to be processed by weight sharing, thereby improving computational efficiency.

[0083] Application in virtual fitting: The application of Cross Attention Map in the virtual fitting system is crucial. It can accurately establish the adaptive relationship between the human body and clothing, thereby improving the realism of the fitting effect.

[0084] In a preferred embodiment, the training of the diffusion model includes: adding set noise to the real model image after fitting, and the amplitude of the noise addition is pre-set by the diffusion model; splicing the noisy clothing, clothing area mask and model reserved area together and inputting them into the diffusion model for training; adding the human body shape and posture to the above input after passing through a multi-layer convolutional network for training together; training together with the clothing feature model; the loss function of the diffusion model includes but is not limited to predicting the mean square error between the added noise and the real noise, so that the model can learn the denoising process and can gradually remove noise from the initial completely random noise to achieve the effect of controllable image generation.

[0085] In a preferred embodiment, the dual-stream structure is implemented by adopting the DWPOSE dual-stream weight sharing human posture estimation method.

[0086] In a preferred embodiment, the generated virtual fitting effect image in step 5 needs to be post-processed to improve the image quality. Specifically, the generated image is alpha-fused with the original image, and the fused mask is the clothing area mask, so that the reserved area can be closer to the reserved area of ​​the original image, while retaining the generated clothing area; The stable diffusion series models are used, including stable diffusion 1.5, stable diffusion XL and stable diffusion 3. The input image is the latent image after VAE processing, and the final raw image effect is displayed after passing through the VAE decoder.

[0087] In summary, the present application provides a virtual fitting method and system based on a diffusion model to solve the problems of existing virtual fitting methods, so as to improve the fitting effect and enhance the user experience. It mainly uses a pre-trained diffusion model and inputs a large number of clothing image data sets to learn the distribution characteristics of clothing styles, materials and textures. When a user selects a piece of clothing, the system generates a high-resolution image of the corresponding clothing through the diffusion model. The diffusion model iteratively generates high-quality clothing images from random noise through a step-by-step "denoising" process. The clothing image not only retains the details of the real clothing (such as wrinkles, texture, gloss, etc.), but can also be adjusted according to the user's preferences, such as changing the color, adding decorations or adjusting the size.

[0088] Finally, it should be noted that the present invention is not limited to the above embodiments, and many variations are possible. All variations that can be directly derived or associated with the content disclosed by a person skilled in the art should be considered as within the protection scope of the present invention.

Claims

1. A virtual fitting system based on a diffusion model to preserve local details, characterized in that: It includes a module for acquiring clothing and images of users to be tried on, a feature processing module for semantic segmentation of model images and posture morphology, a diffusion model generation module for generating virtual fitting effect images, and an output module for feeding back the generated virtual fitting effect images to the user; The diffusion model generation module generates a high-fidelity fitting effect image from a noise image through a pre-trained diffusion model; the local detail enhancement module is based on a dual-stream structure and different weight loss functions to improve the detail fidelity of the clothing area; the feature processing module can obtain the features required by the diffusion model based on the model image to be fitted and the clothing white background image input by the user; The output module can feed back the generated results to the user terminal in real time.

2. The virtual fitting system based on diffusion model and maintaining local details according to claim 1, characterized in that: The diffusion model is used to generate a virtual fitting effect diagram, and gradually removes noise to generate a realistic image of the combination of the model and the clothing; the workflow of the diffusion model includes a forward diffusion process and a reverse generation process; The forward diffusion process: The diffusion model gradually adds noise to the image until the image is completely converted into Gaussian noise. The forward diffusion process is described by the following formula: ； in: is at the time step Image representation of; is a hyperparameter that controls the noise intensity at each step; Indicates that from the image Generate the next step image Gaussian distribution; through multiple iterations, the image Finally, it is converted into a pure noise image ,This process can be regarded as a data destruction process; The reverse generation process: the noise image is gradually denoised to generate the target image, and the parameterized model is learned , the diffusion model can be used to infer the original image from any noisy image: ； in, is the mean function in the reverse process, is the covariance matrix, , The parameters are obtained through learning of neural networks; through multiple iterations, the diffusion model gradually generates a realistic target image from Gaussian noise, and the output process is exactly the opposite of the forward diffusion process.

3. The virtual fitting system based on diffusion model and maintaining local details according to claim 2, characterized in that: The training goal of the diffusion model during training is to minimize the denoising error between each time step, which is achieved through an L2 loss function: ； in: is an image of real data; is the noise function predicted by the model; is the time step, It is the noise sampled from the standard Gaussian distribution; during the training process, the diffusion model continuously adjusts its parameters so that the noise can be removed as accurately as possible during the reverse generation process and a high-quality image matching the original image can be restored.

4. A virtual fitting system based on diffusion model and maintaining local details according to claim 2 or 3, characterized in that: The reverse generation process of the diffusion model is used to generate a virtual fitting effect, which specifically includes: The initial input is the concatenation of Gaussian noise, clothing region mask, and model retention region; In the generation process, control information is superimposed, including human body shape and posture as well as clothing feature maps; Obtained through the clothing diffusion model, the intermediate feature map at a specific position is retained after the clothing image is input; Through the reverse generation process of the diffusion model, the model gradually removes noise and generates realistic fitting effect images under the guidance of control information.

5. A virtual fitting method based on a diffusion model to preserve local details, characterized in that The method comprises the following steps: Step 1: Construct a virtual fitting training set, including model images and their corresponding clothing white background images; Step 2: Preprocess the clothing white background image and the model image: extract the semantic segmentation and morphological posture features of the character through the character visual feature model; according to the result of semantic segmentation, obtain the mask to eliminate the original clothing area, and the area outside the mask is the model reserved area; Step 3: Perform virtual fitting effect generation training based on the diffusion model, and obtain the feature maps of clothing in different stages of the diffusion model through the clothing diffusion model Step 4: Generate clothing areas with high detail retention by splicing corresponding clothing feature maps at the same stage through the virtual fitting model; in the generation process, a two-stream structure is applied to ensure that the texture, wrinkles and structural features of the clothing are realistically presented in the generated image; Step 5: Post-process the virtual fitting effect image to improve the final fitting effect; Step 6: Feedback the post-processed fitting effect image to the user through the network.

6. The virtual fitting method based on diffusion model and maintaining local details according to claim 5, characterized in that: Step 2 specifically includes: obtaining the semantic segmentation map of the model through the mask2former network, selecting the number of the specific clothing semantics to obtain a specific mask, and expanding it according to the clothing type and mask area to be compatible with different styles of clothing; For upper garments, the clothing mask area needs to include the entire torso and the arm area excluding the palms, and expand outward by the set pixel value to completely cover the possible range after the garment is put on; For bottoms, the clothing mask area needs to include the leg area except for the feet or shoes, and expand outward to cover the range if different pants or skirts are worn; For full-body suits, the clothing mask area needs to include all possible areas except the hands, feet, and head.

7. The virtual fitting method based on diffusion model and maintaining local details according to claim 6, characterized in that: The densepose network model is used to obtain the model's densepose image, which is used to reflect the postures of different areas of the human body; the dwpose network model is used to obtain the model's skeleton image, which is used to reflect the overall structure of the human body.

8. The virtual fitting method based on diffusion model and maintaining local details according to claim 5, characterized in that: The training of the diffusion model includes: adding set noise to the real model image after fitting, and the amplitude of the noise is pre-set by the diffusion model; splicing the noisy clothing, clothing area mask and model reserved area together and inputting them into the diffusion model for training; the human body shape and posture are added to the above input after passing through a multi-layer convolutional network for training together; combined with the clothing feature model for training; the loss function of the diffusion model includes but is not limited to predicting the mean square error between the added noise and the real noise, so that the model can learn the denoising process and can gradually remove noise from the initial completely random noise to achieve the effect of controllable image generation.

9. The virtual fitting method based on diffusion model and maintaining local details according to claim 5, characterized in that: The dual-stream structure is achieved by adopting the DWPOSE dual-stream weight sharing human posture estimation method.

10. The virtual fitting method based on diffusion model and maintaining local details according to claim 5, characterized in that: In step 5, the generated virtual fitting effect image needs to be post-processed to improve the image quality. Specifically: the generated image is alpha-fused with the original image, and the fused mask is the clothing area mask, so that the reserved area can be closer to the reserved area of ​​the original image, while retaining the generated clothing area; the stable diffusion series model is used, and the input image is the latent image after VAE processing, and it passes through the VAE decoder when the final raw image effect is displayed.

Images