Sneaker appearance design generation method based on diffusion model
Through the sports shoe appearance design generation method based on diffusion model, using asymmetric gradient guidance and condition masks, the problems of unsatisfactory visual effects and low resolution during style conversion in the prior art are solved, and a more stable and realistic The appearance of sneakers is generated to improve the quality and consistency of the generated results.
Patent Information
- Application Number
- CN202411183155.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-06
AI Technical Summary
The existing sneaker appearance design generation method is difficult to maintain the required style of the input fashion item's base design and integration during style conversion. The visual effect of the generated results is not ideal, the resolution is low, and the appearance and shape of the clothing are lacking. control.
Using a sports shoe appearance design generation method based on diffusion model, a conditional mask and loss function are introduced through asymmetric gradient guidance (AGG) reverse diffusion process to ensure the visual design area and appearance semantic consistency of the generated results, and the DDPM model is used for Coding and denoising processing optimizes the forward diffusion process of structural images.
A more stable and realistic sneaker appearance generation result was achieved, improving the authenticity and stability of the generated result. The FID score reached 103.4, the LPIPS score was better than the existing method, and both structural loss and appearance loss were improved.
Smart Images

Figure CN119089783A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of computer-aided design of footwear products, and in particular relates to a method for generating sports shoe appearance designs based on a diffusion model. Background Art
[0002] In the wave of the intelligent era, the country is actively developing the industrial Internet, promoting intelligent manufacturing, and accelerating the digital transformation of the modern apparel industry. As an important product in the apparel industry, the market for sports shoes has developed rapidly, with an average annual compound growth rate of 20.6%. The production cycle of sports shoes is long, and the design is prone to lag. With the improvement of people's material level, there are higher requirements for sports shoes. Comfort alone cannot meet people's needs. The demand has begun to shift from the original functionality to the direction of spiritual culture, so the fashion attributes of sports shoes have also begun to be fully explored. Nowadays, whether it is the general public or sports stars, they will choose fashionable and practical sports shoes to wear, so as to reflect their own fashion taste.
[0003] Appearance design elements are an important manifestation of the trendy attributes of sports shoes. At present, AIGC-based sports shoe intelligent design platforms such as Vali mainly provide services such as color optimization, single-style extension design, multiple-style fusion design, cultural and creative styles, and style customization. Its technical implementation path mainly relies on the rapid development of image generation, operation, and enhancement methods of complex models. These models provide valuable guidance and inspiration for fashion designers. Most image-guided generative models are based on style transfer, which is a branch of neural style transfer. It takes a fashion style image as the goal and generates an image that reflects the desired fashion style. The main challenge is to retain the underlying design of the input fashion item during style transfer while seamlessly integrating the desired style. Among them, Generative Adversarial Network (GAN) and Diffusion Model have made significant developments and impacts in the field of image style transfer. They provide designers, artists, and researchers with powerful tools and methods to generate images with unique styles and artistic qualities. At the same time, they have also stimulated the research and development of computer vision and deep learning, promoted the understanding and exploration of image generation and processing, and created a technical background for the emergence of image-guided synthesis models.
[0004] The pioneering work in this field is that Jiang et al. proposed an end-to-end feedforward neural network consisting of a fashion style generator G and a discriminator D based on GAN. The global and patch-based style and content losses calculated by the discriminator are alternately back-propagated to the generation network for optimization. In the global optimization stage, the shape and design of the clothing are preserved, while the local optimization stage retains the detailed style patterns. Although this model performs better than previous related global or patch-based neural style transfer works, it still has certain limitations. When the texture in the input image is not prominent, the output result visual effect is not ideal. In addition, the model still retains some original colors from the content image, and the resolution of the generated clothing is relatively low. Subsequently, many researchers proposed GAN-based generative models, such as the SC-FashionG framework for mix-and-match style generation, multimodal unsupervised image-to-image translation (MUNIT), a texture and shape decoupled generative adversarial network (TSD-GAN), a generative adversarial network with heatmap-guided semantic decoupling (HSDGAN), and an intelligent design method based on the FadGAN model. These GAN-based methods can be preliminarily applied to clothing design to generate new clothes. However, these methods lack control over the appearance and shape of clothing when converting from non-fashion domain images. Cao, Chai, et al. proposed a reference-based clothing design that generates more realistic images through structure-aware transfer through a diffusion model, called DiffFashion. It can semantically generate new clothes from the provided clothing image and the reference appearance image. Due to the randomness of diffusion, the mask cannot guarantee a good effect every time, and better masks need to be generated to improve the task. In summary, although the existing methods generate clothing that is closer to human designers than computer-designed clothing, the generation process is relatively random and lacks a specific design direction. Summary of the invention
[0005] The purpose of the present invention is to provide a method for generating sports shoe appearance design based on a diffusion model, a method for generating sports shoe appearance by asymmetric gradient guided (AGG) back diffusion, our method can further maintain the consistency of visual design area and appearance semantics during the synthesis process, and obtain a more stable and realistic appearance generation result.
[0006] In order to solve the above technical problems, the following technical solutions are adopted:
[0007] A method for generating sports shoe appearance design based on a diffusion model, characterized by comprising the following steps:
[0008] (1) Import the DDPM model for encoding;
[0009] (2) Introducing asymmetric gradient guidance;
[0010] (3) Generate conditional mask;
[0011] (4) Optimizing the forward diffusion process of the structural image by the conditional mask generated in step (3):
[0012] (5) Loss function construction.
[0013] The method for generating sports shoe appearance design based on a diffusion model according to claim 1 is characterized in that: in step (1), the original sports shoe image x 0 In the input denoising diffusion probability model DDPM, Gaussian noise is gradually added to the distribution during the forward diffusion process training. The standard deviation of the Gaussian noise is calculated based on a fixed value β t Determined, the mean is determined by the fixed value β t and the current state of t is determined.
[0014] The method for generating sports shoe appearance design based on a diffusion model according to claim 2 is characterized in that: the forward diffusion process training is used to estimate the noise image x generated by the Markovian forward process t The additional noise in; Assume that the distribution of the real image at the initial moment is q(x 0 ), each step is done by adding t-1 Add Gaussian noise to get x t , then the transfer process is expressed as:
[0015]
[0016] From the input x 0 to x T The posterior probability distribution of is expressed as:
[0017]
[0018] Where α t =1-β t , In the reverse process, the diffusion model learns to reconstruct the data by gradually denoising it. t Predict x t-1 , through the learned noise prediction network ∈ β To invert the Gaussian transition, as follows:
[0019]
[0020] After optimization, in order to obtain better image quality, the neural network converts the sample x tand timestamp t as input and added to x in the forward prediction process t-1 The noise in x is not directly predicted t-1 The denoising process can be defined as:
[0021]
[0022] in, σ 2 is the variance when the value is fixed at β in DDPM; during the iteration process, from the initial sample For image x 0 Take samples.
[0023] After optimization, the reverse sampling process in step (2) and formula (4) uses the manifold constraint gradient and Tweedie formula to transform the noise image x t Project to the estimated clean image On this basis, optimization is performed by decomposing diffusion sampling, and:
[0024]
[0025] According to formula (5), the optimized reverse sampling step is optimized as follows:
[0026]
[0027] Specifically, the original forward model of formula (2) can be converted into:
[0028]
[0029] in,
[0030] After optimization, the reverse sampling process is optimized using the variational method and substituted into get:
[0031]
[0032] Here, for s t Conduct a conditional diffusion model guide and set Based on the equivalence of sample update and diffusion model update, the reverse sampling calculation guided by AGG is obtained as follows:
[0033]
[0034] in for:
[0035]
[0036] After optimization, the step (3) converts the original image Input into the diffusion model, after DDPM encoding in the forward diffusion process, the image latent value is obtained in half the time of reverse diffusion Denote the foreground label as y p , represents the foreground sports shoe object; then, the noise map of the foreground sports shoe is obtained by the following formula (11):
[0037]
[0038] in, is an estimate of the source image, from The prediction is obtained, that is
[0039]
[0040] Denote the non-foreground label as y n , represents a negative object; N different non-foreground label conditions are used to obtain the average noise map, namely:
[0041]
[0042] Among them, i∈[1,…,N}. Two noise mappings M can be obtained p and M n The difference between them; then a threshold is set for binarization, which returns an editable semantic mask M for the foreground clothing area.
[0043] After optimization, the step (4) first uses the appearance image The DDPM forward processing is then performed. Mask-guided denoising is then performed; specifically, at each step of the denoising process, a new prediction is estimated from the diffusion model. as follows:
[0044]
[0045] Then transfer the foreground appearance With the corresponding timestamp The mask M generated by the clothing image is combined, that is,
[0046]
[0047] where ω mix is the mixture ratio of appearance images and clothing images.
[0048] After optimization, the loss function in step (5) consists of two parts: appearance loss for image domain change and a regularization loss for preserving the source image structure The total loss value is the sum of the two parts, as shown in formula (16)
[0049]
[0050] Among them, λ str and λ app is the structural loss coefficient and the appearance loss coefficient, through the control parameter λ str To control the structure preservation strength.
[0051] After optimization, the appearance loss The [CLS] token matching loss using DINO-ViT is shown in formula (17):
[0052]
[0053] in, is the last layer [CLS] token, λ MSE is the global statistical loss coefficient between images; in order to better utilize the appearance between the object and the appearance image, the object semantic mask M is used to remove the For background pixels, only the appearance loss of objects within the mask is calculated;
[0054] For structural losses The i-th key vector of the l-th attention layer of the ViT model is used, denoted as The structural information of the i-th block of the original image is guided as follows:
[0055]
[0056] sim(·,·) is the normalized cosine similarity index value, that is
[0057]
[0058] τ is the temperature parameter. Thus, the loss between keys at the same position of the two images is minimized, while the loss between keys at different positions is maximized.
[0059] The above technical solution has the following beneficial effects:
[0060] The present invention proposes a method for generating sports shoe appearance design based on a diffusion model. The method adopts an asymmetric gradient guided (AGG) back diffusion process so that the output follows the properties of the source image to prevent significant deviation. The method uses a Mask to ensure the consistency of the structure in the diffusion process, and uses the intermediate product stored in the forward DDPM step to bridge the gap between the two in the denoising process, so that the latent value of the reference image gradually adapts to the structural characteristics of the sports shoe image. This further maintains the consistency of the visual design area and the appearance semantics in the synthesis process, and obtains a more stable and realistic generation result.
[0061] Compared with the prior art, the main contributions of the present invention can be summarized as follows:
[0062] 1. A sports shoe appearance design method based on the probability diffusion model is proposed. This method can simultaneously respond to the characteristics of the external structure and the internal design area, and realize the generation of realistic fashion images in an unsupervised state. The realistic evaluation FID (Frechet initial distance score) value of the generated results can reach 103.4, which is better than other existing methods.
[0063] 2. A transfer strategy based on asymmetric gradient guidance is introduced to gradually adapt the potential signal of the appearance image to the output clothing image, solving the problem of large domain gaps, guiding the output to follow the properties of the source image, preventing significant deviations, and preventing the processed clean image from deviating from the original manifold, improving stability. In the similarity score, its LPIPS score (perceptual image patch similarity, LPIPS) score value, the appearance loss value L.app is 0.726, and the structure loss value L.Str is 0.469, both of which are better than existing methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] The present invention will be further described below in conjunction with the accompanying drawings:
[0065] Figure 1 A diagram is generated for the appearance design of the sports shoes of the present invention. DETAILED DESCRIPTION
[0066] The present invention aims to provide a method for generating sports shoe appearance design based on a diffusion model, which adopts an asymmetric gradient guided (AGG) back diffusion process so that the output follows the properties of the source image to prevent significant deviations; a Mask is used to ensure the consistency of the structure in the diffusion process, and the intermediate product stored in the forward DDPM step is used to bridge the gap between the two in the denoising process, so that the latent value of the reference image gradually adapts to the structural characteristics of the sports shoe image; thereby further maintaining the consistency of the visual design area and the appearance semantics in the synthesis process, and obtaining a more stable and realistic generation result.
[0067] The present invention comprises the following steps:
[0068] (1) Import the DDPM model for encoding: 0 In the input denoising diffusion probability model DDPM, Gaussian noise is gradually added to the distribution during the forward diffusion process training. The standard deviation of the Gaussian noise is calculated based on a fixed value β t Determined, the mean is determined by the fixed value β t The forward diffusion process training is used to estimate the noise image x generated by the Markovian forward process. tThe additional noise in; Assume that the distribution of the real image at the initial moment is q(x 0 ), each step is done by adding t-1 Add Gaussian noise to get x t , then the transfer process is expressed as:
[0069]
[0070] From the input x 0 to x T The posterior probability distribution of is expressed as:
[0071]
[0072] Where α t =1-β t , In the reverse process, the diffusion model learns to reconstruct the data by gradually denoising it. t Predict x t-1 , through the learned noise prediction network ∈ θ To invert the Gaussian transition, as follows:
[0073]
[0074] To obtain better image quality, the neural network transforms the sample x t and timestamp t as input and added to x in the forward prediction process t-1 The noise in x is not directly predicted t-1 The denoising process can be defined as:
[0075]
[0076] in, σ 2 is the variance when the value is fixed at β in DDPM; during the iteration process, from the initial sample For image x 0 Take samples.
[0077] (2) Introducing asymmetric gradient guidance: In the reverse sampling process in formula (4), the noisy image x is transformed into t Project to the estimated clean image On this basis, optimization is performed through decomposed diffusion sampling (DDS), and the following is made:
[0078]
[0079] According to formula (5), the optimized reverse sampling step is optimized as follows:
[0080] Specifically, the original forward model of formula (2) can be converted into:
[0081]
[0082] in,
[0083] Using the variational method to optimize the reverse sampling process, substitute get:
[0084]
[0085] Here, for s t Conduct a conditional diffusion model guide and set
[0086] Based on the equivalence of sample update and diffusion model update, the reverse sampling calculation guided by AGG is obtained as follows:
[0087]
[0088] in for:
[0089]
[0090] (3) Generate conditional mask: transform the original image Input into the diffusion model, after DDPM encoding in the forward diffusion process, the image latent value is obtained in half the time of reverse diffusion Denote the foreground label as y p , represents the foreground sports shoe object; then, the noise map of the foreground sports shoe is obtained by the following formula (11):
[0091]
[0092] in, is an estimate of the source image, from The prediction is obtained, that is
[0093]
[0094] Denote the non-foreground label as y n , represents a negative object; N different non-foreground label conditions are used to obtain the average noise map, namely:
[0095]
[0096] Among them, i∈{1,…,N}. Two noise mappings M can be obtained p and M n The difference between them; then a threshold is set for binarization, which returns an editable semantic mask M for the foreground clothing area.
[0097] (4) Optimizing the forward diffusion process of the structure image by the conditional mask generated in step (3): First, use the appearance image The DDPM forward processing is then performed. Mask-guided denoising is then performed; specifically, at each step of the denoising process, a new prediction is estimated from the diffusion model. as follows:
[0098]
[0099] Then transfer the foreground appearance With the corresponding timestamp The mask M generated by the clothing image is combined, that is,
[0100]
[0101] where ω mix is the mixed ratio of the appearance image and the clothing image. This change ensures that the appearance information in the mask is transferred while other structural information remains consistent with the clothing image.
[0102] (5) Loss function construction: The loss function consists of two parts: appearance loss for image domain changes and a regularization loss for preserving the source image structure The total loss value is the sum of the two parts, as shown in formula (16)
[0103]
[0104] Among them, λ str and λ app is the structural loss coefficient and the appearance loss coefficient, and the control parameter λ str To control the structure preservation strength.
[0105] Loss of appearance The [CLS] token matching loss using DINO-ViT is shown in formula (17):
[0106]
[0107] in, is the last layer [CLS] token, λ MSE is the global statistical loss coefficient between images; in order to better utilize the appearance between the object and the appearance image, the object semantic mask M is used to remove the For background pixels, only the appearance loss of objects within the mask is calculated;
[0108] For structural losses The i-th key vector of the l-th attention layer of the ViT model is used, denoted as The structural information of the i-th block of the original image is guided as follows:
[0109]
[0110] sim(·,·) is the normalized cosine similarity index value, that is
[0111]
[0112] The technical solution of the present invention is described in detail below in conjunction with specific embodiments:
[0113] In all experiments, the diffusion step size T = 60 and the number of resampling repetitions N = 10 are used. In the mask generation part, the binarization threshold is set to -0.2. In the guidance part, mask guidance is used in the first 50% of the steps of the denoising stage, the mixing ratio is set to 0.98, and the appearance loss coefficient λ app Set to 0.1, the structural loss coefficient λ str is set to 1. For the AGG optimization part, we use the Adam optimizer with a learning rate of 0.02. Only two steps of DDS optimization are used. For all experiments, a moderate random noise, σ t =0.8. In a single 32G RTX 3090 video memory, Figure 1 shown.
[0114] The above are only specific embodiments of the present invention, but the technical features of the present invention are not limited thereto. Any simple changes, equivalent replacements or modifications made based on the present invention to solve basically the same technical problems and achieve basically the same technical effects are all included in the protection scope of the present invention.
Claims
1. A method for generating sports shoe appearance design based on diffusion model, Features The steps include: (1) Import the DDPM model for encoding; (2) Introducing asymmetric gradient guidance; (3) Generate conditional mask; (4) Optimizing the forward diffusion process of the structural image by the conditional mask generated in step (3): (5) Loss function construction.
2. The method for generating sports shoe appearance design based on a diffusion model according to claim 1, Features: In step (1), the original sports shoe image x 0 In the input denoising diffusion probability model DDPM, Gaussian noise is gradually added to the distribution during the forward diffusion process training. The standard deviation of the Gaussian noise is calculated based on a fixed value β t Determined, the mean is determined by the fixed value β t and the current state of t is determined.
3. The method for generating sports shoe appearance design based on a diffusion model according to claim 2, Features: The forward diffusion process is trained to estimate the noise image x generated by the Markovian forward process. t Additional noise in Assume that the distribution of the real image at the initial moment is q(x 0 ), each step is done by adding t-1 Add Gaussian noise to get x t , then the transfer process is expressed as: From the input x 0 to x T The posterior probability distribution of is expressed as: Where α t =1-β t , In the reverse process, the diffusion model learns to reconstruct the data by gradually denoising it. t Predict x t-1 , through the learned noise prediction network ∈ θ To invert the Gaussian transition, as follows:
4. The method for generating sports shoe appearance design based on a diffusion model according to claim 3, Features: To obtain better image quality, the neural network transforms the sample x t and timestamp t as input and added to x in the forward prediction process t-1 The noise in x is not directly predicted t-1 The mean of The denoising process can be defined as: in, σ 2 is the variance when the value is fixed at β in DDPM; during the iteration process, from the initial sample For image x 0 Take samples.
5. The method for generating sports shoe appearance design based on diffusion model according to claim 4, Features: The reverse sampling process in step (2) and formula (4) uses the manifold constraint gradient and Tweedie formula to convert the noise image x t Project to the estimated clean image On this basis, optimization is performed by decomposing diffusion sampling, and: According to formula (5), the optimized reverse sampling step is optimized as follows: Specifically, the original forward model of formula (2) can be converted into: in, 6. The method for generating sports shoe appearance design based on diffusion model according to claim 5, Features: Using the variational method to optimize the reverse sampling process, substitute get: Here, for s t Conduct conditional diffusion model guidance and set Based on the equivalence of sample update and diffusion model update, the reverse sampling calculation guided by AGG is obtained as follows: in for:
7. The method for generating sports shoe appearance design based on a diffusion model according to claim 1, Features: In step (3), the original image Input into the diffusion model, after DDPM encoding in the forward diffusion process, the image latent value is obtained in half the time of reverse diffusion Denote the foreground label as y p , represents the foreground sports shoe object; then, the noise map of the foreground sports shoe is obtained by the following formula (11): in, is an estimate of the source image, from The prediction is obtained, that is Denote the non-foreground label as y n , represents a negative object; N different non-foreground label conditions are used to obtain the average noise map, namely: Where i∈{1,...,N}. Two noise mappings M can be obtained. p and M n The difference between them; then a threshold is set for binarization, which returns an editable semantic mask M for the foreground clothing area.
8. The method for generating sports shoe appearance design based on diffusion model according to claim 7, Features: In step (4), first use the appearance image The DDPM forward processing is then performed. Mask-guided denoising is then performed; specifically, at each step of the denoising process, a new prediction is estimated from the diffusion model. as follows: Then transfer the foreground appearance With the corresponding timestamp The mask M generated by the clothing image is combined, that is, where ω mix is the mixture ratio of appearance images and clothing images.
9. The method for generating sports shoe appearance design based on diffusion model according to claim 7, Features: In step (5), the loss function consists of two parts: appearance loss for image domain change and a regularization loss for preserving the source image structure The total loss value is the sum of the two parts, as shown in formula (16) Among them, λ str and λ app is the structural loss coefficient and the appearance loss coefficient, through the control parameter λ str To control the structure preservation strength.
10. The method for generating sports shoe appearance design based on diffusion model according to claim 9, Features: Loss of appearance The [CLS] token matching loss using DINO-ViT is shown in formula (17): in, is the last layer [CLS] token, λ MSE is the global statistical loss coefficient between images; in order to better utilize the appearance between the object and the appearance image, the object semantic mask M is used to remove the For background pixels, only the appearance loss of objects within the mask is calculated; For structural losses The i-th key vector of the l-th attention layer of the ViT model is used, denoted as The structural information of the i-th block of the original image is guided as follows: sim(·,·) is the normalized cosine similarity index value, that is, τ is the temperature parameter.
