Model virtual fitting effect picture generation method and device based on diffusion model

Through the model virtual fitting effect drawing generation method based on diffusion model, the problem that users cannot preview the design effect in the online design platform is solved, intuitive design preview and low-cost access to the on-demand printing platform are realized, and user experience and platform are improved. competitiveness.

CN118941669APending Publication Date: 2024-11-12GUANGDONG INST OF SCI & TECH
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411084493.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing online design platform lacks intuitive visual effects display, and users cannot see the actual effect of the design on the model before the design is submitted, resulting in poor user experience and increased rework costs and time.

Method used

A model virtual fitting effect generation method based on diffusion model is adopted. Through image segmentation algorithm and diffusion model training, a virtual fitting effect of the model wearing a design pattern is generated, allowing users to intuitively preview the design effect.

Benefits of technology

Reduces rework costs and time, improves user experience and satisfaction, and provides personalized display and low-cost access to on-demand printing platforms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941669A_ABST
    Figure CN118941669A_ABST
Patent Text Reader

Abstract

The invention discloses a model virtual fitting effect picture generation method and device based on a diffusion model, and belongs to the technical field of image processing. The method comprises the following steps: collecting a clothing product design image and a model fitting image corresponding to the product design image in a real scene; segmenting a clothes area of the model fitting image through an image segmentation algorithm, marking the clothes area, and generating a mask model image; constructing a virtual fitting model based on the diffusion model; training the virtual fitting model by using the acquired product design image, the model fitting image and the mask model image; and inputting the target product design image and any model fitting image into the trained virtual fitting model to generate a model virtual fitting effect picture of the target product design image. By utilizing the method and the device, a user can check the intelligently generated virtual fitting effect picture of the model wearing the corresponding design pattern by providing the clothes design pattern and selecting different display models, so that the reworking cost is reduced, and the reworking time is shortened.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for generating a model virtual fitting effect picture based on a diffusion model. Background Art

[0002] At present, the main users of the on-demand printing platform are small and medium-sized e-commerce sellers. The platform provides users with online product design services, and uploads the user's product renderings and basic information to their personal online store with one click. Users can sell products first, and after receiving orders, they can directly place orders on the platform, and the platform will arrange production and delivery.

[0003] For the current on-demand printing online design platforms in the market, users can design personalized clothing patterns on these platforms and submit the designs to the platforms for printing. Existing online design platforms often lack intuitive visualization of the effects. Users cannot see the actual effects of the designs on the models before submitting the designs, and can only judge the design effects through flat images or simple simulations. This leads to a poor user experience and may increase unnecessary rework costs and time. Summary of the invention

[0004] In view of this, the present invention provides a method and device for generating virtual fitting renderings of models based on a diffusion model, which at least solves some of the above technical problems, and is suitable for on-demand printing services of online design platforms. With the present invention, users can view intelligently generated virtual fitting renderings of models wearing corresponding design patterns by providing personalized clothing design patterns and selecting different display models, thereby intuitively previewing the design effects, so that users can fully preview and adjust the design before submitting a printing order, reducing rework costs and time, and improving user experience and satisfaction.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] In a first aspect, an embodiment of the present invention provides a method for generating a model virtual fitting effect diagram based on a diffusion model, the method comprising:

[0007] Collecting product design images of clothing and model fitting images corresponding to the product design images in real scenes; preprocessing the collected image data, segmenting and marking the clothing area of ​​the model fitting image through an image segmentation algorithm, and generating a mask model image;

[0008] Constructing a virtual fitting model based on the diffusion model; and training the virtual fitting model using the collected product design images, model fitting images, and mask model images;

[0009] The target product design image and any model fitting image are input into the trained virtual fitting model to generate a model virtual fitting effect image of the target product design image.

[0010] In an optional embodiment, the image segmentation algorithm includes a posture estimation algorithm and a human body analysis algorithm; the model fitting image is analyzed by the posture estimation algorithm and the human body analysis algorithm, the areas corresponding to the main body and arms in the image are separately marked, and the original image is subjected to regional masking processing using the marks to generate a pre-processed masked model image.

[0011] In an optional embodiment, the training objective function of the virtual fitting model is:

[0012]

[0013] Where ζ=π(x m )+φ(x)+∈,x m , x represents the mask model image and the model fitting image respectively, π is the mask encoder, φ is the VAE encoder, ∈ is Gaussian noise, x c represents the product design image, t represents the step index, ψ is the CLIP encoder, θ is the UNet trainable parameter, ∈ θ Output of UNet.

[0014] In an optional embodiment, the constructed virtual fitting model includes two branches, wherein:

[0015] The input of the first branch is the product design image. The first branch uses VAE encoder, CLIP encoder and UNet to extract the features of the product design image. The output of each layer of UNet in the first branch is used as the Key and Value features of the spatial attention module of UNet in the second branch.

[0016] The input of the second branch is the model fitting image and the mask model image. In the second branch, the VAE encoder is used to first encode the model fitting image to obtain the corresponding feature map φ(x); at the same time, a mask encoder is constructed to extract features from the mask model image and output a feature π(x) of the same size as the feature map φ(x). m ), and finally φ(x), π(x m ) and Gaussian noise∈ are added and the result is input to UNet.

[0017] In an optional implementation, the parameters of the mask encoder and the UNet in the second branch are trained, and the remaining parameters are fixed.

[0018] In an optional embodiment, the trainable parameters are derived according to the back-propagation algorithm, and the parameters are updated using the stochastic gradient descent method until the model converges.

[0019] In a second aspect, an embodiment of the present invention further provides a device for generating a model virtual fitting effect picture based on a diffusion model, comprising: an image acquisition device, a memory and a processor, wherein:

[0020] The image acquisition device is used to collect product design images of clothing and model fitting images corresponding to the product design images in real scenes;

[0021] The memory stores computer instructions;

[0022] The processor is data-connected to the image acquisition device and the memory. The processor executes the computer instructions to execute the above-mentioned method for generating a model virtual fitting rendering based on a diffusion model, thereby generating a model virtual fitting rendering of a target product design image.

[0023] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method for generating a model virtual fitting effect diagram based on a diffusion model is implemented.

[0024] Compared with the prior art, the present invention has at least the following beneficial effects:

[0025] The present invention provides a method and device for generating virtual fitting effect pictures of models based on a diffusion model, which is suitable for on-demand printing services of online design platforms. With the present invention, users can view intelligently generated virtual fitting effect pictures of models wearing corresponding design patterns by providing personalized clothing design patterns and selecting different display models, thereby intuitively previewing the design effect, so that users can fully preview and adjust the design before submitting a printing order, reducing rework costs and time, and improving user experience and satisfaction.

[0026] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0027] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0030] Figure 1 A schematic flow chart of a method for generating a model virtual fitting effect diagram based on a diffusion model provided in an embodiment of the present invention.

[0031] Figure 2 A schematic diagram of clothing region segmentation and labeling of a model image provided by an embodiment of the present invention.

[0032] Figure 3 A schematic diagram of the framework structure of a virtual fitting model provided in an embodiment of the present invention.

[0033] Figure 4 A schematic diagram of the application effect of the method provided in an embodiment of the present invention.

[0034] Figure 5 A schematic structural diagram of a device for generating a model virtual fitting effect diagram based on a diffusion model provided in an embodiment of the present invention.

[0035] Among them, 1-image acquisition device; 2-memory; 3-processor. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0037] In the description of the present invention, it should be noted that: in some processes described in the specification and drawings of this application, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be performed in the order in which they appear in this document or may be performed in parallel. In addition, various serial numbers are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0038] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] See also Figure 1 As shown, the present invention provides a method for generating virtual fitting effect pictures of models based on a diffusion model, which is suitable for on-demand printing services of online design platforms. Through this method, users can provide personalized clothing design patterns and select different display models to view the intelligently generated effect pictures of models wearing clothes with corresponding design patterns, thereby intuitively previewing the design effects and providing manufacturers on the product supply side with a low-cost way to access the on-demand printing platform. The method process mainly includes:

[0040] 1. Dataset collection and preprocessing for model training. Under the premise of complying with the commercial terms and conditions, a large amount of data is crawled from the Internet, including design patterns of clothing products and images of models trying on clothes corresponding to the products in real scenes. Data preprocessing aims to segment and mark the clothing area of ​​the model image through the image segmentation algorithm to generate a marked image (masked model image).

[0041] 2. Build a virtual fitting model based on the diffusion model and perform model training. During the training process, the labeled image, product design image, and model fitting image are input to the model together, and the model outputs a result image of the same size as the model image (model fitting image). The model parameters are updated through the gradient descent algorithm until the model converges, so that the output result image can reconstruct the original input model image.

[0042] 3. Synthesis of renderings. After the model is trained, users can input product design patterns and any model images, and the model will generate renderings of the model trying on clothes virtually.

[0043] The specific implementation methods of the method of the present invention are described in detail below:

[0044] 1. Dataset collection and preprocessing:

[0045] In the embodiment of the present invention, training data is first obtained from a legal channel, including various clothing product design patterns and corresponding images of models trying on the products in real scenes. For example, the simplest and most direct way in this embodiment is to collect from existing online clothing product websites. This collection step requires obtaining image authorization from the product website.

[0046] Furthermore, the model image is analyzed using the existing pose estimation algorithm and human parsing algorithm. Figure 2 The regions corresponding to the body and arms in the human body segmentation map are marked separately, and the marks are used to perform regional mask processing on the original image to generate a masked model image after preprocessing, as shown in Figure 2 As shown in the rightmost picture.

[0047] 2. Model construction and training;

[0048] In the present invention, a virtual fitting model is constructed based on a diffusion model; the constructed virtual fitting model framework and training process refer to Figure 3 As shown, in this embodiment of the present invention, the training objective function of the model is:

[0049]

[0050] Where ζ=π(x m )+φ(x)+∈,x m , x represents the mask model image and the model fitting image respectively, π is the mask encoder, φ is the VAE encoder, ∈ is Gaussian noise, x c represents the product design image, t represents the step index, ψ is the CLIP encoder, θ is the UNet trainable parameter, ∈ θ Output of UNet.

[0051] In a specific embodiment, the model framework of the virtual fitting model is constructed based on the Stable Diffusion model. The model has two branches, of which only the parameters of the mask encoder and the UNet in the second branch can be trained, and the remaining parameters are fixed; such a setting, first, fixing the remaining parameters will not affect the performance of the model, second, it is conducive to preventing the model from forgetting the inherent feature expression ability of the model due to too many training parameters during the training process, and third, the training calculation amount is greatly reduced due to the reduction of training parameters, which improves training efficiency and reduces dependence on computing power. All parameters are initialized from the trained Stable Diffusion model. The input of the first branch is the product design drawing, and the input of the second branch is the model picture and the mask model picture.

[0052] In the first branch, the VAE encoder, CLIP encoder and UNet in the Stable Diffusion model are partially copied to extract the features of the product design drawing. The output of each layer of UNet in this branch is used as the Key and Value features of the spatial attention module of UNet in the second branch; in this way, the features in the two branches can be fused and interacted with each other.

[0053] In the second branch, the VAE encoder of the Stable Diffusion model is also used to encode the features of the model fitting pictures to obtain the corresponding feature map φ(x). At the same time, a lightweight convolution module π( Figure 3 The mask encoder in the mask model image extracts features and outputs features π(x) of the same size as φ(x). m ), and finally φ(x), π(x m ) and Gaussian noise∈ are added and the result is input to UNet.

[0054] The formula corresponding to the spatial attention module in UNet is:

[0055]

[0056] Among them, f is the output of spatial attention, Atten(x q ,x v ,x k ) is the original spatial attention module in UNet, where query, value and key all come from the features generated by UNet itself. It means replacing the key and value in the original spatial attention module with the features obtained from the first branch.

[0057] In the embodiment of the present invention, after the loss function is calculated, the trainable parameters are derived according to the back propagation algorithm, and the parameters are updated using the stochastic gradient descent method until the model converges. The training part of the model only needs to be performed once, and more computing power can be used for better training during training. After the training is completed, the computing power resources used are used for the synthesis of the effect map.

[0058] 3. Synthesis of effect map:

[0059] In the embodiment of the present invention, for the model fitting image uploaded by the user, it is first necessary to obtain a mask model image using the method in the preprocessing process. Next, the product design drawing, the model fitting image, and the mask model image obtained above are Figure 1 The virtual fitting model is inputted from the beginning. The virtual fitting effect image generation step follows the image generation process of the Stable Diffusion model. The difference from the forward process of the training process is that the input of the second branch is π(x m )+∈, after the model obtains the final UNet output, the VAE decoder is used to decode the features into a virtual fitting effect image. Figure 4 The effect of the method of the present invention is demonstrated in that two target style design images are input respectively, and the virtual fitting model intelligently synthesizes the corresponding virtual fitting effect pictures.

[0060] From the description of the above embodiments, those skilled in the art can know that the present invention proposes a method for intelligently generating a virtual fitting rendering of a model based on an artificial intelligence algorithm of a conditional diffusion generation model (referred to as a diffusion model) and image segmentation technology. Using the method of the present invention, a user can select a designed product pattern and another model pattern when using the platform's online product design service. The platform uses computer vision technology to take the two patterns as input and intelligently generate a rendering of a virtual fitting of the corresponding model for the corresponding product. The specific advantages of the method of the present invention include:

[0061] Low cost: Supply-side manufacturers do not need to hire professional models. They can simply upload model photos and product design drawings, and use artificial intelligence algorithms to automatically generate virtual model try-on renderings of the product, and upload them to the platform for users to use.

[0062] Personalized display: Users can freely choose models and preview design effects to improve user participation and satisfaction.

[0063] Improve user experience: Through intuitive renderings, users can fully preview and adjust the design before submitting a printing order, reducing rework costs and time, and improving user experience and satisfaction.

[0064] Technological advancement: The use of advanced computer vision technology has improved the technical level and competitiveness of the online design platform.

[0065] Furthermore, the embodiment of the present invention also provides a device for generating a virtual model fitting effect diagram based on a diffusion model, which is used to generate a virtual model fitting effect diagram of a target product design image. The device can be implemented by software and / or hardware. Figure 5 As shown, the apparatus comprises an image acquisition device 1, a memory 2 and a processor 3. The image acquisition device 1, the memory 2 and the processor 3 may be connected via a bus or other means.

[0066] The image acquisition device 1 is used to collect product design images of clothing and model fitting images corresponding to the product design images in real scenes, and send them to the processor 3. The processor 3 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components and other chips, or a combination of the above chips.

[0067] The memory 2, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as programs or instructions corresponding to a method for generating a model virtual fitting effect diagram based on a diffusion model.

[0068] The processor 3 executes various functional applications and data processing of the processor by running the non-transitory software program or instructions stored in the memory 2, that is, implements the method for generating a model virtual fitting effect picture based on a diffusion model in the above method embodiment.

[0069] The memory 2 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created by the processor 2 and the like.

[0070] In addition, the memory 2 may include a high-speed random access memory and may also include a non-volatile memory. More specific examples of the memory 2 (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, and any suitable combination of the above.

[0071] In some aspects, the memory 2 optionally includes memories remotely located relative to the processor 3 , and these remote memories may be connected to the processor 3 via a network.

[0072] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices or computer program products, etc. Therefore, the present invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0073] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several distinct components, and by means of a suitably programmed computer.

[0074] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0075] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating virtual fitting effect pictures of models based on diffusion model. It is characterized in that The method includes: Collecting product design images of clothing and model fitting images corresponding to the product design images in real scenes; preprocessing the collected image data, segmenting and marking the clothing area of ​​the model fitting image through an image segmentation algorithm, and generating a mask model image; Constructing a virtual fitting model based on the diffusion model; and training the virtual fitting model using the collected product design images, model fitting images, and mask model images; The target product design image and any model fitting image are input into the trained virtual fitting model to generate a model virtual fitting effect image of the target product design image.

2. The method for generating a model virtual fitting effect diagram based on a diffusion model according to claim 1, It is characterized in that The image segmentation algorithm includes a posture estimation algorithm and a human body analysis algorithm; the model fitting image is analyzed by the posture estimation algorithm and the human body analysis algorithm, the areas corresponding to the body main body and the arms in the image are separately marked, and the original image is subjected to regional mask processing using the marks to generate a masked model image after preprocessing.

3. The method for generating a model virtual fitting effect diagram based on a diffusion model according to claim 1, It is characterized in that The training objective function of the virtual fitting model is: Where ζ=π(x m )+φ(x)+∈,x m , x represents the mask model image and the model fitting image respectively, π is the mask encoder, φ is the VAE encoder, ∈ is Gaussian noise, x c represents the product design image, t represents the step index, ψ is the CLIP encoder, θ is the UNet trainable parameter, ∈ θ Output of UNet.

4. The method for generating a model virtual fitting effect diagram based on a diffusion model according to claim 3, It is characterized in that The constructed virtual fitting model includes two branches, wherein: The input of the first branch is the product design image. The first branch uses VAE encoder, CLIP encoder and UNet to extract the features of the product design image. The output of each layer of UNet in the first branch is used as the Key and Value features of the spatial attention module of UNet in the second branch. The input of the second branch is the model fitting image and the mask model image. In the second branch, the VAE encoder is used to first encode the model fitting image to obtain the corresponding feature map φ(x); at the same time, a mask encoder is constructed to extract features from the mask model image and output a feature π(x) of the same size as the feature map φ(x). m ), and finally φ(x), π(x m ) and Gaussian noise∈ are added and the result is input to UNet.

5. The method for generating a model virtual fitting effect diagram based on a diffusion model according to claim 4, It is characterized in that The parameters of the mask encoder and the UNet in the second branch are trained, and the rest of the parameters are fixed.

6. The method for generating a model virtual fitting effect diagram based on a diffusion model according to claim 5, It is characterized in that The trainable parameters are derived according to the back-propagation algorithm, and the parameters are updated using the stochastic gradient descent method until the model converges.

7. A device for generating model virtual fitting effect pictures based on diffusion model, It is characterized in that include: An image acquisition device, a memory and a processor, wherein: The image acquisition device is used to collect product design images of clothing and model fitting images corresponding to the product design images in real scenes; The memory stores computer instructions; The processor is data-connected to the image acquisition device and the memory. The processor executes the computer instructions to execute the method for generating a model virtual fitting rendering based on a diffusion model as described in any one of claims 1 to 6, thereby generating a model virtual fitting rendering of a target product design image.

8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method for generating a model virtual fitting effect picture based on a diffusion model as described in any one of claims 1 to 6 is implemented.

Citation Information

Cited By

  • Model virtual fitting effect picture generation method and system

    CN120182543A

  • Fitting image generation method and related equipment

    CN121391427A

  • Methods and related equipment for generating fitting room images

    CN121391427B