Model training, try-on image generation method and electronic device

Abstract

Embodiments of the present application disclose a model training method, a try-on image generation method and an electronic device, including: obtaining a training data set, the training data set including multiple person images; decoupling a clothing image from the person image, wherein the decoupled clothing image is a clothing flat image, and the decoupled clothing image is used as a pseudo label of the corresponding person image; training a model by using the person image and the corresponding pseudo label in the training data set, wherein the model generates a try-on image of the decoupled clothing image on the original person image according to the input person image and the pseudo label, and the original person image is used to supervise the generation result. Through the embodiments of the present application, the generalization ability of the model in the real generation scene can be improved, and the training cost can be reduced.

Description

Technical Field

[0001] This application relates to the field of image generation technology, and in particular to model training, methods for generating try-on images, and electronic devices. Background Technology

[0002] For product information service systems (often referred to as "e-commerce platforms"), the display effect of clothing products directly affects user click-through rates, dwell time, add-to-cart rates, and final conversion rates. Especially in the women's clothing category, product images not only serve a basic display function, but also directly determine consumers' judgments on fit, fabric, texture, wearing effect, and styling.

[0003] In the daily operation of the platform, merchants typically need to prepare multiple sets of product images for each garment, including flat lay images, model images, main images, detail images, marketing images, and display images for different groups of people, poses, and scenarios. However, in actual business, many merchants, especially small and medium-sized merchants and cross-border sellers, often lack professional modeling skills and high-quality content production capabilities, resulting in the following common problems with clothing product images on the platform: The product display format is too simplistic and lacks the effect of models trying on the products; The efficiency of adding new products is low, and it is impossible to quickly generate image materials that meet the needs of platform activities and marketing. The costs of shooting, retouching, models, and venues are high, making it difficult to support the continuous updating of a massive number of SKUs; In cross-border scenarios, apparel products are numerous, require rapid updates, and exhibit significant differences in merchant levels. Relying on traditional manual photography and design processes is insufficient to meet the demands for large-scale, low-cost, high-quality, and rapid-response product content production. Furthermore, there is the challenge of creating localized display content tailored to different country sites and user preferences. Summary of the Invention

[0004] This application provides a model training method, a method for generating try-on images, and an electronic device, which can improve the model's generalization ability in real-world generated scenarios and reduce training costs.

[0005] This application provides the following solution: A model training method, comprising: Obtain a first training dataset, wherein the first training dataset includes multiple image pairs, the image pairs include clothing images and person images, the clothing worn by the person in the person image in the same image pair is the same clothing in the clothing image; the clothing image is a flat lay image of clothing. The model is trained in the first stage using the first training dataset. In the first stage of training, the model decouples clothing images from the input person images and supervises the decoupling results using clothing images from the original image pairs, so that the model can acquire the ability to decouple clothing images from person images; wherein, the decoupled clothing images are clothing tile images. Obtain a second training dataset, which includes multiple images of people. Use the model that has completed the first stage of training to decouple clothing images from the images of people, and use them as pseudo-labels for the corresponding images of people. The model is trained in the second stage using the images of people in the second training dataset and the corresponding pseudo-labels. In the second stage of training, the model generates a try-on image by putting the decoupled clothing image on the person in the original image based on the input image of people and the pseudo-labels, and supervises the generation result using the original image of people.

[0006] The training in the first stage also includes: the model generating a try-on image based on the input image pair, which puts the clothing image on the person in the image pair, and using the original person image in the image pair to supervise the generation result, so that the model can obtain the initial ability to generate try-on images. In the second stage of training, the model is trained using the images of people and their corresponding pseudo-labels from the second training dataset, building upon the model trained in the first stage.

[0007] In the first stage, through a shared architecture, the model acquires the ability to generate try-on images and the ability to decouple clothing images from person images during the same training process. Specifically, for the same image pair, a first input pair and a second input pair are constructed. The first input pair includes: the clothing image in the image pair and the image of the person after occlusion processing of the target editing area. The second input pair includes: a blank image and the image of the person in the same image pair corresponding to the first input pair. By inputting prompt text, the model performs the task of generating a try-on image based on the first input pair and supervises the generation result using the image of the person in the original image pair. It also performs the task of decoupling the clothing image from the image of the person based on the second input pair and supervises the decoupling result using the image of the clothing in the original image pair.

[0008] A model training method, comprising: Obtain a training dataset, which includes multiple images of people; The clothing image is decoupled from the person image, wherein the decoupled clothing image is a clothing tile image, and the decoupled clothing image is used as a pseudo-label for the corresponding person image. The model is trained using the images of people in the training dataset and their corresponding pseudo-labels. The model generates a try-on image by placing the decoupled clothing image onto the person in the original image, based on the input image of the person and the pseudo-labels. The original image of the person is used to supervise the generation of the result.

[0009] The method further includes the following steps prior to the above: Multiple image pairs are acquired, each image pair including an image of clothing and an image of a person. The clothing worn by the person in the image of the same image pair is the same clothing as the clothing in the clothing image. The clothing image is a flat lay image of the clothing. The model is initially trained using the multiple image pairs, wherein the model generates a try-on image by placing the clothing image onto the person in the image based on the person image and clothing image in the input image pair, and the person image is used to supervise the generation result. The step of training the model using the person images and corresponding pseudo-labels in the training dataset includes: Using the images of people and their corresponding pseudo-labels in the training dataset, the model is trained on the basis of the initial training.

[0010] The step of decoupling the clothing image from the person image includes: Using a pre-trained model capable of decoupling clothing images from person images, clothing images are decoupled from the person images; The similarity between clothing images in the decoupled clothing image domain of the person images is calculated by using a visual model, and / or the clothing integrity and background cleanliness of the decoupled clothing images are calculated. In order to filter out clothing images that do not meet the quality conditions through a multi-criteria filtering method, the person images and their corresponding pseudo-labels in the training dataset are determined.

[0011] A method for generating try-on images, comprising: Determine a first clothing image for which a try-on image needs to be generated, and an original person image, wherein the second clothing worn by the person in the original person image is different from the first clothing in the first clothing image; Based on the body parts to be tried on the first garment, a target editing area is determined in the original character image; wherein, the part of the original character image located within the target editing area is the part that needs to be modeled, and the part outside the target editing area is preserved; A try-on image of the first garment is generated within the target editing area using a pre-trained model; the pre-trained model is trained according to any one of claims 1 to 5.

[0012] The step of determining the target editing region in the original portrait image includes: Based on the body area to be tried on the first garment, a first editing area is determined in the original character image; Using a pre-trained model, a first-stage try-on image is generated for the first garment within the first editing area; Extract the outline of the first garment from the try-on images generated in the first stage, and determine the second editing area based on the area within the extracted garment outline; Extract the outline of the second garment from the original image of the person, and determine the third editing area based on the area within the extracted garment outline; The second editing region and the third editing region are superimposed, and the target editing region is determined in the original human image based on the superimposition result. The target editing region is smaller than the first editing region so that by shrinking the first editing region, more human body structure regions in the original human image are preserved. The pre-trained model is then used to generate a second-stage try-on image of the first garment in the target editing region to obtain the target try-on image.

[0013] The step of determining the first editing area in the original character image based on the body area to be tried on the first garment includes: Based on the body area to be tried on the first garment, a general area without garment boundaries, length, or tightness restrictions is determined in the original human image; Based on the outline of the second garment in the original figure image, the area of ​​the second garment is determined in the original figure image; The first editing area is determined by overlaying the general area with the area within the bounding box of the second clothing area.

[0014] This also includes: Based on the body parts to be tried on for the first garment, the outlines of key body parts are extracted from the original image of the person, and a fourth editing region is determined based on the outlines of the key body parts. The second, third, and fourth editing regions are then superimposed, and the target editing region is determined in the original image based on the superimposition result.

[0015] The step of determining the fourth editing region based on the contour of the key body parts includes: The area within the outline of the key body part is expanded within a preset range to determine the fourth editing area.

[0016] The pre-trained model is based on the diffusion model. The number of denoising loops performed during the first stage of generating fitting images is less than the number of denoising loops performed during the second stage of generating fitting images.

[0017] A method for generating try-on images, comprising: Determine a first clothing image for which a try-on image needs to be generated, and an original person image, wherein the second clothing worn by the person in the original person image is different from the first clothing in the first clothing image; Based on the body parts to be tried on the first garment, a first editing area is determined in the original character image; Using a pre-trained model, a first-stage try-on image is generated for the first garment within the first editing area; Extract the outline of the first garment from the try-on images generated in the first stage, and determine the second editing area based on the area within the extracted garment outline; Extract the outline of the second garment from the original image of the person, and determine the third editing area based on the area within the extracted garment outline; The second editing region and the third editing region are superimposed, and the target editing region is determined in the original human image based on the superimposition result. The target editing region is smaller than the first editing region so that by shrinking the first editing region, more human body structure regions in the original human image are preserved. The pre-trained model is then used to generate a second-stage try-on image in the target editing region to obtain the target try-on image.

[0018] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the preceding methods.

[0019] An electronic device, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of any of the preceding methods.

[0020] A computer program product includes a computer program / computer executable instructions that, when executed by a processor in an electronic device, implement the steps of any of the preceding methods.

[0021] According to the specific embodiments provided in this application, the following technical effects are disclosed: In this embodiment, the model training process is divided into two stages. The first stage uses strictly paired image pairs to train the model, enabling it to decouple clothing images from person images. Then, a second training dataset is obtained, which only includes person images and does not need to include strictly paired clothing images. However, the clothing images decoupled from the person images can be extracted using the model trained in the first stage, serving as pseudo-labels for the person images. Next, the model is trained in the second stage using these pseudo-labeled person images. The model generates "trying-on" images, where the decoupled clothing images are placed on the person in the original person image, based on the input person image and the pseudo-labels. The original person image is then used to supervise the generation of the results. In this way, the model training process no longer relies entirely on the traditional one-to-one strongly supervised training paradigm of "clothing image—person trying on image." Instead, a unified model transforms a large number of person image resources in the product information service system, which were originally unusable directly, into effective training resources, allowing them to be fully utilized during model training. Furthermore, thanks to the diversity and complexity of real product images in the system, the trained model supports virtual fitting image generation under conditions of cross-category, cross-person, and complex poses, enhancing its generalization ability in real-world applications. Moreover, by transforming the person images themselves into supervisory signals, the supervisory objective is no longer "a real clothing image is required," but rather "reconstructing the original person image is sufficient." Therefore, as long as a large number of real person images are available, a training task can be constructed without needing to prepare corresponding standard clothing images for each person image, nor incurring the expensive manual pairing cost for all samples. In other words, the solution provided in this application transforms the original "paired label dependency" into "closed-loop supervision based on person images," thereby significantly reducing the data acquisition threshold.

[0022] Furthermore, when generating try-on images using the trained model, a two-stage clothing-aware mask generation method addresses the issues of maintaining character consistency and preventing the leakage of original clothing information. The first stage uses a larger, coarser mask, providing a sufficiently large editing area for the first garment, allowing for adequate deformation and reconstruction space. This avoids limitations imposed by the outline, length, and pattern of the second garment in the original character image, making it suitable for cross-category or significantly structurally varied clothing replacement generation. The second stage uses a finer mask, shrinking and refining the editing area. This allows for the preservation of stable areas such as hands, feet, belts, shoes, bags, accessories, body contours, and the background. The model only needs to generate the clothing areas that truly require redrawing, reducing the risk of erroneous rewriting of the human body structure under complex poses. In summary, this approach allows the model to handle significant clothing changes while preserving character details, supporting high-fidelity generation under complex categories and poses, achieving the goal of "ensuring sufficient clothing generation first, then ensuring character consistency."

[0023] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of the first method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the model training process provided in the embodiments of this application; Figure 3 This is a flowchart of the second method provided in the embodiments of this application; Figure 4 This is a flowchart of the third method provided in the embodiments of this application; Figure 5 This is a schematic diagram of the fitting image generation process provided in the embodiments of this application; Figure 6 This is a flowchart of the fourth method provided in the embodiments of this application; Figure 7 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0027] First, it's important to note that with the development of AI technology and the application of generative models, image generation models, represented by diffusion models, have become the mainstream technology in the field of virtual try-on. These methods typically leverage the strong image generation capabilities of diffusion models, using input images of a person, target clothing, pose information, text prompts, or reference features to achieve highly realistic clothing transfer, thereby automatically generating try-on images (i.e., images of a model wearing specific clothing) for a particular product. These methods perform well in academic metrics and visual effects, especially in high-resolution synthesis and overall realism. However, most diffusion-based virtual try-on solutions rely on strictly paired training samples of "product clothing image + person wearing the clothing image." Furthermore, these training datasets usually require relatively uniform shooting conditions, clear person poses, and minimal background interference. In real e-commerce scenarios, however, person images present challenges such as complex backgrounds, diverse poses, and complex clothing categories and fabrics. Therefore, models trained on simple, standard paired training datasets do not perform well when applied to real-world business for try-on image generation, exhibiting limited generalization ability.

[0028] On the other hand, in real-world business scenarios, while specific product information service systems possess a vast amount of product and person images, the vast majority do not constitute strict pairings, resulting in a significant amount of "dirty data." Consequently, the paired data samples required for the aforementioned model training methods are extremely scarce, and the costs of collection, cleaning, and registration are exceptionally high, making it difficult for such methods to directly and fully utilize the massive unpaired image resources accumulated on the platform.

[0029] Therefore, in order to reduce the reliance on paired data samples during model training and to fully utilize the massive unpaired image resources in the product information service system, thereby improving the model's generalization ability in generating try-on images in real e-commerce scenarios, this application provides a corresponding solution. In this solution, the model training method is optimized firstly, and secondly, the specific generation process can be optimized when using the trained model to generate try-on images.

[0030] Firstly, optimizing the model training method specifically includes dividing the model training into two stages. In the first training stage, high-quality image pairs can be acquired as the first training dataset for training the model. These image pairs can include clothing images (flat lay images of clothing) and corresponding images of people, where the clothing worn by the person in the person image is the same as that in the clothing image. This training data can then be used to train the model for the first stage.

[0031] After completing the first stage of training, the second stage can begin. In this second stage, multiple images of people can be acquired as a second training dataset. These images can be any images obtained from the product information service system; only the images of people are needed, without the need for strictly paired clothing images. Specifically, in the second stage of training, the clothing images (flat lay images of clothing) can be decoupled from the images of people using the model trained in the first stage. These clothing images are then used as pseudo-labels for the images of people. During the formal training of the model, the model can generate "try-on" images based on these pseudo-labels and the original images of people, using the previously decoupled clothing images. The generated results are then supervised using the original images of people, thus completing the training of the model.

[0032] After completing the second stage of training described above, the trained model can be used to generate corresponding try-on images for the product. For example, for a certain clothing product, only clothing images are available, but no images of people trying it on are available. In this case, the model provided in this embodiment can generate try-on images of people for the clothing product. The specific implementation method for generating try-on images using the trained model will be described in detail later.

[0033] The specific implementation schemes provided in the embodiments of this application will be described in detail below.

[0034] Example 1 In this first embodiment, a model training method is first provided, see [link to example]. Figure 1 The method may specifically include: S101: Obtain a first training dataset, wherein the first training dataset includes multiple image pairs, the image pairs include clothing images and person images, the clothing worn by the person in the person image in the same image pair is the same clothing as the clothing in the clothing image; the clothing image is a clothing tile image.

[0035] The first training dataset can be obtained from a known open-source dataset, which may include multiple pre-paired image pairs. In each image pair, there is a strict correspondence between the clothing image and the person image; that is, the person image wears the clothing in the clothing image. Furthermore, the person images in these image pairs are typically of high quality, resulting in a high-quality first training dataset, although its quantity is relatively small. Additionally, these person images usually have simple backgrounds, unlike the complex and diverse backgrounds found in commercial photos taken in real e-commerce scenarios.

[0036] S102: The model is trained in the first stage using the first training dataset. In the first stage of training, the model decouples the clothing image from the input person image and supervises the decoupling result using the clothing image in the original image pair, so that the model can obtain the ability to decouple the clothing image from the person image; wherein, the decoupled clothing image is a clothing tile image.

[0037] After obtaining the first training dataset, the model can undergo its first stage of training. The primary training objective in this first stage is to enable the model to decouple clothing images from person images (trying out clothing). That is, the model can be prompted to decouple clothing portions from person images using prompts, and the decoupling results can be supervised using the original clothing images from the original image pairs. It's important to note that decoupling clothing images from the original person images in this embodiment does not involve simply "cutting out" the clothing portion of the image. Instead, it involves generating a flattened image of the clothing (i.e., an image taken with the clothing laid flat) based on the clothing worn by the person in the original image, and using this flattened image as the decoupled clothing image.

[0038] Alternatively, during the first stage of training, the model can be instructed to try on clothing images from image pairs onto person images, and the original person images from the image pairs can be used to supervise the generation of the results, thereby enabling the model to acquire a basic ability to generate try-on images. However, the first training dataset still suffers from drawbacks such as limited data volume and poor diversity. Therefore, the model trained in this first stage will still exhibit shortcomings in generalization ability regarding try-on image generation.

[0039] In a preferred implementation, the model's ability to perform try-on and try-off tasks can be addressed through a shared architecture, allowing the model to acquire these capabilities during a single training session. Specifically, before inputting the image pair into the model, two input paths can be constructed. The first input can be the original image pair, or an editable region can be defined within the person image (this region can be determined based on the body area to be tried on for the first garment; for example, the upper body could be used as the editable region for a top, etc.), and this editable region can be occluded. The second input can be the person image from the image pair and a blank image. Furthermore, by inputting prompt text, the model can perform the task of generating a try-on image based on the first input, and the task of decoupling the garment image from the person image based on the second input. Then, the generated result is supervised using the person image and clothing image in the original image pair respectively. That is, the loss between the generated try-on image and the original person image is calculated using a pre-constructed loss function, and the loss between the generated clothing image and the clothing image in the original image pair is optimized.

[0040] S103: Obtain a second training dataset, which includes multiple images of people. Use the model that has completed the first stage of training to decouple clothing images from the images of people, and use them as pseudo-labels for the corresponding images of people.

[0041] After completing the first stage of training, the second stage of training can begin. Specifically, a second training dataset can be obtained. This dataset can consist of any images of people obtained from the product information service system, including images of models wearing specific products. It may contain complex and diverse backgrounds, and there is no need to strictly match clothing images to the specific images of people. In other words, the initial second training dataset may only contain images of people, without matching clothing images.

[0042] After obtaining the initial second training dataset, the process can be divided into two sub-stages. In the first sub-stage, the clothing images can be decoupled from the person images in the specific second training dataset using the model trained in the first stage. These clothing images can then be used as pseudo-labels for the person images. The term "pseudo-label" is used in contrast to the clothing images corresponding to the person images in the specific image pairs of the first training dataset. In the first training dataset, there is a strict correspondence between the clothing images and the person images, and the clothing images can serve as strong supervisory signals. However, in this second stage, since the original training dataset only includes person images, the clothing images are decoupled from the person images by the model. Since this involves model generation, they are not strictly labels; hence, they are called "pseudo-labels." In short, after adding pseudo-labels, the person images in the second dataset can also have corresponding clothing images, but these clothing images are decoupled from the person images by the model. Of course, these decoupled clothing images are also tiled images of the clothing.

[0043] S104: Using the person images and corresponding pseudo-labels in the second training dataset, the model is trained in the second stage. In the second stage of training, the model generates a try-on image by putting the decoupled clothing image on the person in the original person image based on the input person image and the pseudo-labels, and supervises the generation result using the original person image.

[0044] After generating pseudo-labels for the images of people in the second dataset, we can proceed to the second sub-stage of the second stage. That is, we can use the images of people in the second dataset and these pseudo-labels to further train the model. Specifically, the model can generate a "try-on" image based on the input images of people and pseudo-labels, by putting the clothing images corresponding to the pseudo-labels (i.e., the decoupled clothing images mentioned above) onto the people in the original images of people. The original images of people are then used to supervise the generation of the results, thus completing the second stage of training for the model.

[0045] In other words, in the second training phase, the clothing model is first decoupled from the images of people in the second dataset using the model after the first phase of training, which is equivalent to "removing" the clothing from the people; then the model "puts" the decoupled clothing back on the people, and the training of the model is completed by using the original images of people for supervision.

[0046] Although the second training phase uses images of people and pseudo-labels, the quality of individual image pairs may not be as good as the first training dataset. However, because a large amount of such pseudo-labeled data can be obtained, the final training effect can usually meet expectations. Furthermore, since the second training dataset can be constructed using product images from a product information service system, the massive amount of images in the system can be fully utilized. Thanks to the diversity and complexity of real product images in the system, the generalization ability of the trained model in real-world applications is also enhanced.

[0047] Furthermore, as mentioned earlier, during the first stage of model training, its ability to generate try-on images can also be trained simultaneously. Therefore, in this case, the specific second stage of training can be conducted based on the model trained in the first stage. This allows the final trained model to not only learn high-fidelity try-on image generation capabilities from the high-quality image pairs in the first training dataset, but also enhance its generalization ability in real-world product information service systems through the more diverse and complex training data in the second training dataset.

[0048] In summary, under a preferred implementation, the specific model training architecture can be as follows: Figure 2 As shown. Specifically, a shared architecture can be provided to simultaneously perform "trying on" and "trying off" tasks using task-specific masks. Figure 2 As can be seen, in the first training phase, a set of images from the same image pair can simultaneously train the model's "Try-on" (trying on clothes, i.e., using digital technology to overlay clothing (such as clothes, glasses, cosmetics, jewelry, etc.) onto an image or video of a person to simulate a realistic wearing / dressing effect) and "Try-off" (trying off clothes, i.e. using generative AI technology to extract clothing from an image and generate a front-facing product image containing only that clothing) capabilities. The model can have two inputs (of course, it can have only one input at a time, but it supports two input methods). The first input corresponds to the "Try-on" part, which can be constructed from the original image pair. Specifically, it can include an image containing clothing (G) and an image of the person after selecting an editable area from the original image (P) and occluding the content within that editable area. The second input corresponds to the "Try-off" part, which can be constructed from the same original image pair corresponding to the first input. This includes the original image of the person (P) and a blank area (representing the clothing to be removed). The two input pairs are processed by the encoder to generate a latent representation. ), noise (ε) and ( The sum of these variables forms the latent variable at time step (t). Meanwhile, the text prompt is converted into an embedding vector by a text encoder (T).

[0049] Next, we move on to the core network section, where the latent variables ( The input is fed into the model to be trained (specifically, it could be a diffusion model, in...). Figure 2 In the context of the UDiT module, this module consists of two sub-modules: MM-DiT (Multi-Modal Diffusion Transformer) is used to process multimodal information (image + text).

[0050] Single DiT (Single-Mode Diffusion Transformer) is used to process single-mode information. All components in the UDiT module described above are trainable.

[0051] After UDiT, the latent variables are obtained ( Then, through the decoder (D), the output signal can also have two types (only one type can be available at any given time). The first output signal is the fitting image. Specifically, it can be generated based on the first input signal; the second output signal corresponds to the trial removal image. It is generated based on the second input signal.

[0052] Among them, the person image (P) in the original image pair can be used to generate the try-on image from the first output. Supervision is performed by using the clothing image (G) from the original image pair to analyze the first-path output fitting image. The training was supervised to complete the first phase, enabling the model to acquire basic capabilities in generating try-on images and decoupling clothing images from images of people (“trying on”).

[0053] Next, the second stage of training can begin, where the model trained in the first stage can be further trained. The input data consists of multiple real-world images of people (P), each of which may contain different clothing. In the second training stage, the model completed in the first stage (UDiT) is first used to decouple the clothing images (P) from the person images (P). Specifically, the original image of the person (P) and a blank area can be constructed as an input pair. After processing such as compression / noise addition, the input is fed into the UdiT module, and the output is the decoupled clothing image (P). This type of clothing image decoupled from the model ( ) can be used as a pseudo-label for the original human image (P).

[0054] After generating pseudo-labels for the original person image (P), the extracted pseudo-labels can be filtered. Specifically, techniques such as the Visual Transformer (ViT) model can be used to evaluate the quality of the decoupled clothing images. For example, the similarity between the pseudo-labels and the clothing images in the original person image can be calculated, and samples that meet the similarity criteria can be selected. Additionally, visual models can be used to determine the reasonableness of the decoupled clothing images, evaluating aspects such as clothing integrity and background cleanliness. The aforementioned similarity, clothing integrity, and background cleanliness can all serve as quality evaluation indicators for the decoupled clothing images, yielding a quality assessment result. Ultimately, only high-quality pseudo-label samples can be retained, while low-quality ones are discarded. After completing this multi-criteria filtering, the constructed (person image, pseudo-label) can be used as training samples for the second stage of training.

[0055] After constructing multiple (character images, pseudo-labels) models, they can be input in pairs into the model trained in the first stage for the second stage of training. Specifically, the target edit region can be determined from the character images and then paired with the pseudo-labels (…). Together with the model, these are used as inputs to produce a try-on image as the model's output. That is, the previously decoupled clothing images ( The image is then "transformed" back into the original character image (P). Then, the original task image (P) is used to refine the generated try-on image (P). This process is supervised. After multiple iterations, the second stage of training for the model can be completed.

[0056] The above Figure 2 The overall design shown can achieve high-quality, consistent virtual try-on effect generation, and improve the robustness and generalization ability of the model through a variety of strategies.

[0057] In summary, through Embodiment 1 of this application, the model training process is divided into two stages. In the first stage, the model is trained using strictly paired image pairs, enabling it to decouple clothing images from person images. Then, a second training dataset is obtained, which only includes person images and does not need to include strictly paired clothing images. However, the clothing images decoupled from the person images can be extracted using the model trained in the first stage, serving as pseudo-labels for the person images. Next, the model is trained in the second stage using these pseudo-labeled person images. The model generates "trying on" images, where the decoupled clothing images are placed on the person in the original person image, based on the input person image and the pseudo-labels. The original person image is then used to supervise the generation of the results. In this way, the model training process no longer relies entirely on the traditional one-to-one strongly supervised training paradigm of "clothing image - person trying on" image. Instead, a unified model transforms a large number of person image resources in the product information service system, which were originally unusable directly, into effective training resources, allowing them to be fully utilized during model training. Furthermore, thanks to the diversity and complexity of real product images in the system, the trained model supports virtual fitting image generation under conditions of cross-category, cross-person, and complex poses, enhancing its generalization ability in real-world applications. Moreover, by transforming the person images themselves into supervisory signals, the supervisory objective is no longer "a real clothing image is required," but rather "reconstructing the original person image is sufficient." Therefore, as long as a large number of real person images are available, a training task can be constructed without needing to prepare corresponding standard clothing images for each person image, nor incurring the expensive manual pairing cost for all samples. In other words, the solution provided in this application transforms the original "paired label dependency" into "closed-loop supervision based on person images," thereby significantly reducing the data acquisition threshold.

[0058] In an optional implementation, the model's ability to generate try-on images can be trained using specific image pairs during the first stage of training. The second stage can then be further trained based on the results of the first stage. Because the image pairs used in the first stage are of relatively high quality, the model not only improves its generalization ability as mentioned above, but also better ensures the quality of the generated try-on images, resulting in more "high-fidelity" try-on images.

[0059] Example 2 In the aforementioned Embodiment 1, the model training process is divided into two stages. However, in other methods, the task of decoupling clothing images from person images can also be performed in other ways. For example, other pre-trained models capable of decoupling clothing images from person images can be used to complete this task, and so on. Therefore, Embodiment 2 also provides another model training method, which is equivalent to only including the second stage of training in Embodiment 1. Specifically, see... Figure 3 The method may include: S301: Obtain a training dataset, which includes multiple images of people; S302: Decouple the clothing image from the person image, wherein the decoupled clothing image is a clothing tile image, and use the decoupled clothing image as a pseudo-label for the corresponding person image. S303: The model is trained using the person images and corresponding pseudo-labels in the training dataset. The model generates a try-on image by placing the decoupled clothing image onto the person in the original person image based on the input person image and the pseudo-labels. The original person image is used to supervise the generation result.

[0060] This second embodiment also enables the effective use of a large number of real human images generated in the commodity information service system, thereby improving the generalization ability of the trained model when generating clothing try-on in real-world scenarios.

[0061] Example 3 The foregoing Embodiments 1 and 2 introduced the model training method provided by the embodiments of this application. In Embodiment 2, a method for generating try-on images is also provided. See [link to Embodiment 2]. Figure 4 The method may specifically include: S401: Determine a first clothing image for which a try-on image needs to be generated, and an original person image, wherein the second clothing worn by the person in the original person image is different from the first clothing in the first clothing image.

[0062] In practical implementation, a virtual try-on function can be provided to merchant users or consumer users in the product information service system. Specifically, for merchant users, the product information service system often contains clothing items where merchants only take photos of the items themselves and publish the images directly. These items lack images of actual models trying on the clothing. In this case, the clothing image can be the first clothing image for generating virtual try-on images. In practical implementation, this application embodiment can be provided to merchant users as a merchant-side tool. Merchant users can determine which specific products require virtual try-on images based on their needs, and then provide the clothing image and the original human image. The model provided in this application embodiment can then be used to automatically generate virtual try-on images for publication in the product information service system, and so on. The clothing image can be obtained by the merchant user through photographing the actual clothing. The original human image can be selected from a library provided in the system, or it can be a human image uploaded by the merchant user, etc. Of course, the human images selected from the library must meet copyright requirements.

[0063] For consumers, while browsing product information, they may want to see how specific clothing items would look on them. In this case, a "try it on" option can be provided on product detail pages or similar pages. Consumers can use this option to access the try-on page and upload a photo of themselves (or another person) as their image. Then, the system can identify clothing images from the currently viewed product detail pages and use the user-uploaded image as the original image for generating subsequent try-on images, and so on.

[0064] S402: Based on the body parts to be tried on the first garment, determine the target editing area in the original character image; wherein, the part of the original character image located within the target editing area is the part that needs to be modeled, and the part outside the target editing area is retained.

[0065] After determining the first clothing image and the original person image, the previously trained model can be used to generate the corresponding try-on image. Specifically, the target editing region can be determined from the original person image based on the specific body parts that need to be tried on with the first clothing, and then the trained model can be used to generate a try-on image within that target editing region.

[0066] In other words, in practical implementation, the original person image includes not only the original clothing worn by the person (e.g., what can be called the "second outfit"), but also elements such as the background, matching clothing of the second outfit, and body parts of the person that are not obscured by clothing, such as the head and hands. Specifically, when generating a try-on image for the first outfit, the second outfit in the original person image can be replaced with the first outfit, while other original content in the original person image can be retained. This reduces the workload of the model generation task and preserves the background, matching clothing, and other elements already present in the original person image, facilitating the presentation of the first outfit's fit. Furthermore, by replacing the original clothing in the person image, more information about the person can be preserved in the generated image; that is, the consistency between the generated try-on image and the original person image is ensured. This is especially important in try-on requests initiated by consumer users, where the requirement for consistency is usually higher, thus highlighting the value of this generation method.

[0067] S403: Using a pre-trained model, generate a try-on image of the first garment within the target editing area; the pre-trained model is trained according to the method described in Embodiment 1 or 2.

[0068] After determining the target editing region, the model can generate an image within that region. Specifically, in determining the target editing region, one approach is to directly determine the editing region from the original image of the person, based on the body part to be tried on with the first garment (this can be a coarse-grained division of body parts, such as the upper or lower body). The primary principle in determining this editing region is to provide sufficient editing space for the first garment, removing prior constraints such as the garment's boundaries, length, and tightness. For example, suppose the first garment is a long-sleeved T-shirt that needs to be tried on on the upper body. In the original image, the person is wearing a short-sleeved T-shirt, and their forearms are not covered by the clothing. Furthermore, the long-sleeved and short-sleeved T-shirts differ in length and looseness. In this case, the entire upper body area below the head in the original image can be considered the editing region. Taking into account the maximum possible looseness of the garment, a sufficiently large editing region is determined, including the forearms that were not originally covered by the clothing. In this way, the content within the edit area is all that needs to be generated by the model, while the content outside the edit area is preserved. After determining the edit area, the previously trained model can generate a try-on image within this edit area based on the clothing image of the first garment and the original image of the person.

[0069] Of course, the edit area defined above is relatively coarse-grained; for example, the entire upper body, and even parts like the hands in the original image, might be included in the edit area. However, hands are usually not obscured by clothing. In this case, the model not only needs to generate the appearance of the first garment on the person but also needs to complete and generate the hands and other details. However, generative models often produce poor results when generating such detailed body parts (e.g., incorrect number of fingers or finger joints), which may affect the final appearance of the try-on image.

[0070] To address this situation, in the preferred embodiment provided in this application, the process of generating the try-on image can be divided into two stages. In the first stage, as described above, a sufficiently large editing region (e.g., the entire upper body below the head, etc., which can be referred to as the first editing region for ease of distinction) is determined in the original character image based on the body parts that the first garment needs to be tried on. Then, the trained model generates the try-on image within the aforementioned first editing region. That is, the content within the editing region of the original task image is generated using the first garment image.

[0071] Specifically, there are several ways to determine the first editing area. One approach is to first determine a general area in the original person image without restrictions on clothing boundaries, length, or tightness, based on the body area (e.g., the upper body) that the first garment needs to be tried on (ensuring the editing area is large enough). Alternatively, the second garment area can be determined in the original person image based on the outline of the second garment. Then, the first editing area is determined by overlaying the general area with the area within the bounding box of the second garment area.

[0072] It should be noted that the base model of the try-on image generation model used in this embodiment can be a diffusion model, etc. The characteristic of the diffusion model is that the generation process is actually a process of adding noise M times and denoising N times to obtain the final generation result. However, in the first stage of trying-on image generation in this embodiment, a relatively small number of denoising steps can be performed, for example, 10 times, so that the model can initially generate the position and outline of the first garment when it is tried on on the person in the original image. Specific texture and other details do not need to be processed for the time being.

[0073] Then, the second stage of image generation for the try-on image can be initiated. In this second stage, the position and outline of the first garment when it is tried on by the person in the original image can be extracted from the generation results of the first stage. Based on this position and outline information, a second editing region can be determined. This second editing region can be the same as or slightly larger than the region enclosed by the aforementioned outline. Additionally, the position and outline of the second garment can be extracted from the original image, and a third editing region can be determined from the original image based on the outline of the second garment. The second and third editing regions can then be superimposed to obtain a target editing region. It should be noted that when superimposing the second and third editing regions, the original image can first be pixel-level binarized (masked) according to the second and third editing regions respectively. For example, pixels within the editing region are set to 1, and pixels outside the editing region are set to 0. Then, during the superposition process, the union of the two binarized images can be performed. If a pixel is 1 in the binarized image of the second editing region and 0 in the binarized image of the third editing region, then after overlay processing, that pixel will have a value of 1. The target editing region determined in the above way will be smaller than the first editing region determined in the first stage. That is, the target editing region is shrunk, and the part that needs to be generated by the model is reduced accordingly. More of the original content in the original human image can be preserved, including human body details such as hands and feet.

[0074] Furthermore, in practical implementation, directly determining the target editing region based on the second and third editing regions may not be accurate enough. For example, for some "sleeveless" garments, the "shoulder straps" are relatively thin compared to the overall garment. During garment contour extraction or mask generation, the "shoulder straps" may be overlooked, leading to the "sleeveless" garment being identified as a "tube top" garment. In this case, if the target editing region is determined based on the contour of the "tube top" garment, the final target editing region may be too small (not including the shoulders of the person's body). Consequently, when the model generates a try-on image, there may not be enough space to generate a try-on image of the "shoulder straps" on the person's body.

[0075] To address the above issues, an alternative approach is to extract key body part outlines from the original image based on the body parts to be tried on with the first garment, and then determine the fourth editing region based on these outlines. For example, if the first garment is a top and the upper body needs to be tried on, the key body part could be the "torso." In this case, the "torso" outline can be predicted using other models to determine the key body part outline and further define the fourth editing region. Then, the target editing region can be obtained by overlaying the second, third, and fourth editing regions. Specifically, mask images can be generated for each of the second, third, and fourth editing regions, and then the union of these mask images can be used to obtain the final target editing region. This method allows the fourth editing region to correct any potential deviations in the second and third editing regions.

[0076] After determining the target editing region using the above method, the previously trained model is used to complete the subsequent generation tasks within the target editing region, based on the generation in the first stage.

[0077] As mentioned earlier, the specific model can be trained based on a diffusion model. In the first stage, a few denoising steps can be performed, while in the second stage, subsequent denoising steps can be completed to generate the final try-on image. For example, assuming a complete generation task requires N denoising steps, in the above-mentioned staged generation process, N1 denoising steps can be performed in the first stage, and N2 denoising steps in the second stage. N1 is less than N2, but the sum of N1 and N2 can be equal to or close to N. In other words, although the generation process is divided into two stages, the total number of denoising steps does not increase, or if it does increase, it is only slightly, and will not lead to an excessively long overall generation time.

[0078] By generating clothing-aware masks in a two-stage process, the problems of maintaining character consistency and preventing the leakage of original clothing information can be solved. The first stage uses a larger, coarser mask to provide a sufficiently large editing area for the first garment, offering ample space for deformation and reconstruction. This avoids limitations imposed by the outline, length, and pattern of the second garment in the original character image, making it suitable for clothing replacement generation across categories or with significant structural changes. The second stage uses a finer mask to shrink and refine the editing area, focusing on preserving stable areas such as hands, feet, belts, shoes, bags, accessories, and the character's background and body outline. The model only needs to generate the clothing areas that truly require redrawing, thus reducing the problem of erroneous rewriting of the human body structure under complex poses. In short, this approach allows the model to handle significant clothing changes while preserving character details as much as possible, supporting high-fidelity generation under complex categories and poses, achieving the goal of "ensuring sufficient clothing generation first, then ensuring character consistency."

[0079] In summary, under a preferred implementation, the specific process of generating try-on images can be as follows: Figure 5 As shown, this is a two-stage image processing flowchart for generating fitting images, mainly involving image segmentation, mask generation, and image synthesis. The overall process uses two stages (the first stage and the second stage) and multiple mask operations to achieve accurate replacement of clothing in the image of a person.

[0080] Specifically, in the original image A on the left, a woman is wearing a tank top (referred to as the second outfit) and shorts, with a beach background. The goal is to have a different tank top (referred to as the first outfit) worn by a woman in another image B be applied to the woman in image A. The specific generation process involves... Figure 5The multiple masks shown are: ① (cloth-agnostic mask): A general mask that does not depend on specific clothing types, covering body parts (such as the upper or lower body), but not distinguishing specific clothing; ② (cloth mask): The outline area mask of the second garment (e.g., the upper garment area); ③ (bbox of ②): The bounding box of ②, used to locate the clothing area; ④ (cloth-bbox mask): A rectangular clipping mask generated by combining ② and ③, restricting the clothing area within the bounding box; ⑤ (cloth mask): The clothing area mask of the first garment; ⑥ (Densepose-driven mask): A human pose-driven mask generated based on DensePose technology, which can accurately match the shape of human body parts, such as the torso, arms, and legs. In this example, it corresponds to the torso; ⑦ (garment-aware mask): The final "garment-aware" mask, which is formed by merging multiple masks through a union operation, more accurately describing the target editing area.

[0081] The specific generation process is as follows: Take the union of ① and ③ above to determine the first editing region. Within this first editing region, the model tries on the top from image B onto the person in image A, obtaining the intermediate state of the try-on image C. Of course, only a few denoising steps are required here.

[0082] Then, the clothing area mask ⑤ of the first garment can be extracted from the intermediate state try-on image C after the first garment is tried on by the person in the character image A.

[0083] Next, mask ② (i.e., the clothing area mask of the second garment), mask ⑤, and mask ⑥ (i.e., the mask of the human torso in image A) are superimposed. That is, the union of the images after binarization according to the masks is taken to obtain the target editing region ⑦. Then, the model continues to perform multiple denoising steps within the target editing region ⑦ in image A to generate the final try-on image D, which shows the first garment in image B being tried on by the person in image A.

[0084] It should be noted that, as mentioned above, the base model of the try-on image generation model used in this application embodiment can be a diffusion model. A diffusion model is a generative AI model whose core idea is to create data by simulating the process of "information destruction and restoration." Its operation includes two steps: First, forward diffusion (adding noise): Random noise is continuously added to a clear image, and after repeated iterations, the image eventually becomes a completely random noise field (pure noise). This process is fixed. Second, backward diffusion (denoising): The neural network learns how to "operate in reverse," starting from a pure noise field, predicting and removing minute noise step by step, ultimately restoring a clear, new image.

[0085] In image generation tasks, U-Net is the main architecture used in diffusion models, and its core design can include: Codec structure: It is symmetrical U-shaped and achieves the generation of the target from noise by combining downsampling (reducing size to extract semantics) and upsampling (restoring details).

[0086] Skip Connection: Directly transmits shallow details and assists in upsampling paths to repair the image.

[0087] Attention mechanism: The embedded attention module enhances global modeling and long-distance dependencies, which is crucial for generating the overall structure.

[0088] Timestep Embedding: The denoising step number t is encoded and injected into the network to inform the model of the current "denoising stage" so that corresponding operations can be performed.

[0089] Multi-resolution processing: Improve image quality by extracting features at different levels through multi-scale analysis and residual blocks.

[0090] While the U-Net architecture is good, its structure has significant inductive bias for images. Therefore, the DiT approach can also be used, which uses the Vision Transformer (ViT) architecture. It utilizes self-attention to capture long-range dependencies, exhibiting stronger scalability under large-scale data and computing power, and is gradually becoming one of the mainstream choices.

[0091] In addition, to achieve high-quality and controllable generation, a complete diffusion model can also include two major auxiliary components: Component 1: Employing MMDiT, a two-stream Transformer with independent weight sets is used for both images and text, enabling deep fusion of the two. Building upon this, Flow Matching training is used to efficiently generate high-quality images from noise within a few steps.

[0092] Component 2: First, the image is compressed to a low-dimensional latent space (compression ratio up to 48 times). Most of the calculations are done here, which greatly improves efficiency without affecting quality.

[0093] Regarding the loss function, the training objective of the standard diffusion model is very intuitive: to teach U-Net to predict the noise added in the current step. Therefore, the most commonly used simplified loss function is to calculate the mean squared error (MSE) between the actual noise and the model's predicted noise. In addition, L1 (mean absolute error) loss, perceptual loss, etc., are also often used for auxiliary or special scenarios.

[0094] The core of L1 loss is to calculate the sum of the absolute differences between predicted and actual noise, unlike MSE (L2), which imposes a squared penalty on large errors. Compared to MSE, L1 is more lenient in penalizing large errors and is therefore less sensitive to outliers in the data. However, theoretically, MSE is the optimal solution when the noise distribution strictly follows a Gaussian distribution. Because the forward noise addition in the diffusion model explicitly assumes Gaussian noise, using MSE for standard training is theoretically necessary. However, in some consistency models or variants requiring faster generation, L1 is occasionally used to obtain better numerical stability or generate images with sharper edges.

[0095] Perceptual loss doesn't compare pixel values, but rather high-level semantic features. It uses a pre-trained neural network (such as VGG) to extract "feature maps" from both the real and generated images, then calculates the L1 or L2 distance between these features. Its core principle is to align with human perception of "similarity." Two images may differ significantly in pixels, but if their content, style, or object structure is similar, perceptual loss will classify them as close; conversely, if the pixels are almost identical but the overall image is shifted by one pixel, perceptual loss will classify them as different. Perceptual loss is generally not used to train the core denoising U-Net because it would disrupt the mathematical foundation of noise prediction. However, it is commonly used in two scenarios: Conditional guidance: During sampling and generation, perceptual loss is used as an additional guiding signal to make the generated image more semantically consistent with expectations.

[0096] Fine-tuning and post-processing: Without damaging the already trained diffusion model, fine-tuning the decoder, super-resolution module, or image inpainting network using perceptual loss can significantly improve visual realism and detail richness.

[0097] In summary, MSE loss can be used to train the core noise prediction network; L1 loss is occasionally used to pursue robustness; and perceptual loss is a "seasoning" to improve the visual quality of the final image, often used in the guidance or fine-tuning stage.

[0098] In addition, during inference (generation) in the diffusion model, several key parameters significantly affect the final result: 1. Noise Reduction Steps That is, the number of iterations in the reverse process (commonly represented by N or T).

[0099] More steps generally result in higher generation quality and richer details. However, inference time increases linearly, at the cost of slower performance.

[0100] Fewer steps: faster generation speed, but the image may be coarse or retain residual noise.

[0101] Common values: The default is usually 50 or 100 steps. For faster speed, the number of steps can be reduced to 20-30 (with advanced algorithms). For the best quality, 200-1000 steps are sometimes used.

[0102] 2. Sampling Algorithm The specific method for determining how to recover the image step by step from the noise.

[0103] DDPM (Denoising Diffusion Probabilistic Models): The original algorithm, which requires many steps (usually 1000 steps) and is slow.

[0104] DDIM (Delayed Diffusion Importance Sampling): Faster, supports skipping steps (20-50 steps are sufficient), provides good deterministic results, and is commonly used.

[0105] DPM (Deformable Parts Model) / DPM++ series: Designed for acceleration, variants such as DPM++ 2M Karras can achieve high-quality results in 20-30 steps.

[0106] Euler / Heun: The basic solver with good stability; Euler a, on the other hand, has some randomness and produces more "creative" results.

[0107] 3. Guiding strength The parameter that controls the degree of influence of conditions (such as text prompts) on the generated results is usually called CFG Scale (no classifier guidance strength), and its range is generally between 1 and 20.

[0108] Low values ​​(e.g., 1-3): The model has a large degree of freedom, may ignore input conditions, is highly creative but sometimes goes off-topic.

[0109] Values ​​such as 7-12 represent a balance point. 7-9 is the recommended value in most cases, providing both obedience and good results.

[0110] High values ​​(e.g., 15-20): Strictly adhere to the conditions, but the image may be oversaturated with colors and have strange contrast.

[0111] 4. Random Seed Control the randomness of the initial noise map.

[0112] Fixed seed: The results are basically the same each time they are generated, which is used to reproduce or compare the effects of parameters.

[0113] Random seed: Generates different results each time to explore multiple possibilities.

[0114] Function: When the parameters and models are completely identical, the seed is the only random variable that determines the final image.

[0115] 5. Noise Reduction Intensity Commonly used in image-to-image (img2img) scenarios, the range is 0-1, controlling the degree of change to the original image.

[0116] Intensity 0: hardly changes the original image, and the output is almost the same as the original image.

[0117] Intensity 1: Completely ignores the original image, equivalent to text-to-image (txt2img).

[0118] Intensity 0.4-0.7: A common fine-tuning range that retains the original image composition while incorporating the characteristics of new prompt words.

[0119] Inference parameter combination suggestions If speed is a priority (e.g., less than 10 seconds per frame): Sampler options include DPM++ 2M Karras, 20-30 steps, and CFG Scale 7.

[0120] If quality is a priority: the sampler can be DPM++ 2M SDE Karras, with 40-50 steps and CFG Scale 7-9.

[0121] If stability / reproducibility is a priority: the sampler can be DDIM, with 50 steps and a fixed seed.

[0122] For image processing (fine-tuning details): noise reduction intensity 0.4-0.6, number of steps 30-40.

[0123] Furthermore, diffusion models can encompass various types. For example, based on control conditions: this is the most intuitive perspective, determining whether the model can "follow instructions," mainly divided into unconditional generation (only noise input) and conditional generation (guided by external inputs such as text descriptions and images). Based on modeling framework: this represents a divergence in technical approaches. DDPM (Denoising Diffusion Probabilistic Models) is a foundational work, using discrete Markov chains to define noise; DDIM (Denoising Diffusion Implicit Models) uses non-Markov processes, resulting in faster generation; SDE (Score-Based Generative Models) extends the process to continuous stochastic differential equations; ODE (Probability Flow ODE) is its deterministic equivalent, allowing efficient sampling using standard ODE solvers; and FlowMatching is a recent paradigm that further improves generation speed by constructing a "flow" path from noise distribution to data distribution. Additionally, classification can be based on other dimensions; in specific implementation, the specific model type can be selected according to actual needs.

[0124] Example 4 In the aforementioned Embodiment 3, a method for generating try-on images in stages was provided. In practical applications, if the model is trained using ordinary training methods, this generation method can also be used to generate try-on images. Therefore, in this Embodiment 4, a method for generating try-on images is provided, see [link to Embodiment 4]. Figure 6 The method may include: S601: Determine a first clothing image for which a try-on image needs to be generated, and an original person image, wherein the second clothing worn by the person in the original person image is different from the first clothing in the first clothing image. S602: Determine a first editing area in the original character image based on the body parts to be tried on the first garment; S603: Using a pre-trained model, generate a first-stage try-on image of the first garment within the first editing area; S604: Extract the outline of the first garment from the try-on image generated in the first stage, and determine the second editing area based on the area within the extracted garment outline; S605: Extract the outline of the second garment from the original character image, and determine the third editing area based on the area within the extracted garment outline; S606: The second editing region and the third editing region are superimposed, and the target editing region is determined in the original human image according to the superimposition result. The target editing region is smaller than the first editing region so that by shrinking the first editing region, more human body structure regions in the original human image are retained. The pre-trained model is then used to generate a second-stage try-on image in the target editing region to obtain the target try-on image.

[0125] Through the above-described embodiment four, the two-stage generation of clothing-aware masks can solve the problems of maintaining character consistency and easily leaking original clothing information. The first stage uses a larger coarse mask, providing a sufficiently large editing area for the first garment, offering ample space for deformation and reconstruction. This avoids limitations imposed by the outline, length, and pattern of the second garment in the original character image, making it suitable for clothing replacement generation across categories or with significant structural changes. The second stage uses a fine mask, shrinking and refining the editing area. This allows for the preservation of stable areas such as hands, feet, shoes, bags, accessories, character background, and body outlines. The model only needs to generate the clothing areas that truly require redrawing, thus reducing the problem of erroneous rewriting of the human body structure under complex poses. In summary, this approach allows the model to have sufficient space for significant clothing changes while preserving character details as much as possible, supporting high-fidelity generation under complex categories and poses, achieving the goal of "ensuring sufficient clothing generation first, then ensuring character consistency."

[0126] For any parts not described in detail in the above embodiments, please refer to other embodiments and other parts of this specification, which will not be repeated here.

[0127] It should be noted that the embodiments of this application may involve the use of user data (including human image data, etc.). In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0128] Corresponding to Embodiment 1, this application also provides a model training apparatus, which may include: The first training dataset acquisition unit is used to acquire a first training dataset, wherein the first training dataset includes multiple image pairs, the image pairs include clothing images and person images, the clothing worn by the person in the person image in the same image pair is the same clothing in the clothing image; the clothing image is a clothing tile image. The first training unit is used to train the model in the first stage using the first training dataset. In the first stage of training, the model decouples the clothing image from the input person image and uses the clothing image in the original image pair to supervise the decoupling result, so that the model can acquire the ability to decouple the clothing image from the person image; wherein, the decoupled clothing image is a clothing tile image. The second training dataset acquisition unit is used to acquire a second training dataset, which includes multiple images of people. The clothing images are decoupled from the images of people using the model that has completed the training in the first stage, and used as pseudo-labels for the corresponding images of people. The second training unit is used to train the model in the second stage using the person images and corresponding pseudo-labels in the second training dataset. In the second stage of training, the model generates a try-on image by putting the decoupled clothing image on the person in the original person image based on the input person image and the pseudo-label, and supervises the generation result using the original person image.

[0129] The training in the first stage also includes: the model generating a try-on image based on the input image pair, which puts the clothing image on the person in the image pair, and using the original person image in the image pair to supervise the generation result, so that the model can obtain the initial ability to generate try-on images. In the second stage of training, the model is trained using the images of people and their corresponding pseudo-labels from the second training dataset, building upon the model trained in the first stage.

[0130] In the first stage, through a shared architecture, the model acquires the ability to generate try-on images and the ability to decouple clothing images from person images during the same training process. Specifically, for the same image pair, a first input pair and a second input pair are constructed. The first input pair includes: the clothing image in the image pair and the image of the person after occlusion processing of the target editing area. The second input pair includes: a blank image and the image of the person in the same image pair corresponding to the first input pair. By inputting prompt text, the model performs the task of generating a try-on image based on the first input pair and supervises the generation result using the image of the person in the original image pair. It also performs the task of decoupling the clothing image from the image of the person based on the second input pair and supervises the decoupling result using the image of the clothing in the original image pair.

[0131] Corresponding to Embodiment 2, this application also provides a model training apparatus, which may include: The training dataset acquisition unit is used to acquire a training dataset, which includes multiple images of people. A decoupling unit is used to decouple the clothing image from the person image, wherein the decoupled clothing image is a clothing tile image, and the decoupled clothing image is used as a pseudo-label for the corresponding person image. The pseudo-label determination unit is used to train the model using the person images and corresponding pseudo-labels in the training dataset. The model generates a try-on image by putting the decoupled clothing image on the person in the original person image based on the input person image and the pseudo-labels, and supervises the generation result using the original person image.

[0132] In a specific implementation, the device may further include: The image pair acquisition unit is used to acquire multiple image pairs, wherein the image pairs include clothing images and person images, and the clothing worn by the person in the person image in the same image pair is the same clothing as the clothing in the clothing image; the clothing image is a flat lay image of clothing. The preliminary training unit is used to perform preliminary training on the model using the multiple image pairs. The model generates a try-on image by putting the clothing image on the person in the image pair based on the person image and clothing image in the input image pair, and uses the person image to supervise the generation result. The step of training the model using the person images and corresponding pseudo-labels in the training dataset includes: Using the images of people and their corresponding pseudo-labels in the training dataset, the model is trained on the basis of the initial training.

[0133] Specifically, the decoupling unit can be used for: Using a pre-trained model capable of decoupling clothing images from person images, clothing images are decoupled from the person images; The device may further include: The filtering unit is used to calculate the similarity between clothing images in the decoupled clothing image domain of the person images through a visual model, and / or calculate the clothing integrity and background cleanliness of the decoupled clothing images, so as to filter out clothing images that do not meet the quality conditions through multi-criteria filtering, and then determine the person images and corresponding pseudo-labels in the training dataset.

[0134] Corresponding to Embodiment 3, this application also provides a fitting image generation device, which may include: The original information determination unit is used to determine a first clothing image for which a try-on image needs to be generated, and an original person image, wherein the second clothing worn by the person in the original person image is different from the first clothing in the first clothing image. The target editing region determination unit is used to determine the target editing region in the original human image based on the body parts to be tried on by the first garment; wherein, the part of the original human image located within the target editing region is the part that needs to be modeled, and the part outside the target editing region is retained; The generation unit is used to generate a try-on image of the first garment within the target editing area using a pre-trained model; the pre-trained model is trained according to the method described in Embodiment 1 or 2 above.

[0135] Specifically, the target editing region determination unit can be used for: Based on the body area to be tried on the first garment, a first editing area is determined in the original character image; Using a pre-trained model, a first-stage try-on image is generated for the first garment within the first editing area; Extract the outline of the first garment from the try-on images generated in the first stage, and determine the second editing area based on the area within the extracted garment outline; Extract the outline of the second garment from the original image of the person, and determine the third editing area based on the area within the extracted garment outline; The second editing region and the third editing region are superimposed, and the target editing region is determined in the original human image based on the superimposition result. The target editing region is smaller than the first editing region so that by shrinking the first editing region, more human body structure regions in the original human image are preserved. The pre-trained model is then used to generate a second-stage try-on image of the first garment in the target editing region to obtain the target try-on image.

[0136] Specifically, based on the body area to be tried on the first garment, a general area without garment boundaries, length, or tightness restrictions can be determined in the original human image; Based on the outline of the second garment in the original figure image, the area of ​​the second garment is determined in the original figure image; The first editing area is determined by overlaying the general area with the area within the bounding box of the second clothing area.

[0137] In addition, based on the body parts that need to be tried on with the first garment, the outline of key body parts can be extracted from the original human image, and a fourth editing region can be determined based on the outline of key body parts, so that the second editing region, the third editing region and the fourth editing region can be superimposed, and the target editing region can be determined in the original human image based on the superimposition result.

[0138] Specifically, the area within the outline of the key body part can be expanded within a preset range to determine the fourth editing area.

[0139] The pre-trained model is based on the diffusion model. The number of denoising loops performed during the first stage of generating fitting images is less than the number of denoising loops performed during the second stage of generating fitting images.

[0140] Corresponding to Embodiment 4, this application also provides a fitting image generation device, which may include: The original information determination unit is used to determine a first clothing image for which a try-on image needs to be generated, and an original person image, wherein the second clothing worn by the person in the original person image is different from the first clothing in the first clothing image. The first editing area determination unit is used to determine the first editing area in the original human image based on the body parts to be tried on the first garment. The first-stage generation unit is used to generate a first-stage try-on image of the first garment within the first editing area using a pre-trained model. The second editing region determination unit is used to extract the outline of the first garment from the try-on image generated in the first stage, and determine the second editing region based on the region within the extracted garment outline. The third editing region determination unit is used to extract the outline of the second garment from the original character image and determine the third editing region based on the region within the extracted garment outline. The overlay processing unit is used to overlay the second editing region and the third editing region, and determine the target editing region in the original human image according to the overlay processing result. The target editing region is smaller than the first editing region, so that by shrinking the first editing region, more human body structure regions in the original human image are retained. Then, a pre-trained model is used to generate a second-stage try-on image in the target editing region to obtain the target try-on image.

[0141] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0142] And an electronic device, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0143] A computer program product includes a computer program / computer executable instructions that, when executed by a processor in an electronic device, implement the steps of the method described in the foregoing method embodiments.

[0144] in, Figure 7 An exemplary architecture of an electronic device is shown, which may include a processor 710, a video display adapter 711, a disk drive 712, an input / output interface 713, a network interface 714, and a memory 720. The processor 710, video display adapter 711, disk drive 712, input / output interface 713, network interface 714, and memory 720 can communicate with each other via a communication bus 730.

[0145] The processor 710 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to achieve the technical solution provided in this application.

[0146] The memory 720 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 720 can store the operating system 721 for controlling the operation of the electronic device 700, and the basic input / output system (BIOS) for controlling the low-level operations of the electronic device 700. Additionally, it can store a web browser 723, a data storage management system 724, and a try-on image generation and processing system 725, etc. The aforementioned try-on image generation and processing system 725 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 720 and is called and executed by the processor 710.

[0147] Input / output interface 713 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0148] Network interface 714 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0149] Bus 730 includes a pathway for transmitting information between various components of the device, such as processor 710, video display adapter 711, disk drive 712, input / output interface 713, network interface 714, and memory 720.

[0150] It should be noted that although the above-described device only shows the processor 710, video display adapter 711, disk drive 712, input / output interface 713, network interface 714, memory 720, bus 730, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0151] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0152] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as decoupling components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0153] The model training, try-on image generation method, and electronic device provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A model training method, characterized in that, include: Obtain a first training dataset, wherein the first training dataset includes multiple image pairs, the image pairs include clothing images and person images, the clothing worn by the person in the person image in the same image pair is the same clothing in the clothing image; the clothing image is a flat lay image of clothing. The model is trained in the first stage using the first training dataset. In the first stage of training, the model decouples clothing images from the input person images and supervises the decoupling results using clothing images from the original image pairs, so that the model can acquire the ability to decouple clothing images from person images; wherein, the decoupled clothing images are clothing tile images. Obtain a second training dataset, which includes multiple images of people. Use the model that has completed the first stage of training to decouple clothing images from the images of people, and use them as pseudo-labels for the corresponding images of people. The model is trained in the second stage using the images of people in the second training dataset and the corresponding pseudo-labels. In the second stage of training, the model generates a try-on image by putting the decoupled clothing image on the person in the original image based on the input image of people and the pseudo-labels, and supervises the generation result using the original image of people.

2. The method according to claim 1, characterized in that, The first stage of training also includes: the model generating a try-on image based on the input image pair, which puts the clothing image on the person in the image pair, and using the original person image in the image pair to supervise the generation result, so that the model can obtain the initial ability to generate try-on images. In the second stage of training, the model is trained using the images of people and their corresponding pseudo-labels from the second training dataset, building upon the model trained in the first stage.

3. The method according to claim 2, characterized in that, In the first stage, through a shared architecture, the model acquires the ability to generate try-on images and decouple clothing images from person images during the same training process; Specifically, for the same image pair, a first input pair and a second input pair are constructed. The first input pair includes: the clothing image in the image pair and the image of the person after occlusion processing of the target editing area. The second input pair includes: a blank image and the image of the person in the same image pair corresponding to the first input pair. By inputting prompt text, the model performs the task of generating a try-on image based on the first input pair and supervises the generation result using the image of the person in the original image pair. It also performs the task of decoupling the clothing image from the image of the person based on the second input pair and supervises the decoupling result using the image of the clothing in the original image pair.

4. A model training method, characterized in that, include: Obtain a training dataset, which includes multiple images of people; The clothing image is decoupled from the person image, wherein the decoupled clothing image is a clothing tile image, and the decoupled clothing image is used as a pseudo-label for the corresponding person image. The model is trained using the images of people in the training dataset and their corresponding pseudo-labels. The model generates a try-on image by placing the decoupled clothing image onto the person in the original image, based on the input image of the person and the pseudo-labels. The original image of the person is used to supervise the generation of the result.

5. The method according to claim 4, characterized in that, The method is preceded by: Multiple image pairs are acquired, each image pair including an image of clothing and an image of a person. The clothing worn by the person in the image of the same image pair is the same clothing as the clothing in the clothing image. The clothing image is a flat lay image of the clothing. The model is initially trained using the multiple image pairs, wherein the model generates a try-on image by placing the clothing image onto the person in the image based on the person image and clothing image in the input image pair, and the person image is used to supervise the generation result. The step of training the model using the person images and corresponding pseudo-labels in the training dataset includes: Using the images of people and their corresponding pseudo-labels in the training dataset, the model is trained on the basis of the initial training.

6. The method according to claim 4, characterized in that, The step of decoupling the clothing image from the person image includes: Using a pre-trained model capable of decoupling clothing images from person images, clothing images are decoupled from the person images; The similarity between clothing images in the decoupled clothing image domain of the person images is calculated by using a visual model, and / or the clothing integrity and background cleanliness of the decoupled clothing images are calculated. In order to filter out clothing images that do not meet the quality conditions through a multi-criteria filtering method, the person images and their corresponding pseudo-labels in the training dataset are determined.

7. A method for generating try-on images, characterized in that, include: Determine a first clothing image for which a try-on image needs to be generated, and an original person image, wherein the second clothing worn by the person in the original person image is different from the first clothing in the first clothing image; Based on the body parts to be tried on the first garment, a target editing area is determined in the original character image; wherein, the part of the original character image located within the target editing area is the part that needs to be modeled, and the part outside the target editing area is preserved; A try-on image of the first garment is generated within the target editing area using a pre-trained model; the pre-trained model is trained according to any one of claims 1 to 5.

8. The method according to claim 7, characterized in that, Determining the target editing region in the original image includes: Based on the body area to be tried on the first garment, a first editing area is determined in the original character image; Using a pre-trained model, a first-stage try-on image is generated for the first garment within the first editing area; Extract the outline of the first garment from the try-on images generated in the first stage, and determine the second editing area based on the area within the extracted garment outline; Extract the outline of the second garment from the original image of the person, and determine the third editing area based on the area within the extracted garment outline; The second editing region and the third editing region are superimposed, and the target editing region is determined in the original human image based on the superimposition result. The target editing region is smaller than the first editing region so that by shrinking the first editing region, more human body structure regions in the original human image are preserved. The pre-trained model is then used to generate a second-stage try-on image of the first garment in the target editing region to obtain the target try-on image.

9. The method according to claim 8, characterized in that, The step of determining the first editing area in the original character image based on the body area to be tried on the first garment includes: Based on the body area to be tried on the first garment, a general area without garment boundaries, length, or tightness restrictions is determined in the original human image; Based on the outline of the second garment in the original figure image, the area of ​​the second garment is determined in the original figure image; The first editing area is determined by overlaying the general area with the area within the boundary frame of the second clothing area.

10. The method according to claim 8, characterized in that, Also includes: Based on the body parts to be tried on for the first garment, the outlines of key body parts are extracted from the original image of the person, and a fourth editing region is determined based on the outlines of the key body parts. The second, third, and fourth editing regions are then superimposed, and the target editing region is determined in the original image based on the superimposition result.

11. The method according to claim 10, characterized in that, The step of determining the fourth editing region based on the contour of the key body parts includes: The area within the outline of the key body part is expanded within a preset range to determine the fourth editing area.

12. The method according to claim 7, characterized in that, The pre-trained model is based on the diffusion model; The number of denoising loops performed during the first stage of generating fitting images is less than the number of denoising loops performed during the second stage of generating fitting images.

13. A method for generating try-on images, characterized in that, include: Determine a first clothing image for which a try-on image needs to be generated, and an original person image, wherein the second clothing worn by the person in the original person image is different from the first clothing in the first clothing image; Based on the body parts to be tried on the first garment, a first editing area is determined in the original human image; Using a pre-trained model, a first-stage try-on image is generated for the first garment within the first editing area; Extract the outline of the first garment from the try-on images generated in the first stage, and determine the second editing area based on the area within the extracted garment outline; Extract the outline of the second garment from the original image of the person, and determine the third editing area based on the area within the extracted garment outline; The second editing region and the third editing region are superimposed, and the target editing region is determined in the original human image based on the superimposition result. The target editing region is smaller than the first editing region so that by shrinking the first editing region, more human body structure regions in the original human image are preserved. The pre-trained model is then used to generate a second-stage try-on image in the target editing region to obtain the target try-on image.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program performs the steps of the method described in any one of claims 1 to 13.

15. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 13.

16. A computer program product, comprising computer program instructions or computer executable instructions, characterized in that, When the computer program / computer-executable instructions are executed by a processor in an electronic device, they implement the steps of the method according to any one of claims 1 to 13.

Images