High-fidelity image virtual try-on method based on dual-network architecture and frequency domain constraint
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ROBOTICS RESEARCH CENTER OF YUYAO CITY
- Filing Date
- 2026-04-13
- Publication Date
- 2026-07-14
Smart Images
Figure CN122391403A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision, image processing, and artificial intelligence generated content (AIGC), specifically to a high-fidelity image virtual try-on method, computer equipment, and medium based on a dual-network architecture and frequency domain constraints. Background Technology
[0002] Virtual try-on technology aims to realistically composite target clothing onto a given human image, and has wide applications in e-commerce, social media, and virtual reality. With the development of deep learning, virtual try-on methods based on diffusion models are gradually becoming mainstream.
[0003] However, existing technologies still face two major challenges when dealing with complex scenarios: (1) Insufficient fidelity in clothing texture details. Existing methods (such as diffusion models based on restoration paradigms) tend to smooth images during the denoising process, resulting in severe loss or blurring of high-frequency information such as fine patterns, tiny logo text, complex embroidery or plaid on clothing, lacking realism.
[0004] (2) Limited shape generation capability. Most existing methods rely on the de-clothing mask extracted from the original image of the person to guide the generation. This strategy of strictly following the original outline cannot provide the generation space required for new clothing when dealing with cross-category or cross-size try-on (such as "short sleeve to long sleeve" or "shorts to trousers"), resulting in the generated clothing being truncated or having unreasonable shapes (e.g., the generated long sleeve is truncated to a short sleeve). Summary of the Invention
[0005] In view of the above-mentioned technical problems mentioned in the background art, the purpose of this invention is to provide a high-fidelity image virtual try-on method based on dual network architecture and frequency domain constraints.
[0006] The first aspect of the present invention provides a high-fidelity image virtual try-on method based on a dual-network architecture and frequency domain constraints, the method comprising the following steps: S1. Obtain the target person image, target clothing image, and person pose skeleton diagram; S2. Preprocess the target person image to generate a clothing removal mask, and execute a mask inflation strategy to adaptively expand the generation area of the clothing removal mask according to the target clothing category to obtain the final mask and background image. S3. Construct a dual-network architecture including a clothing feature network and a denoising network with consistent structure, and use the clothing feature network to extract multi-scale texture features of the target clothing image; S4. Input the background image, the skeleton diagram of the human pose, and the noise into the denoising network, and inject the multi-scale texture features into the corresponding layer of the denoising network through the reference attention mechanism to perform the reverse denoising process to generate the try-on image. S5. During training, the reconstruction loss between the generated predicted image and the real target image is calculated, and the frequency domain loss is calculated using discrete Fourier transform. The total loss function constructed by the reconstruction loss and the frequency domain loss is weighted based on the dynamic loss weight strategy to optimize the network parameters. S6. In the inference stage, the trained network is used to output the final high-fidelity try-on images.
[0007] As an example, in step S2, the mask inflation strategy includes differentiated processing for different clothing categories: For the jumpsuit category, the maximum bounding box strategy is adopted to construct a vertical rectangular bounding box covering from the shoulders to the ankles, and the hand and shoe areas are excluded as the generated range; For the upper garment category, a long-sleeve preset mechanism is adopted to preserve the original torso mask and morphologically expand the mask area towards the arm based on human key points to connect the shoulder and wrist. For the bottom category, a rectangular area from the lower edge of the waist to the key point of the foot is selected as the generation range, and the shoe area is excluded.
[0008] As an example, the calculation process of the frequency domain loss in step S5 includes: The generated predicted image and the real target image are transformed from the spatial domain to the frequency domain using the two-dimensional discrete Fourier transform. The predicted image and the real target image are spatially weighted using the final mask; The spectral distance between the predicted image and the real target image in the frequency domain is calculated based on the weighted frequency domain representation and used as the frequency domain loss.
[0009] As an example, in step S5, the dynamic loss weight strategy includes: The loss weights are dynamically adjusted based on the changes in the denoising time step, and the total loss function is weighted using the loss weights so that the loss weights corresponding to the low noise stage are higher than the loss weights corresponding to the high noise stage.
[0010] As an example, in step S4, the specific process of the reference attention mechanism is as follows: Feature writing stage: The clothing feature network propagates forward without adding noise, extracting the intermediate layer features of each self-attention module. And store it in the feature library; Feature reading stage: At denoising time step t, the denoising network reads the corresponding clothing features from the feature library and compares them with the currently generated features. Concatenate the sequences along the length dimension to construct extended key-value pairs for attention computation.
[0011] As an example, in step S1, the character pose skeleton map is subjected to feature extraction through a lightweight pose guidance module to obtain character pose features; This lightweight pose guidance module includes sequentially connected convolutional layers and several downsampling blocks. It uses 4×4 convolutional kernels with a stride of 2 for downsampling and the SiLU activation function. Finally, it outputs human pose features that are consistent with the input noise dimension of the denoising network through a projection convolutional layer.
[0012] A second aspect of the present invention provides a computer device including a memory and a processor, the memory storing a computer program, the processor being configured to, when executing the computer program, implement the high-fidelity image virtual try-on method as described in any of the preceding claims.
[0013] A third aspect of the present invention provides a computer-readable storage medium storing a computer program that, when run on a computer device, causes the computer device to perform the high-fidelity image virtual try-on method as described in any of the preceding claims.
[0014] Compared with the prior art, the present invention has at least the following beneficial technical effects: The dual-network architecture (dual U-Net) proposed in this invention decouples the clothing feature extraction and denoising generation processes, and achieves pixel-level high-fidelity feature transfer by using an attention mechanism, thus avoiding feature contamination by noise.
[0015] The mask inflation strategy proposed in this invention breaks the physical constraints of the original clothing mask, giving the model the ability to adaptively predict the shape of the target clothing, and effectively solving the problem of topological mismatch in cross-category try-on.
[0016] The frequency domain loss function and dynamic loss weight strategy introduced in this invention constrain the generation process from both the frequency and time domains, significantly solving the problem of "strong mid-frequency and weak high-frequency" in existing methods, and greatly improving the clarity and realism of complex textures, text and embroidery. Attached Figure Description
[0017] Figure 1 This is a flowchart illustrating a high-fidelity image virtual try-on method based on a dual-network architecture and frequency domain constraints disclosed in an embodiment of the present invention. Figure 2This is a schematic diagram of the dual-network architecture and feature injection mechanism disclosed in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a computer device disclosed in an embodiment of the present invention. Detailed Implementation
[0018] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.
[0019] Please see Figure 1 , Figure 2 This invention provides a high-fidelity image virtual try-on method based on a dual-network architecture and frequency domain constraints. This method abandons the traditional two-stage paradigm of geometric deformation and image fusion, and adopts a single-stage end-to-end generation strategy based on feature injection. It can simultaneously complete garment shape adaptation and texture preservation in a unified inference process. The method includes the following steps: S1. Obtain the target person image, target clothing image, and person pose skeleton diagram; In this embodiment, the target person image is used to provide overall appearance information of the person, including human body outline, limb posture, original clothing form and background environment; the target clothing image is used to provide appearance information of the clothing to be tried on, including the texture details, pattern structure, color distribution and high-frequency visual features of the clothing; the person posture skeleton map is used to provide explicit geometric constraints for the subsequent clothing generation process to ensure that the generated clothing is consistent with the person posture in spatial structure.
[0020] Understandably, the process of extracting a person's pose involves, for example, using keypoint detection tools such as OpenPose or DensePose to process the target person's image, extracting keypoint information of the person's limbs (such as shoulders, elbows, wrists, knees, ankles, etc.), and generating a person's pose skeleton map with the same resolution as the input image.
[0021] By introducing the three types of data mentioned above simultaneously in the initial stage of the method, a balance can be achieved between the consistency of the character's structure and the realism of the clothing texture in the subsequent generation process, avoiding the morphological distortion problems caused by relying solely on image restoration or image fusion methods. In addition, to adapt to high-resolution generation models, all the above images can be uniformly adjusted to a standard resolution (e.g., 768×1024 pixels).
[0022] As an example, in step S1, the character pose skeleton map is subjected to feature extraction through a lightweight pose guidance module to obtain character pose features; This lightweight pose guidance module includes sequentially connected convolutional layers and several downsampling blocks. It uses 4×4 convolutional kernels with a stride of 2 for downsampling and the SiLU activation function. Finally, it outputs human pose features that are consistent with the input noise dimension of the denoising network through a projection convolutional layer.
[0023] This invention transforms the human posture skeleton diagram, which is represented in a structural form, into a posture feature representation that can directly participate in the diffusion model denoising generation process, thereby providing stable and continuous constraints on the spatial structure and geometric consistency of clothing generation in the subsequent virtual try-on image generation process.
[0024] It should be noted that the human posture skeleton map itself is usually represented in the form of key points or line segments, only containing the spatial positional relationships of human joints, making it difficult to directly align with the feature space of a denoising network built based on convolution and attention mechanisms. If this posture skeleton map is directly input into a denoising network as a regular image, it can easily lead to insufficient utilization of posture information or inadequate fusion with noise features. Therefore, this invention further introduces a lightweight posture guidance module in step S1 to perform feature processing on the human posture skeleton map to obtain human posture features, enabling the posture information to be effectively fused with noise features and background image features at the feature level.
[0025] In this embodiment, the character pose skeleton map is used for feature extraction via a lightweight pose guidance module to obtain character pose features. This process does not alter the geometric and semantic information contained in the character pose skeleton map, but rather performs multi-layer convolution and downsampling processing to extract pose feature representations with spatial contextual relationships.
[0026] Specifically, the lightweight pose guidance module consists of several sequentially connected convolutional layers and downsampling blocks. Each downsampling block uses a 4×4 convolutional kernel and a convolutional operation with a stride of 2. While gradually reducing the spatial resolution of the feature map, it expands the receptive field, enabling the network to understand the human pose structure from a holistic perspective, rather than just focusing on the location of local key points.
[0027] After each convolution or downsampling operation, the SiLU activation function is used to perform a non-linear mapping on the features. Compared to the traditional ReLU activation function, the SiLU activation function has a smoother gradient change in the negative range, which helps to maintain the continuity and stability of pose features in deep networks and avoids abrupt changes or loss of pose information during propagation.
[0028] After completing multi-level convolution and downsampling processing, a projection convolutional layer is used to map the pose features to the channel dimension, so that the final output of the human pose features is consistent with the dimension of the input noise of the denoising network in terms of feature dimension.
[0029] Through the above design, the human pose features can be directly aligned with the noise features input to the denoising network in the channel dimension, thereby achieving feature-level fusion in the subsequent inverse denoising process. This fusion method ensures that the denoising network is always constrained by the human pose as it gradually restores the image structure, ensuring that the generated spatial structures such as clothing outline, sleeve length, and pant leg direction are consistent with the human's actual pose.
[0030] S2. Preprocess the target person image to generate a clothing removal mask, and execute a mask inflation strategy to adaptively expand the generation area of the clothing removal mask according to the target clothing category to obtain the final mask and background image. In this embodiment, this step is one of the key preprocessing steps that distinguishes the present invention from traditional virtual try-on methods. Its core purpose is not simply to remove the clothing area from the original person image, but to provide sufficient and reasonable generation space for the subsequent clothing generation process through an adaptive mask inflation strategy.
[0031] It should be noted that existing virtual try-on technologies generally adopt the approach of "clothing removal masking + image inpainting," which means strictly constructing the generated area based on the outline of the person's current clothing. This method is applicable when the clothing category and size are consistent. However, in practical applications, when there are significant differences between the target clothing and the original clothing in terms of category or coverage (e.g., changing from short sleeves to long sleeves, shorts to trousers, tops to dresses, etc.), the original clothing removal mask often cannot cover the entire generated area required for the new clothing. This results in problems such as truncated sleeves, insufficient trouser leg length, or incomplete overall shape in the generated result.
[0032] To address the aforementioned technical issues, this embodiment introduces a mask expansion strategy in step S2, and adaptively expands the generation area based on the category differences of the target clothing, thereby actively breaking the limitations of the original clothing outline on the generation space. The final mask obtained through this step not only includes the original clothing area but also covers the potential new clothing areas that need to be generated; simultaneously, the background image is extracted from the target person image using this final mask, providing a stable input for subsequent diffusion generation.
[0033] In this embodiment, the target person image is first processed by human body analysis or semantic segmentation to distinguish the person's body, original clothing, head, hands, shoes, and background area. This process yields a basic de-clothing mask representing the original clothing area. Subsequently, this basic de-clothing mask is used to perform preliminary processing on the target person image, removing or replacing the original clothing area with a preset value (e.g., noise or blank areas), while preserving the person's head, hands, shoes, and background environment. It should be noted that this step is only used to construct the initial generation framework; the generated area will still be further adjusted using subsequent mask inflation strategies.
[0034] As an example, in step S2, the mask inflation strategy includes differentiated processing for different clothing categories: For the jumpsuit category, the maximum bounding box strategy is adopted to construct a vertical rectangular bounding box covering from the shoulders to the ankles, and the hand and shoe areas are excluded as the generated range; In practical implementation, when the target clothing category is a dress, since this type of clothing typically covers most of the human torso and lower limbs, and has a wide range of variations in length, outline, and cut, relying solely on the original clothing mask for generation can easily lead to insufficient generation area. Therefore, this invention adopts a maximum bounding box strategy. Specifically, based on the shoulder and ankle key points in the human posture skeleton diagram, a vertical rectangular bounding box is constructed from top to bottom, covering the entire area from the shoulder to the ankle vertically; horizontally, it covers the maximum width range of the human torso. Subsequently, the hand and shoe areas are explicitly removed from this bounding box to avoid erroneous generation of the human limb ends and footwear.
[0035] The generation range constructed in the above manner provides ample generation space for long dresses, gowns, or irregular skirts, enabling the model to autonomously learn and generate reasonable skirt lengths and shapes within this range.
[0036] For the upper garment category, a long-sleeve preset mechanism is adopted to preserve the original torso mask and morphologically expand the mask area towards the arm based on human key points to connect the shoulder and wrist. In practical implementation, when the target clothing category is a top, considering the significant differences in sleeve length among tops, this invention introduces a long-sleeve preset mechanism to avoid the generated area being limited to the original short-sleeved or sleeveless clothing range. Specifically: while preserving the original torso mask, morphological expansion operations are performed on the mask area along the arm direction using key points of the shoulder, elbow, and wrist in the human posture skeleton diagram, so that the mask area extends continuously from the shoulder to the wrist.
[0037] In this way, the generated area spatially covers the entire arm, thus providing the necessary canvas for the generation of long-sleeved clothing.
[0038] It should be noted that even if the target garment is actually short-sleeved, the above-mentioned extended area will not have a negative impact on the final generated result, because the specific garment shape generated will be determined autonomously by the subsequent denoising network in combination with the garment texture features; while when the target garment is long-sleeved, this strategy can effectively avoid the problem of the sleeves being truncated.
[0039] For the bottom category, a rectangular area from the lower edge of the waist to the key point of the foot is selected as the generation range, and the shoe area is excluded.
[0040] In practice, when the target clothing category is bottoms, a keypoint-based rectangular region generation method is adopted to address the shape differences between shorts and trousers. Specifically, the lower edge keypoint of the waist in the human pose skeleton image is selected as the upper boundary of the generation region, and the foot keypoint is selected as the lower boundary to construct a rectangular region covering the entire lower limb. At the same time, using the human body analysis results, the shoe area is removed from this rectangular region to maintain the integrity of the footwear.
[0041] This strategy generates a region that covers the entire vertical range from the waist to the ankle, allowing the model to autonomously determine the ending position and shape of the trouser leg within this region, thus adapting to different bottom styles such as shorts, capri pants, or trousers.
[0042] After performing the mask dilation processing for different clothing categories, the dilated generated region is merged with the original base de-clothing mask to obtain the final mask. This final mask is then used to process the target image, removing or replacing pixels within the generated region while preserving the head, hands, shoes, and background environment outside the generated region, thus obtaining the background image used for subsequent denoising.
[0043] S3. Construct a dual-network architecture including a clothing feature network and a denoising network with consistent structure, and use the clothing feature network to extract multi-scale texture features of the target clothing image; The core of this step lies in constructing a dual-network architecture that is structurally decoupled but hierarchically aligned, and through this architecture, effectively separating and coordinating clothing texture information with the diffusion generation process.
[0044] It should be noted that existing virtual try-on methods based on diffusion models typically model clothing information and the generation process together in the same network. In this case, clothing texture features are inevitably affected by random noise during the denoising process, especially in the early stages of multi-step denoising. High-frequency details of the clothing (such as fine patterns, embroidery textures, or text labels) are often gradually smoothed out or even lost, resulting in a lack of realism in the final generated result.
[0045] To address the aforementioned issues, this invention introduces a dual-network architecture design in step S3. Specifically, the extraction process of clothing texture features is decoupled from the image denoising generation process at the network structure level, with the clothing feature network and the denoising network each undertaking different functions. This avoids repeated perturbation of clothing features in the noise space. The dual-network architecture includes a structurally consistent clothing feature network (Garment U-Net) and a denoising network (Denoising U-Net). It should be noted that "structural consistency" means that the two networks maintain consistency in network layers, downsampling and upsampling structures, and the placement of self-attention modules, enabling the establishment of stable feature mapping relationships between corresponding layers in subsequent steps.
[0046] Among them, the clothing feature network is used to extract forward features from the target clothing image. Its main task is to capture the texture, semantic and structural information of clothing at different spatial resolutions. The denoising network is used to perform the inverse denoising process of the latent diffusion model. Its main task is to combine the background image, pose features, noise and clothing texture features to gradually generate the final try-on image.
[0047] Through the above dual-network architecture design, the clothing feature network can focus on extracting high-quality clothing texture features without introducing noise interference; while the denoising network can selectively call these high-quality features during the generation process through the reference attention mechanism in subsequent steps, thereby achieving the synergy between texture fidelity and form generation.
[0048] In this embodiment, the input to the clothing feature network is a target clothing image. This target clothing image is typically a tiled or standardized photograph of a single garment, with a relatively simple background that clearly reflects the texture and visual details of the clothing itself. Unlike denoising networks, the clothing feature network does not introduce random noise during forward propagation, nor does it participate in the time-step evolution of the diffusion process. Instead, it performs a single complete forward inference on the target clothing image. In this way, the clothing feature network can fully preserve the high-frequency texture information and local structural features of the clothing under stable input conditions.
[0049] During network propagation, the clothing feature network can extract intermediate feature representations (Hidden States) of the corresponding level (Downsample and Upsample stages) through the self-attention module in each downsampled and upsampled layer. These intermediate feature representations change gradually from coarse to fine in spatial resolution, corresponding to the overall outline structure of the clothing, the distribution of medium-scale patterns, and the fine-grained texture details, respectively.
[0050] In this embodiment, during the forward propagation process, the clothing feature network extracts multi-scale clothing texture features from multiple levels and stores the extracted features in a feature library for subsequent use. Specifically, in the low-resolution levels of the network, the extracted features mainly reflect the overall shape, regional division, and large-scale semantic structure of the clothing; in the medium-resolution levels, the extracted features mainly reflect the main patterns, texture distribution, and color block structure of the clothing; and in the high-resolution levels, the extracted features include information such as the fine texture of the clothing surface, high-frequency details, embroidery edges, and text outlines.
[0051] By extracting features at multiple levels simultaneously and aligning them with the corresponding levels of the denoising network in a structurally consistent manner, the necessary prerequisite for feature injection based on the reference attention mechanism in the subsequent step S4 is provided, enabling the denoising network to call clothing texture features of appropriate scale at different generation stages.
[0052] S4. Input the background image, the skeleton diagram of the human pose, and the noise into the denoising network, and inject the multi-scale texture features into the corresponding layer of the denoising network through the reference attention mechanism to perform the reverse denoising process to generate the try-on image. This step is the core generation step for realizing the virtual try-on image generation. Based on the inverse denoising mechanism of the latent diffusion model, this step comprehensively utilizes background image information, human pose constraints, and multi-scale texture features from the clothing feature network in the process of gradually eliminating noise, and finally generates a try-on image that is consistent with the human pose and has high-fidelity clothing texture.
[0053] It should be noted that in traditional diffusion models, the denoising network typically relies solely on latent noise variables and conditional inputs (such as text or image conditions) for generation. Conditional information is easily weakened or blurred during multi-step denoising, especially high-frequency texture information, which is easily lost after multiple convolutions and attention operations. To address this issue, this invention introduces a reference attention mechanism in step S4. By explicitly introducing multi-scale texture features extracted from the clothing feature network into the denoising network, clothing texture information can continuously and stably participate in generation throughout the entire inverse denoising process.
[0054] In this embodiment, the denoising network is used to perform the inverse denoising process of the latent diffusion model. Its input is not a single noise signal, but rather a combination of various information, including: Noise input: The noise input is random noise corresponding to the diffusion time step, which is used to drive the inverse denoising process.
[0055] The latent representation corresponding to the background image: After encoding the background image obtained in step S2 into the latent space, it is used together with the noise input as one of the inputs to the denoising network, so that the generation result is consistent in the background environment and the non-generated area of the person.
[0056] Character pose features: The character pose skeleton map is used to extract the character pose features through the lightweight pose guidance module (Pose Encoder), which are then input into the denoising network to provide explicit geometric constraints during the generation process, ensuring that the generated clothing matches the character pose in spatial structure.
[0057] Through the fusion of the above multi-source inputs, the denoising network can simultaneously perceive the noise state, background structure, and character pose at each time step of the inverse denoising process, thus providing a stable foundation for subsequent clothing generation.
[0058] As an example, in step S4, the specific process of the reference attention mechanism is as follows: Feature writing stage: The clothing feature network propagates forward without adding noise, extracts the intermediate layer features of each self-attention module, and stores them in the feature library; Feature reading stage: At the denoising time step t, the denoising network reads the corresponding clothing features from the feature library and concatenates them with the currently generated features in the sequence length dimension to construct extended key-value pairs for attention calculation.
[0059] In this embodiment, unlike the traditional cross-attention mechanism based on text as key-value pairs, the attention mechanism uses spatial features extracted by the clothing feature network as reference information, and concatenates the spatial features with the current features of the denoising network to expand the key-value pairs in the attention calculation. This allows the denoising network to directly reference the texture information in the reference clothing image during the generation process, thereby improving texture consistency.
[0060] The attention mechanism first includes a feature writing phase. In this phase, the clothing feature network performs forward propagation on the target clothing image without introducing random noise. Unlike denoising networks, the clothing feature network does not participate in the diffusion time step evolution, and its propagation process remains stable, thus enabling the complete and accurate extraction of texture and structural information from the clothing image.
[0061] During the forward propagation process, the clothing feature network extracts intermediate layer features from self-attention modules at multiple levels. These intermediate layer features correspond to clothing texture representations at different spatial resolutions, including overall garment outline information, mid-scale pattern structure, and fine-grained high-frequency texture details. The extracted intermediate layer features are uniformly stored in a feature library for later use in the denoising process.
[0062] Through this feature writing stage, the clothing texture information is frozen into a stable feature representation before entering the generation process, thereby avoiding interference from random noise during the denoising process.
[0063] In this embodiment, the second stage of the attention mechanism is the feature reading stage. This stage occurs at each time step t during the inverse denoising process performed by the denoising network. Specifically, at each denoising time step t, the denoising network reads the clothing features corresponding to the current network layer from the feature library. Subsequently, the read clothing features are compared with the currently generated features. By concatenating along the sequence length dimension, extended key-value pairs are constructed for attention computation.
[0064] During the attention calculation process, the features generated by the denoising network serve as the query vector, while the expanded key-value pairs simultaneously contain both the currently generated features and the clothing texture features. In this way, when updating features, the denoising network can explicitly query and reference the high-quality texture information extracted from the clothing feature network, thereby copying and injecting the clothing texture into the generated result.
[0065] It should be noted that the above splicing and attention calculation method is not a simple feature superposition, but rather an adaptive determination of the influence of clothing texture features on the generation process through attention weights, thereby ensuring texture fidelity while avoiding interference with the overall generated structure.
[0066] In this embodiment, by simultaneously introducing the background image, the figure's pose features, and the clothing texture features into the denoising network, the inverse denoising process becomes a multi-condition collaborative constraint generation process.
[0067] In the early stages of denoising, the network mainly focuses on the overall structure and semantic layout, determining the outline of the figure and the general shape of the clothing through the background image and the figure's posture features. In the later stages of denoising, as the noise intensity gradually decreases, the attention mechanism is used to guide the network to gradually enhance the details of the clothing texture, so that high-frequency patterns, embroidery edges and text structures can be clearly presented in the generated results.
[0068] Through the aforementioned collaborative mechanism, the generated virtual try-on images not only maintain consistency with the person's posture in terms of clothing shape, but also closely resemble real clothing images in terms of texture details, achieving a high-fidelity virtual try-on effect.
[0069] S5. During training, the reconstruction loss between the generated predicted image and the real target image is calculated, and the frequency domain loss is calculated using discrete Fourier transform. The total loss function constructed by the reconstruction loss and the frequency domain loss is weighted based on the dynamic loss weight strategy to optimize the network parameters. This step occurs during the model training phase and is used to optimize the network parameters in the dual-network architecture. By simultaneously introducing spatial and frequency domain constraints at the loss function level and combining the dynamic characteristics of the diffusion model at different denoising stages, this step applies multi-dimensional constraints to the generated results, thereby improving the performance of the virtual try-on image in terms of texture detail and overall consistency.
[0070] It should be noted that in the image generation process based on the diffusion model, traditional loss functions typically focus on pixel reconstruction errors in the spatial domain. While this form of loss can constrain the generated image to approximate the real image in terms of overall structure and low-frequency information, in the multi-step denoising process, the model often tends to prioritize optimizing low-frequency components, resulting in excessive smoothing of high-frequency texture details (such as fine patterns on clothing, embroidery edges, or text structures). To address this issue, this embodiment introduces frequency domain loss in step S5 and further combines it with a dynamic loss weighting strategy, enabling the model to focus on different types of error information at different denoising stages, thereby achieving a balance between overall structure and local details.
[0071] As an example, the calculation process of the frequency domain loss in step S5 includes: The generated predicted image and the real target image are transformed from the spatial domain to the frequency domain using the two-dimensional discrete Fourier transform. The predicted image and the real target image are spatially weighted using the final mask; The spectral distance between the predicted image and the real target image in the frequency domain is calculated based on the weighted frequency domain representation and used as the frequency domain loss.
[0072] In practice, the generated predicted image is first generated using a two-dimensional discrete Fourier transform. and real target image The transformation is performed from the spatial domain to the frequency domain; subsequently, the final mask obtained in step S2 is used. Spatially weight the predicted image and the real target image; after the above processing, calculate the spectral distance between them in the frequency domain as the frequency domain loss. .
[0073] The formula for calculating the frequency domain loss is as follows:
[0074] in: This indicates the Fast Fourier Transform operation; This indicates element-wise multiplication; For frequency coordinates; H and W represent the dimensions of the frequency domain representation in two dimensions, respectively. This represents the predicted image generated during the current denoising stage; Represents a real target image; This represents the final mask.
[0075] By defining the frequency domain loss as described above, the model not only constrains the consistency between the generated result and the real image in the pixel space during training, but also explicitly aligns their spectral distributions in the frequency domain space, especially strengthening the ability to constrain high-frequency information such as clothing texture and edge structure.
[0076] To accommodate the differences in learning focus of the diffusion model at different denoising time steps, a dynamic loss weighting strategy is further introduced to weight the total loss function.
[0077] As an example, in step S5, the dynamic loss weight strategy includes: The loss weights are dynamically adjusted based on the changes in the denoising time step, and the total loss function is weighted using the loss weights so that the loss weights corresponding to the low noise stage are higher than the loss weights corresponding to the high noise stage.
[0078] In practice, a weight function is defined that monotonically decreases with the denoising time step t. As shown below:
[0079] in: Indicates the current denoising time step; This represents the maximum number of time steps during the diffusion process; As an enhancing factor; This is the temperature coefficient.
[0080] In this embodiment, the above-mentioned weighting function is used. The total loss function, composed of reconstruction loss and frequency domain loss, is weighted to dynamically change the influence of the loss term on network parameter optimization at different denoising stages. Specifically, in the early stages of denoising, when the corresponding time step is relatively large, the weight function... The value of is relatively small, allowing the model to primarily focus on the overall structure and semantic layout during the high-noise phase; as the denoising process progresses, the time step t gradually decreases, and the weight function... The value of is gradually increased, so that the loss weight in the low noise stage is higher than that in the high noise stage, guiding the model to pay more attention to the fine reconstruction of clothing texture details and high frequency structures.
[0081] During training, the weighted total loss function is used as the optimization objective, and the network parameters in the dual-network architecture are updated through backpropagation. By leveraging the synergistic effect of frequency domain loss and dynamic loss weighting strategies, the model significantly improves the clarity and realism of clothing textures in the generated try-on images while maintaining overall structural consistency.
[0082] S6. In the inference stage, the trained network is used to output the final high-fidelity try-on images.
[0083] This step occurs during the inference phase after model training is complete. Its core purpose is to perform a forward generation process on new input samples using the network parameters trained in step S5, without relying on supervision from real target images, thereby outputting the final high-fidelity virtual try-on image.
[0084] It should be noted that while the inference and training phases maintain the same network structure, they differ significantly in data input format, loss calculation, and parameter update mechanisms. Specifically, in the inference phase, real target images are no longer introduced, nor is reconstruction loss or frequency domain loss calculated. Instead, the inverse denoising generation process is performed using only the network parameters optimized during the training phase, thereby achieving end-to-end virtual try-on effect output.
[0085] In this embodiment, the input data for the inference phase includes: Target person image: used to provide the appearance information of the person and the background environment. Its processing method is the same as in step S2, and the background image is obtained through the clothing removal mask and mask dilation strategy.
[0086] Target clothing image: used to provide texture and structure information of the clothing to be tried on. Multi-scale clothing texture features are extracted through the clothing feature network in step S3 and stored in the feature library.
[0087] Character pose skeleton diagram: The character pose features are extracted through the lightweight pose guidance module in step S1 and used to provide geometric constraints on the clothing shape during the generation process.
[0088] Random noise: serves as the initial input to the inverse denoising process of the latent diffusion model, driving the generation process.
[0089] Through the synergistic effect of the aforementioned input data, the reasoning stage can complete the task of generating high-quality clothing without the need for real try-on images as a reference.
[0090] In this embodiment, the trained network includes at least: Denoising network: Its parameters have been optimized through the training process in step S5, and it is used to perform inverse denoising generation of the potential diffusion model; Clothing Feature Network: Used to extract multi-scale texture features from target clothing images during the inference phase, and its structure remains consistent with that of the training phase.
[0091] During the inference phase, the clothing feature network first performs a forward propagation on the target clothing image, extracting clothing texture features at each level and storing them in the feature library. Subsequently, the denoising network, under the initial noise input conditions, combines the background image, the person's pose features, and the clothing texture features introduced through the reference attention mechanism to perform a multi-step inverse denoising process.
[0092] It is important to emphasize that during the inference phase, the denoising process relies entirely on the parameter distribution and feature mapping relationship learned during the training phase, and no further parameter update operations are involved, thus ensuring the determinism and stability of the generation process.
[0093] In this embodiment, the denoising network starts from the time step with the maximum noise and performs inverse denoising operations step by step according to a preset denoising scheduling strategy. As the denoising time step decreases, random noise is gradually removed, and the image latent variables gradually converge to an image representation with clear structure and texture details.
[0094] During the denoising process, the reference attention mechanism is continuously effective at each time step, enabling the multi-scale texture features extracted from the clothing feature network to be dynamically referenced at different generation stages, thereby gradually enhancing the texture details of the clothing while ensuring the consistency of the character's pose.
[0095] When the inverse denoising process reaches the final time step, the denoising network outputs a stable latent variable representation of the image. Subsequently, the latent variable is mapped back from the latent space to the image space through a decoding operation to obtain the final virtual try-on image.
[0096] It is understandable that the final high-fidelity try-on image output in step S6 has the following characteristics: Character pose consistency: The generated clothing maintains the same spatial structure as the input character pose, without any limb misalignment or proportional distortion.
[0097] Clothing texture fidelity: The pattern, color distribution, and high-frequency details of the clothing are highly consistent with the target clothing image, avoiding blurry or distorted textures.
[0098] Background and non-generated area integrity: The character's head, hands, shoes, and background environment retain their original appearance and are not affected by the generation process.
[0099] Understandably, the generated high-fidelity try-on images can be directly used in applications such as virtual try-on displays, clothing recommendations, e-commerce visualization, or digital human generation.
[0100] like Figure 3As shown, this application provides a computer device including a processor, a memory, and a network interface. The memory stores a computer program containing all the instruction code required to execute the aforementioned high-fidelity virtual try-on algorithm, as well as pre-trained Stable Diffusion model weights. The processor is typically a high-performance GPU (such as a graphics processing unit) used to perform convolution operations, Fourier transforms, and matrix multiplications in parallel. When the processor executes the program in the memory, it implements the steps described in S1 to S5, outputting a high-fidelity try-on image.
[0101] This invention also provides a computer-readable storage medium storing a computer program that, when run on a computer device, causes the computer device to perform the high-fidelity image virtual try-on method as described in any of the preceding claims.
[0102] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A high-fidelity image virtual try-on method based on a dual-network architecture and frequency domain constraints, characterized in that, The method includes the following steps: 0001.S1. Obtain the target person image, target clothing image, and person pose skeleton diagram; S2. Preprocess the target person image to generate a clothing removal mask, and execute a mask inflation strategy to adaptively expand the generation area of the clothing removal mask according to the target clothing category to obtain the final mask and background image. S3. Construct a dual-network architecture including a clothing feature network and a denoising network with consistent structure, and use the clothing feature network to extract multi-scale texture features of the target clothing image; S4. Input the background image, the skeleton diagram of the human pose, and the noise into the denoising network, and inject the multi-scale texture features into the corresponding layer of the denoising network through the reference attention mechanism to perform the reverse denoising process to generate the try-on image. S5. During training, the reconstruction loss between the generated predicted image and the real target image is calculated, and the frequency domain loss is calculated using discrete Fourier transform. The total loss function constructed by the reconstruction loss and the frequency domain loss is weighted based on the dynamic loss weight strategy to optimize the network parameters. S6. In the inference stage, the trained network is used to output the final high-fidelity try-on images.
2. The high-fidelity image virtual try-on method based on a dual-network architecture and frequency domain constraints according to claim 1, characterized in that: In step S2, the mask inflation strategy includes differentiated processing for different clothing categories: For the jumpsuit category, the maximum bounding box strategy is adopted to construct a vertical rectangular bounding box covering from the shoulders to the ankles, and the hand and shoe areas are excluded as the generated range; For the upper garment category, a long-sleeve preset mechanism is adopted to preserve the original torso mask and morphologically expand the mask area towards the arm based on human key points to connect the shoulder and wrist. For the bottom category, a rectangular area from the lower edge of the waist to the key point of the foot is selected as the generation range, and the shoe area is excluded.
3. The high-fidelity image virtual try-on method based on a dual-network architecture and frequency domain constraints according to claim 1, characterized in that: The calculation process of frequency domain loss in step S5 includes: The generated predicted image and the real target image are transformed from the spatial domain to the frequency domain using the two-dimensional discrete Fourier transform. The predicted image and the real target image are spatially weighted using the final mask; The spectral distance between the predicted image and the real target image in the frequency domain is calculated based on the weighted frequency domain representation and used as the frequency domain loss.
4. The high-fidelity image virtual try-on method based on a dual-network architecture and frequency domain constraints according to claim 3, characterized in that: In step S5, the dynamic loss weight strategy includes: The loss weights are dynamically adjusted based on the changes in the denoising time step, and the total loss function is weighted using the loss weights so that the loss weights corresponding to the low noise stage are higher than the loss weights corresponding to the high noise stage.
5. The high-fidelity image virtual try-on method based on a dual-network architecture and frequency domain constraints according to claim 4, characterized in that: In step S4, the specific process of the reference attention mechanism is as follows: Feature writing stage: The clothing feature network propagates forward without adding noise, extracting the intermediate layer features of each self-attention module. And store it in the feature library; Feature reading stage: At denoising time step t, the denoising network reads the corresponding clothing features from the feature library and compares them with the currently generated features. Concatenate the sequences along the length dimension to construct extended key-value pairs for attention computation.
6. The high-fidelity image virtual try-on method based on a dual-network architecture and frequency domain constraints according to claim 1, characterized in that: In step S1, the character pose skeleton map is subjected to feature extraction through a lightweight pose guidance module to obtain character pose features; This lightweight pose guidance module includes sequentially connected convolutional layers and several downsampling blocks. It uses 4×4 convolutional kernels with a stride of 2 for downsampling and the SiLU activation function. Finally, it outputs human pose features that are consistent with the input noise dimension of the denoising network through a projection convolutional layer.
7. A computer device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to, when executing the computer program, implement the high-fidelity image virtual try-on method as described in any one of claims 1 to 6.
8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when run on a computer device, causes the computer device to perform the high-fidelity image virtual try-on method as described in any one of claims 1 to 6.