Virtual fitting method based on large-model semantic recognition and double-model cascading
By using a virtual try-on method based on large-scale semantic recognition and dual-model cascading, the problems of limited clothing types and distortion in existing technologies are solved, achieving high-quality and natural virtual try-on effects.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-05
- Publication Date
- 2026-03-13
AI Technical Summary
Existing virtual try-on technology models are limited to a single type of clothing, and the generated results are easily affected by the original clothing pattern. It is difficult to balance structural integrity and texture detail, and the posture retention and naturalness of the face and hands are insufficient, resulting in a decline in user experience.
By employing a large-model semantic recognition and dual-model cascading approach, the collaborative execution path of IDM-VTON and CAT-VTON is dynamically selected through semantic understanding, and post-processing enhancement is performed in conjunction with traditional computer vision modules to achieve high-quality costume replacement effects.
It enables intelligent decision-making and dynamic scheduling for different outfit types, improving the structural integrity, texture detail, and pose consistency of the generated images, thus enhancing the user experience.
Smart Images
Figure CN121661182A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and artificial intelligence technology. Specifically, it is a virtual try-on method based on large-model semantic recognition and dual-model cascading. By introducing a large language model to perform semantic understanding of the input image, combining the collaborative mechanism of IDM-VTON and CAT-VTON models, and integrating traditional computer vision technology for post-processing enhancement, a high-quality and high-fidelity virtual dress-up effect can be achieved. Background Technology
[0002] With the rapid development of e-commerce and the digital content industry, Virtual Try-On (VTON) technology has shown great potential in e-commerce platforms, social applications, and digital human creation. Current mainstream methods mainly rely on deep generative models to complete clothing replacement tasks. Among them, IDM-VTON performs excellently in texture detail restoration and edge sharpness, but its capabilities are limited to supporting only top replacements, and the generated results are easily affected by the original garment pattern—for example, when replacing a short dress with a long one, "truncated" generation often occurs, resulting in missing hems or proportional distortion. On the other hand, while CAT-VTON supports full-body dress replacements and is less dependent on the original pattern, possessing stronger structural generation capabilities, its generated images often suffer from quality issues such as pose distortion and facial blurring, lacking sufficient detail.
[0003] Current research has attempted to improve the quality of character transformations through multi-model fusion, but most methods employ fixed pipelines or simple overlays, lacking intelligent perception and dynamic scheduling capabilities for the input content. Some works have introduced posture control signals or attention mechanisms for optimization, but it is still difficult to balance structural integrity and texture detail. Furthermore, existing systems generally neglect the pose consistency and naturalness of facial and hand features in the generated results, leading to a degraded user experience.
[0004] Therefore, there is an urgent need for a virtual try-on method that can make dynamic decisions based on the semantics of the input image, give full play to the advantages of different models, and effectively suppress generation distortion, so as to achieve a more natural and realistic dressing effect. Summary of the Invention
[0005] Purpose of the invention: To address the limitations of existing technologies, such as model capabilities, restrictions on clothing types, and generation distortion, the purpose of this invention is to provide a virtual try-on method based on large-model semantic recognition and dual-model cascading. By introducing a large model to achieve deep understanding of the input image, and combining a policy routing mechanism to dynamically select the collaborative execution path of IDM-VTON and CAT-VTON, and introducing a traditional computer vision module to post-process and enhance the output results, the quality and stability of the clothing try-on images are significantly improved.
[0006] Technical solution: A virtual try-on method based on large-model semantic recognition and dual-model cascading, comprising the following steps: Step S101: Call the large model to perform semantic understanding on the input model image and target clothing image, and identify the current clothing type of the model and the target clothing type. The clothing type includes top, short dress, long dress or no clothing. Step S102: Based on the recognition results, make a dress-changing path decision based on preset strategy rules: if the model is half-body and the target garment is a dress, or the model is full-body and changes from a short dress to a long dress, or the original garment and the target garment have different styles, then select the "fusion version" dress-changing strategy; if only the top is changed or the styles are the same, then select the IDM-VTON separate execution strategy; if the target garment is a full-body dress change that is not a dress, then it is determined that the scene is not supported and a prompt is returned. Step S103: For the scenario of the "fusion version" strategy, first call the CAT-VTON model to complete the initial dressing change and generate a dress image with a complete structure; Step S104: Perform image preprocessing on the CAT-VTON output image, including pose key point alignment, clothing area mask generation and background unification, to adapt to IDM-VTON input requirements; Step S105: Using the preprocessed image as input, the IDM-VTON model is called for a second makeover process to refine the texture, edges and details, and generate a high-quality makeover result. Step S106: The final generated image is post-processed and enhanced using a traditional computer vision module. OpenPose is used to detect the consistency of pose key points before and after generation. The face and hand regions are segmented using the Media Pipe model. Image inpainting techniques are used to correct distorted local areas to ensure stable pose and natural face and hand features.
[0007] 2. The virtual try-on method based on large-model semantic recognition and dual-model cascading as described in claim 1, characterized in that the semantic understanding in step S101 specifically includes the following sub-steps: Step S1011: Image structure judgment: For the model image An overall structural analysis was conducted, and it was determined to be a half-body shape. or the whole body Image, output structural labels ,Right now:
[0008] Step S1012: Current Clothing Classification: Classify and identify the model's current clothing, and output a numerical label for the clothing type. ,Right now:
[0009] Step S1013: Target Style Analysis: Analyze the target garment image. Perform style analysis and output its clothing type numerical label. And extract its key style feature vectors. ,Right now:
[0010] in, , It is a multidimensional vector that includes features such as sleeve shape and skirt hem; Step S1014: Semantic Tag Structuring: Integrate the above recognition results to generate a structured semantic tag set that can be used for subsequent policy judgment. ,Right now: 。
[0011] 3. The virtual fitting method based on large-model semantic recognition and dual-model cascading as described in claim 2, characterized in that, in step S1013, the multi-dimensional vector... , Represents sleeve type code, This represents the skirt hem code.
[0012] 4. The virtual try-on method based on large-model semantic recognition and dual-model cascading as described in claim 1, characterized in that the preset strategy rule in step S102 specifically includes the following sub-steps: set up The semantic tag set output in step S101; the strategy selection function. The definition is as follows: The merged strategy will be executed when any of the following conditions are met:
[0013]
[0014]
[0015] Execute when any of the following conditions are met Individual processing strategy:
[0016]
[0017] The following conditions must be met for a change of outfit to be considered unsupported:
[0018] in: Represents a collection of all dress types, for example ; Numerical labels representing long dresses, such as 4; A numerical label representing the top, such as 1; Feature vectors of the current clothing obtained from model images; Determine the feature vector and If the represented patterns are consistent, return the result. Otherwise return ; Based on the feature vector Returns the length attribute of the garment (e.g., long or short).
[0019] 5. The virtual try-on method based on large-model semantic recognition and dual-model cascading according to claim 1, characterized in that the image preprocessing in step S104 specifically includes the following sub-steps: Step S1041: Pose key point extraction and alignment: Extract and align the original model images respectively. and CAT-VTON output images By calling the Open Pose model, two sets of two-dimensional pixel coordinates of human pose key points are parsed and extracted from the Open Pose output, resulting in two coordinate sets. and ;exist and Matching is performed between them, an optimal similarity transformation matrix T is calculated, and it is applied to the entire... The image is geometrically transformed to make its human pose consistent with... To maintain consistency, obtain an aligned image. ; Step S1042: Precise masking of clothing area and background removal: The aligned image... An initial mask is generated by inputting the data into a pre-trained semantic segmentation model, where each pixel is labeled as belonging to the "clothing" or "non-clothing" category; based on this segmentation map, a binarized initial clothing mask is generated. And perform post-processing optimization; apply the optimized clothing masking. Aligned image By performing pixel-by-pixel multiplication, the pure clothing area is extracted, and the background is set to a uniform solid color to obtain the clothing image after removing the background. ; Step S1043: Image Normalization: Read the configuration file of the IDM-VTON model to obtain its required standard input size; use an interpolation algorithm to normalize the image. and Scale to the standard resolution and unify the color space; package to generate the final preprocessed data, including the standardized clothing images. and standardized clothing covers .
[0020] 6. The virtual try-on method based on large-model semantic recognition and dual-model cascading as described in claim 5, characterized in that the initial clothing mask is binarized. The clothing area is white (1), and the background area is black (0).
[0021] 7. The virtual try-on method based on large-model semantic recognition and dual-model cascading according to claim 1, characterized in that the post-processing enhancement in step S106 specifically includes the following sub-steps: Step S1061: Posture Consistency Verification and Geometric Correction: This involves verifying the original model image... and the final generated image Using the Open Pose model, extract two sets of pose keypoint coordinates. and Calculate offset ; Set an attitude offset threshold ,if If a significant attitude deviation is detected, the geometric correction process is triggered; the calculation from... arrive Similarity transformation matrix ,right Perform an inverse geometric transformation to bring its pose back to the same position as the original model image. After correction, fill or redraw the image edges to repair the blank areas caused by the transformation. S1062: Pixel-level segmentation of face and hand regions: This will segment the final image... The model is input into a high-precision facial and hand analysis model to generate a detailed facial mask; all sub-regions are merged to generate a complete facial and hand mask. It is used to protect the face and hands from damage in subsequent steps; S1063: Natural Reconstruction of Locally Distorted Areas: Combining pose key points and facial and hand analysis results, automatically detects areas that may contain distortions, such as incorrectly modified eyes and mouths, or stretching and deformation of the hands due to pose correction. Based on the located distortion areas, an inpainting restoration mask is generated. The areas requiring repair are marked as white (1), and other areas are marked as black (0). The Stable Diffusion Inpainting model is invoked: the above inputs and conditions are passed to the Stable Diffusion Inpainting model to generate and fuse the results; the generated repaired image is then compared with the original image. The images are then fused together to output high-quality virtual try-on images with stable postures and natural facial and hand features.
[0022] The present invention has the following beneficial effects: 1. It pioneered a cascading fusion mechanism of "CAT generating structure + IDM refining details", which fully leverages the complementary advantages of the two types of models and effectively solves the technical challenge of balancing structural integrity and texture refinement in long dress changing. 2. Construct an intelligent strategy routing system based on large-scale model semantic understanding to achieve automated decision-making and dynamic scheduling of the changing process, significantly improving the system's adaptability and robustness; 3. Introduce traditional computer vision modules to post-process and enhance the generated results. Without changing the main generation model, effectively suppress pose shift and facial and hand distortion, and ensure the realism and consistency of the output image. Attached Figure Description
[0023] Figure 1 This is the overall architecture diagram of the virtual try-on system of the present invention.
[0024] Figure 2 This is a schematic diagram of the semantic recognition and policy routing process based on a large model. Figure 3 This is a comparison chart of the result of the transformation based on the policy routing and the original model. Detailed Implementation
[0025] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0026] This invention discloses a virtual try-on method based on large-model semantic recognition and dual-model cascading, aiming to solve four key technical defects of existing virtual try-on models in practical applications: First, limited clothing types; existing models mostly only support single replacement of basic items such as tops and bottoms, unable to adapt to complex dressing scenarios such as dresses and suits, and difficult to meet users' diverse dressing simulation needs; Second, weak posture maintenance ability; during virtual try-on, human postures such as arm angles and waist curves are prone to misalignment with the clothing pattern, resulting in distorted try-on effects; Third, insufficient detail representation; unable to accurately reproduce key details of clothing, such as fabric texture, tailoring structure, and decorative elements, affecting users' judgment of the actual effect of the clothing; Fourth, severely affected by the original clothing pattern; when changing clothes based on images of original clothing already worn by the user, the outline of the original clothing will interfere with the virtual fit effect of the new clothing, causing the new clothing to fail to naturally fit the human body shape and reducing the accuracy of try-on.
[0027] This invention introduces a large model to perform semantic understanding on the input model and clothing images, automatically identifying the model's current clothing type and the target clothing type, and dynamically selecting the optimal dressing strategy based on the recognition results, achieving collaborative work between the IDM-VTON and CAT-VTON models. This invention designs and implements a complete optimization scheme of "semantic recognition - strategy routing - cascaded execution - post-processing enhancement": First, the large model is called to analyze the input image and extract semantic tags for the model and clothing; then, the execution path is intelligently decided according to preset strategy rules—when the model is a half-body image and the target is a dress, or when a full-body model changes from a short dress to a long dress, a "fusion" strategy is adopted, that is, CAT-VTON first generates the complete dress structure, and then after pose alignment and masking processing, it is input into IDM-VTON for detailed refinement; when the pattern is the same or only the top is changed, IDM-VTON is directly called to improve efficiency; for unsupported scenarios (such as full-body dressing changes that are not dresses), a prompt is returned. During the generation stage, Open Pose pose key point detection and Media Pipe face and hand segmentation technology are further combined to perform consistency verification and local repair on the output image, ensuring pose stability and natural face and hand features.
[0028] Example 1: like Figures 1 to 3 As shown, the virtual try-on system of this invention includes four core modules: a semantic recognition module, a policy routing module, a dual-model cascaded execution module, and a post-processing enhancement module. The system workflow is as follows: First, in the semantic recognition stage, a large model is invoked to perform joint analysis on the input model image and the target clothing image. For example... Figure 2As shown, the system identifies the model's current clothing status (such as "half-naked", "wearing a short dress") and the target clothing type (such as "long dress", "top" etc.), and structures the results into computable semantic tags.
[0029] The system then enters the strategy routing phase, where it determines the appropriate outfit change path based on preset rules. When the system detects a half-body model and the target is a dress, or when a full-body model changes from a short dress to a long dress, the system determines that the "fusion" strategy should be used. When only the top is changed or the style remains the same, IDM-VTON is invoked directly. If the target is a full-body outfit change that is not a dress, a "not supported" message is returned.
[0030] For the "fusion" strategy, the system first calls the CAT-VTON model to complete the initial dress transformation, generating an image of a dress with a complete structure. The output image is then preprocessed: Open Pose is used to extract and align pose keypoints, a semantic segmentation model is used to generate precise masks of the clothing region, and the image resolution and color space are unified to ensure they meet the input requirements of IDM-VTON. The preprocessed image is then used as conditional input by IDM-VTON for a second dress transformation, focusing on refining details such as texture, wrinkles, and edges to generate a high-quality transformation result.
[0031] Finally, in the post-processing enhancement stage, the system introduces a traditional computer vision module for quality optimization: OpenPose is used to compare the pose key points of the original model image with the final generated image, and geometric correction is performed if there is a slight offset; the Media Pipe Face Mesh model and Media Pipe Hand are used to perform pixel-level segmentation of the face and hand regions respectively, and the distorted regions are combined with the original face and hand features and contextual information, and Stable DiffusionInpainting is used for natural reconstruction to ensure that facial expressions and hands are realistic and natural.
[0032] This invention is achieved through the following steps: Step S101: Call the large model to perform semantic understanding on the input model image and target clothing image, and identify the current clothing type of the model (top, short dress, long dress, no clothing) and the target clothing type.
[0033] Step S1011: Image Structure Judgment: Analyze the Model Image It was determined to be half-body. or the whole body Output ,Right now:
[0034] Step S1012: Current Clothing Classification: Identify the model's current clothing and output a numerical label. ,Right now:
[0035] Step S1013: Target Style Analysis: Analyze the target garment Output type label and eigenvectors ,Right now:
[0036] Step S1014: Semantic Tag Structuring: Integrate the results to generate a set for policy judgment.
[0037] Step S102: Make a strategy decision based on the recognition results, let:
[0038] Strategy selection function Defined as: The merged strategy is executed when any of the following conditions are met:
[0039]
[0040]
[0041] The IDM-VTON standalone processing strategy is implemented when any of the following conditions are met:
[0042]
[0043] The following conditions are met to determine if changing outfits is not supported:
[0044] Step S103: For the "fusion version" scene, call the CAT-VTON model to generate a preliminary dress image.
[0045] Step S104: Preprocess the CAT-VTON output, including aligning pose key points, generating clothing area masks, and unifying the background, to adapt to IDM-VTON.
[0046] Step S1041: Attitude key point extraction and alignment: for and Use OpenPose to extract key points and Calculate the similarity transformation matrix and to Perform a transformation to align the orientation, and obtain .
[0047] Step S1042: Clothing area mask generation and background removal: The input semantic segmentation model generates the initial mask. After morphological optimization, etc., By extracting pure clothing areas through pixel-by-pixel multiplication and unifying the background, we obtain... .
[0048] Step S1043: Image Normalization: and Scaled to IDM-VTON standard resolution, unified color space, and packaged as follows:
[0049] Step S105: Input the preprocessed image as a conditional input to IDM-VTON for a second processing step to refine textures, edges, and details.
[0050] Step S106: Post-processing enhancement of the final image: Use Open Pose to verify and correct pose consistency; combine Media Pipe Face Mesh and Media Pipe Hands to segment the face and hands; use image inpainting techniques to correct distorted areas to ensure pose stability and natural face and hands.
[0051] Step S1061: Attitude Consistency Verification and Geometric Correction: For and Call OpenPose to calculate attitude offset. .like Exceeding the threshold Then calculate the correction matrix. right Perform geometric correction and fix edge gaps.
[0052] Step S1062: Pixel-level segmentation of the face and hand regions: Input facial and hand analysis models, merge all sub-regions, and generate complete facial and hand protective masks. .
[0053] Step S1063: Local Distortion Repair: Automatically or manually locate distorted areas on the face and hands, and generate a repair mask. Calling the Stable Diffusion Inpainting model, based on... , and Perform local reconstruction, fuse the results, and output the final high-quality image.
[0054] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.
Claims
1. A virtual try-on method based on large-model semantic recognition and dual-model cascading, characterized in that, Includes the following steps: Step S101: Call the large model to perform semantic understanding on the input model image and target clothing image, and identify the current clothing type of the model and the target clothing type. The clothing type includes top, short dress, long dress or no clothing. Step S102: Based on the recognition results, make a dress change path decision based on preset strategy rules: if the model is half-body and the target garment is a dress, or the model is full-body and changes from a short dress to a long dress, or the original garment and the target garment have different styles, then select the "fusion version" dress change strategy; if only the top is changed or the styles are the same, then select the IDM-VTON separate execution strategy; if the target garment is a full-body dress change that is not a dress, then it is determined that the scene is not supported and a prompt is returned. Step S103: For the scenario of the "fusion version" strategy, first call the CAT-VTON model to complete the initial dressing change and generate a dress image with a complete structure; Step S104: Perform image preprocessing on the CAT-VTON output image, including pose key point alignment, clothing area mask generation and background unification, to adapt to IDM-VTON input requirements; Step S105: Using the preprocessed image as input, the IDM-VTON model is called for a second makeover process to refine the texture, edges and details, and generate a high-quality makeover result. Step S106: The final generated image is post-processed and enhanced using a traditional computer vision module. Open Pose is used to detect the consistency of pose key points before and after generation. The face and hand regions are segmented using the Media Pipe model. Image inpainting techniques are used to correct distorted local areas to ensure stable pose and natural face and hand features.
2. The virtual try-on method based on large-model semantic recognition and dual-model cascading as described in claim 1, characterized in that, The semantic understanding described in step S101 specifically includes the following sub-steps: Step S1011: Image structure judgment: For the model image An overall structural analysis was conducted, and it was determined to be a half-body shape. or the whole body Image, output structural labels ,Right now: Step S1012: Current Clothing Classification: Classify and identify the model's current clothing, and output a numerical label for the clothing type. ,Right now: Step S1013: Target Style Analysis: Analyze the target garment image. Perform style analysis and output its clothing type numerical label. And extract its key style feature vectors. ,Right now: in, , It is a multidimensional vector that includes features such as sleeve shape and skirt hem; Step S1014: Semantic Tag Structuring: Integrate the above recognition results to generate a structured semantic tag set that can be used for subsequent policy judgment. ,Right now: 。 3. The virtual try-on method based on large-model semantic recognition and dual-model cascading as described in claim 2, characterized in that, In step S1013, the multidimensional vector , Represents sleeve type code, This represents the skirt hem code.
4. The virtual try-on method based on large-model semantic recognition and dual-model cascading as described in claim 1, characterized in that, The preset strategy rule mentioned in step S102 specifically includes the following sub-steps: set up The semantic tag set output in step S101; the strategy selection function. The definition is as follows: The merged strategy will be executed when any of the following conditions are met: Execute when any of the following conditions are met Individual processing strategy: The following conditions must be met for a change of outfit to be considered unsupported: in: Represents a collection of all dress types; Numerical tags representing long dresses; The number tag represents the top; Feature vectors of the current clothing obtained from model images; Determine the feature vector and If the represented patterns are consistent, return the result. Otherwise return ; Based on the feature vector Returns the length attribute of the garment.
5. The virtual try-on method based on large-model semantic recognition and dual-model cascading as described in claim 1, characterized in that, The image preprocessing described in step S104 specifically includes the following sub-steps: Step S1041: Pose key point extraction and alignment: Extract and align the original model images respectively. and CAT-VTON output images By calling the Open Pose model, two sets of two-dimensional pixel coordinates of human pose key points are parsed and extracted from the Open Pose output, resulting in two coordinate sets. and ;exist and Matching is performed between them, an optimal similarity transformation matrix T is calculated, and it is applied to the entire... The image is geometrically transformed to make its human pose consistent with... To maintain consistency, obtain an aligned image. ; Step S1042: Precise masking of clothing area and background removal: The aligned image... The input is fed into a pre-trained semantic segmentation model to generate an initial mask, where each pixel is labeled as belonging to the "clothing" or "non-clothing" category; based on this segmentation map, a binarized initial clothing mask is generated. And perform post-processing optimization; apply the optimized clothing masking. Aligned image By performing pixel-by-pixel multiplication, the pure clothing area is extracted, and the background is set to a uniform solid color to obtain the clothing image after removing the background. ; Step S1043: Image Normalization: Read the configuration file of the IDM-VTON model to obtain its required standard input size; use an interpolation algorithm to normalize the image. and Scale to the standard resolution and unify the color space; package to generate the final preprocessed data, including the standardized clothing images. and standardized clothing covers .
6. The virtual try-on method based on large-model semantic recognition and dual-model cascading as described in claim 5, characterized in that, Binarized initial clothing mask The clothing area is white (1), and the background area is black (0).
7. The virtual try-on method based on large-model semantic recognition and dual-model cascading as described in claim 1, characterized in that, The post-processing enhancement described in step S106 specifically includes the following sub-steps: Step S1061: Posture Consistency Verification and Geometric Correction: This involves verifying the original model image... and the final generated image Using the Open Pose model, extract two sets of pose keypoint coordinates. and Calculate offset ; Set an attitude offset threshold ,if If the attitude is significantly off, the geometric correction process is triggered. Calculation from arrive Similarity transformation matrix ,right Perform an inverse geometric transformation to bring its pose back to the same position as the original model image. After correction, fill or redraw the image edges to repair the blank areas caused by the transformation. S1062: Pixel-level segmentation of face and hand regions: This will segment the final image... The model is input into a high-precision facial and hand analysis model to generate a detailed facial mask; all sub-regions are merged to generate a complete facial and hand mask. It is used to protect the face and hands from damage in subsequent steps; S1063: Natural Reconstruction of Locally Distorted Areas: Combining pose key points and facial and hand analysis results, automatically detects areas that may contain distortions, such as incorrectly modified eyes and mouths, or stretching and deformation of the hands due to pose correction. Based on the located distortion areas, an inpainting restoration mask is generated. The areas requiring repair are marked as white (1), and other areas are marked as black (0). The Stable Diffusion Inpainting model is invoked: the above inputs and conditions are passed to the Stable Diffusion Inpainting model to generate and fuse the results; the generated repaired image is then compared with the original image. The images are then fused together to output high-quality virtual try-on images with stable postures and natural facial and hand features.