Multi-architecture virtual fitting model-oriented pluggable clothing pattern control model construction method

Abstract

The invention belongs to the technical field of computer vision related to virtual fitting, and particularly relates to a pluggable garment pattern control model construction method for a multi-architecture virtual fitting model, and the method comprises the steps: eliminating the contour leakage of original garments through a sample construction module, and generating a standardized mask and a dense posture for the constructed control model; utilizing a model perception layout generator to generate a body-clothing segmentation layout map according to the target model label, and extracting multi-scale model features; the features are matched and injected into the target virtual try-on model through a multi-scale model injector, and a try-on image conforming to the target model is driven to be generated. The garment pattern control model constructed by the method serves as a universal plug-in, accurate pattern control can be realized without changing the structure of the existing virtual fitting model, the problems of pattern fixing, control capability and model architecture binding and the like in the prior art are solved, and the customization level and user experience of virtual fitting are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology related to virtual try-on, and more specifically, relates to a method for constructing a pluggable clothing pattern control model for multi-architecture virtual try-on models. Background Technology

[0002] Virtual try-on technology aims to realistically composite images of specified clothing onto images of a target person, providing users with an online try-on experience. Existing virtual try-on methods primarily focus on the fidelity of clothing textures and alignment with the human body, but generally overlook a crucial factor that determines the overall style of the outfit—the garment's cut. The cut defines the spatial fit between the clothing and the human body, directly affecting the appearance of the garment.

[0003] Current mainstream methods, whether traditional methods based on distortion-synthesis or advanced methods based on diffusion models, can only generate fixed-pattern try-on effects and cannot customize different patterns such as loose, fitted, or slim-fit according to user preferences. Some customized try-on methods attempt to control the local wearing style through clothing key points or text descriptions, but their control capabilities are limited, and they are usually strongly coupled with specific model architectures, making them difficult to transfer to other virtual try-on models.

[0004] Therefore, existing technologies lack a universal solution that can be flexibly integrated into different virtual try-on models and provide accurate and controllable pattern generation. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a pluggable clothing pattern control model construction method for multi-architecture virtual try-on models. Its purpose is to propose a method that can be seamlessly integrated into a variety of mainstream virtual try-on models as an independent plug-in, so as to realize the user's customized control of clothing patterns.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for constructing a pluggable garment pattern control model for a multi-architecture virtual try-on model is provided, comprising: Construct a training sample set, where each training sample includes a mask image of a person. Simulation diagram of dense poses of people in portrait images and target clothing images The sample construction module is used to determine a binary image that can completely occlude the clothing worn by the person in the image based on the set of two-dimensional key points of the half-body corresponding to the target clothing in the image. The binary image is then multiplied by the image to obtain the mask image of the person. Based on the preset human body proportions and the key points of the person in the image, the human torso and limbs are simulated to construct a dense pose simulation image. The U-Net structure, built from ResNet blocks, is modified as follows: An affine transformation layer is added after the normalization layer in each ResNet block of the decoding module within the U-Net. Two linear layers to be learned are constructed, and their outputs are connected to each affine transformation layer. The output of each affine transformation layer is then connected to the next layer connected to the normalization layer in the corresponding ResNet block. This modified U-Net structure serves as the layout generator to be trained. Using the training sample set, a layout generator is generated based on the actual body-clothing spatial layout corresponding to each training sample. The calculated loss of the generated body-clothing spatial layout is related to the layout generator. Training; in this process, two linear layers are used to receive and modulate the one-hot vectors corresponding to the target pattern labels, respectively, resulting in two modulation parameters, which are then input into each affine transformation layer. This affine transformation layer is used to calculate the intermediate features generated by the ResNet block. Mean values ​​among the features of each channel with standard deviation Perform an affine transformation according to the following formula and output the result: ; Input each training sample into the trained layout generator. Extract layout generator The output of each ResNet block constitutes the multi-scale pattern-aware features corresponding to that training sample. Construct a multi-scale pattern injector, including semantic alignment units and resolution alignment units. The semantic alignment unit is used to implement... Alignment with the intermediate feature semantic hierarchy of the target virtual fitting model; resolution alignment units are used to achieve... Alignment with the intermediate feature resolution of the target virtual fitting model; freeze layout generator The parameters of the target virtual try-on model are used to calculate the loss based on the real virtual try-on effect image corresponding to each training sample and the virtual try-on effect image output by the target virtual try-on model, and the multi-scale pattern injector is trained. Layout generator after training Together with a multi-scale pattern injector and a sample construction module, they constitute a pluggable clothing pattern control model for the target virtual try-on model.

[0007] Furthermore, the sample construction module constructs binary images in the following way: Based on the set of two-dimensional key points of the upper half of the body corresponding to the target clothing in the portrait image. Calculate the horizontal span and vertical span Among them, if the target clothing in the target clothing image is a top, then the two-dimensional keypoint set... This is the set of two-dimensional key points for the shoulder, elbow, and wrist joints of a person in an image; if the target clothing in the image is a bottom garment, then the set of two-dimensional key points is... This is a set of two-dimensional key points for the hip, knee, and ankle joints of a person in an image. Construct a rectangular region. Widen its horizontal span by the same first preset amount at both ends, defining its horizontal span. Extend its vertical span by the same second preset amount at both ends, defining its vertical span. The center point of this rectangular region... and width and height Determined by the following formula: ; ; In the formula, and The horizontal widening ratio and the vertical lengthening ratio; Construct a binary image of the same size as the person's image, with pixel values ​​set to 1 within a rectangular region and 0 elsewhere.

[0008] Furthermore, the sample construction module constructs the dense pose simulation graph in the following way: For the torso, quadrilaterals are constructed with the shoulder and hip joints as vertices, and the shoulder and hip joints are connected by line segments. Arcs are used to connect the two shoulder joints and the two hip joints. For the arms, circles are constructed with the shoulder, elbow, and wrist joints of the person in the image as centers, each with a specific diameter preset based on human proportions. Tangents connecting these three circles near the torso and those away from the torso form two cylinders to simulate the upper arm and forearm. For the legs, circles are constructed with the hip, knee, and ankle joints of the person in the image as centers, each with a specific diameter preset based on human proportions. Tangents connecting these three circles near the human midline and those away from the human midline form two cylinders to simulate the thigh and calf. This yields a composite dense pose based on human proportions and key points. The composite dense pose is then color-filled, with the colors of each body part matching the corresponding parts predicted by the preset model. The intersection of the color-filled composite dense pose and the model-predicted dense pose is used as the simulation image of the person's dense pose in the image.

[0009] Furthermore, when constructing circles centered on the shoulder, elbow, and wrist joints of the person in the image, the diameters correspond to 0.06, 0.048, and 0.033 times the person's height, respectively; when constructing circles centered on the hip, knee, and ankle joints of the person in the image, the diameters correspond to 0.09, 0.055, and 0.03 times the person's height, respectively. The height of a person is defined as 3.2 times the torso height (the vertical height between the shoulder and hip in the image) or 2.3 times the leg length. The vertical height between the shoulder and hip in the image is taken as the torso height. If the person in the image is standing, the vertical height between the hip and ankle is taken as the leg length. If the person in the image is sitting, twice the vertical height between the knee and ankle is taken as the leg length.

[0010] Furthermore, the U-Net adopts a pre-trained U-Net from Stable Diffusion, and the structure of the U-Net is modified, including changing the input layer to 13 channels and the output layer to 3 channels; And each training sample is input to the layout generator. Previously, the VAE encoder in the Stable Diffusion was used. The mask image in the training sample Simulated dense posture diagram Target clothing image and noise graph Mapping these to the latent space yields four-channel tensors, which are then used to generate the layout generator. Input In the formula, For all-zero tensors, To splice along the channel dimension, To splice together in the spatial dimension.

[0011] Furthermore, the resolution alignment unit aligns the intermediate feature resolutions of the multi-scale pattern-aware features and the target virtual fitting model as follows: Determine the architecture of the target virtual fitting model. If the architecture is a U-Net-like architecture, perform tensor space segmentation on the multi-scale pattern-aware features after convolution to align the multi-scale pattern-aware features with the intermediate feature resolution of the target virtual fitting model. If the architecture is a DiT-like architecture, upsample the features of each resolution in the multi-scale pattern-aware features after convolution to the resolution of the corresponding layer features in the DiT class through interpolation and flatten them into sequence tokens to align the multi-scale pattern-aware features with the intermediate feature resolution of the target virtual fitting model.

[0012] Furthermore, the semantic alignment unit is a convolutional module used for... Perform convolution operations on each scale feature; When training the multi-scale pattern injector begins, each convolutional layer in the convolutional module is a zero-initialized convolutional layer.

[0013] According to another aspect of the present invention, a virtual try-on method is provided, comprising: Based on the received person image, target clothing image, and target pattern label, a pluggable clothing pattern control model is used to obtain multi-scale pattern perception features that are semantically and resolutionally aligned with the intermediate features of the target virtual try-on model. The original inputs required for the target virtual try-on model and the multi-scale pattern perception features are input into the target virtual try-on model to obtain the virtual try-on effect image, thus completing the virtual try-on.

[0014] According to another aspect of the present invention, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0015] According to another aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is run by a processor, it controls the device where the storage medium is located to perform the steps of the method described above.

[0016] In summary, compared with the prior art, the technical solutions conceived by this invention have the following main advantages: 1. This invention provides a pluggable clothing pattern control model construction method for multi-architecture virtual try-on models. This method, based on a sample construction module, processes the input human image, generating a rectangular mask and simulating dense poses based on human key points and standard body proportions to eliminate leakage of the original clothing outline. The method also constructs a pattern-aware layout generator based on the U-Net architecture. This generator injects the target pattern label into the residual block of the U-Net decoding module using a feature linear modulation method to conditionally generate the body-clothing spatial layout. In other words, it dynamically adjusts the feature distribution according to the target pattern label to generate the body-clothing layout corresponding to the target pattern. Spatial layout; additionally, a multi-scale pattern injector is constructed. At this point, both the pattern-aware layout generator and the target virtual try-on model remain frozen. Based on the target virtual try-on model to be adapted, the multi-scale pattern-aware features are customized and adjusted, i.e., transformed and aligned, to ensure compatibility with the feature dimensions and resolution of the target virtual try-on model. When facing different virtual try-on models, the sample construction module, the pattern-aware layout generator, and the target virtual try-on model can be used as pre-trained, frozen modules. For adapting a new target virtual try-on model, only a specific multi-scale pattern injector needs to be trained, demonstrating the pluggability and efficiency of the control model. Therefore, the pattern control model constructed by the method of this invention, as an independent plug-in, can be flexibly and seamlessly integrated into various mainstream diffusion model virtual try-on architectures without modifying the original model structure, breaking through the limitation of customized control being bound to a specific model. Furthermore, this invention explicitly models the body-clothing spatial relationship under different patterns through the pattern-aware layout generator and uses a multi-scale feature injection mechanism to precisely guide image generation, achieving high-precision control of clothing fit. Furthermore, through the sample construction module, standardized representations based on human body key points are generated, effectively avoiding the bias of the original clothing outline on the generated results, making pattern control purer and more effective. When integrated into the new virtual try-on model, only a lightweight pattern injector needs to be trained, with fast convergence speed, and good control effect can be achieved with only a small number of samples while significantly reducing the number of training parameters. Attached Figure Description

[0017] Figure 1 A flowchart illustrating a method for constructing a pluggable garment pattern control model for a multi-architecture virtual try-on model, provided in an embodiment of the present invention; Figure 2 This is an overall framework diagram of the virtual try-on method based on a pluggable garment pattern control model provided in an embodiment of the present invention. Figure 3 A schematic diagram illustrating the method for generating mask images and dense pose simulation images using the sample construction module provided in this embodiment of the invention; Figure 4A framework diagram of a layout-aware generator provided in an embodiment of the present invention; Figure 5 A schematic diagram of a human figure provided for an embodiment of the present invention; Figure 6 The image shown is the fitting effect diagram output by the target virtual fitting model after inserting the pattern control model provided in the embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0019] Example 1 A method for constructing a pluggable garment pattern control model for multi-architecture virtual try-on models, such as... Figure 1 As shown, it includes: Construct a training sample set, where each training sample includes a mask image of a person. Simulation diagram of dense poses of people in portrait images and target clothing images The sample construction module is used to determine a binary image that can completely occlude the clothing worn by the person in the image based on the set of two-dimensional key points of the half-body corresponding to the target clothing in the image. The binary image is then multiplied by the image to obtain the mask image of the person. Based on the preset human body proportions and the key points of the person in the image, the human torso and limbs are simulated to construct a dense pose simulation image. The U-Net structure, built from ResNet blocks, is modified as follows: An affine transformation layer is added after the normalization layer in each ResNet block of the decoding module within the U-Net. Two linear layers to be learned are constructed, and their outputs are connected to each affine transformation layer. The output of each affine transformation layer is then connected to the next layer connected to the normalization layer in the corresponding ResNet block. This modified U-Net structure serves as the layout generator to be trained. Using the training sample set, a layout generator is generated based on the actual body-clothing spatial layout corresponding to each training sample. The calculated loss of the generated body-clothing spatial layout is related to the layout generator. Training; in this process, two linear layers are used to receive and modulate the one-hot vectors corresponding to the target pattern labels, respectively, resulting in two modulation parameters, which are then input into each affine transformation layer. This affine transformation layer is used to calculate the intermediate features generated by the ResNet block. Mean values ​​among the features of each channel with standard deviation Perform an affine transformation according to the following formula and output the result: ; Input each training sample into the trained layout generator. Extract layout generator The output of each ResNet block constitutes the multi-scale pattern-aware features corresponding to that training sample. Construct a multi-scale pattern injector, including semantic alignment units and resolution alignment units. The semantic alignment unit is used to implement... Alignment with the intermediate feature semantic hierarchy of the target virtual fitting model; resolution alignment units are used to achieve... Alignment with the intermediate feature resolution of the target virtual fitting model; freeze layout generator The parameters of the target virtual try-on model are used to calculate the loss based on the real virtual try-on effect image corresponding to each training sample and the virtual try-on effect image output by the target virtual try-on model, and the multi-scale pattern injector is trained. Layout generator after training Together with a multi-scale pattern injector and a sample construction module, they constitute a pluggable clothing pattern control model for the target virtual try-on model.

[0020] like Figure 2 As shown, the method in this embodiment involves the following points: Sample construction module: Processes the input human image, generates a rectangular mask and simulates dense poses based on human key points and standard body proportions to eliminate the leakage of the original clothing outline.

[0021] Pattern-Aware Layout Generation: The mask, dense pose, target garment image, and target pattern label are input into the pattern-aware layout generator to generate a body-garment segmentation layout map corresponding to the target pattern. In the application phase after generator training, multi-scale pattern-aware features are extracted as input to the target virtual try-on model. The pattern-aware layout generator is based on the U-Net architecture and injects pattern labels into the residual blocks of the U-Net decoding module using a feature linear modulation method for conditional layout generation. The core of this step is generating a body-garment spatial layout map corresponding to the target pattern (e.g., "slim," "regular," "loose"). This operation allows the generator to dynamically adjust the feature distribution based on the pattern label, thereby generating the body-garment spatial layout corresponding to the target pattern, i.e., a three-class segmentation map. (Background, human body, clothing) (Training phase, used to calculate loss).

[0022] Multi-scale pattern feature injection: The obtained multi-scale pattern-aware features are transformed and aligned using a lightweight multi-scale pattern injector to ensure compatibility with the feature dimensions and resolution of the target virtual try-on model. During training, both the pattern-aware layout generator and the target virtual try-on model remain frozen, and only the parameters of the pattern injector are trained.

[0023] In the subsequent application stage of the control model, layout-driven virtual try-on involves injecting the pattern features processed by the multi-scale pattern injector into the target virtual try-on model through the control network interface, guiding it to generate try-on images that conform to the target pattern layout.

[0024] This embodiment presents a pluggable garment pattern control method for multi-architecture virtual try-on models. When deploying a system using this method, the sample construction module, the pattern-aware layout generator, and the target virtual try-on model can be pre-trained and frozen modules for different virtual try-on models. When a new target virtual try-on model needs to be adapted, only a specific multi-scale pattern injector needs to be trained for it, demonstrating the system's pluggability and efficiency.

[0025] In summary, the method in this embodiment processes the input human image based on a sample construction module. It generates a rectangular mask and simulates dense poses based on human key points and standard body proportions to eliminate leakage of the original clothing outline. The construction method also builds a pattern-aware layout generator based on the U-Net architecture. This generator injects the target pattern label into the residual block of the U-Net decoding module using a feature linear modulation method to conditionally generate the body-clothing spatial layout. That is, it dynamically adjusts the feature distribution according to the target pattern label to generate the body-clothing spatial layout corresponding to the target pattern. Additionally, it constructs multi-scale patterns. In this process, both the pattern-aware layout generator and the target virtual try-on model remain frozen. Based on the target virtual try-on model to be adapted, the multi-scale pattern-aware features are customized and adjusted, i.e., transformed and aligned, to ensure compatibility with the feature dimensions and resolution of the target virtual try-on model. When facing different virtual try-on models, the sample construction module, the pattern-aware layout generator, and the target virtual try-on model can be used as pre-trained, frozen modules. For adapting a new target virtual try-on model, only a specific multi-scale pattern injector needs to be trained, demonstrating the pluggability and efficiency of the control model. Therefore, the pattern control model constructed by the method of this invention, as an independent plug-in, can be flexibly and seamlessly integrated into various mainstream diffusion model virtual try-on architectures without modifying the original model structure, breaking through the limitation of customized control being bound to a specific model. In addition, this invention explicitly models the body-clothing spatial relationship under different patterns through the pattern-aware layout generator and uses a multi-scale feature injection mechanism to accurately guide image generation, achieving high-precision control of clothing fit. Furthermore, through the sample construction module, standardized representations based on human body key points are generated, effectively avoiding the bias of the original clothing outline on the generated results, making pattern control purer and more effective. When integrated into the new virtual try-on model, only a lightweight pattern injector needs to be trained, with fast convergence speed, and good control effect can be achieved with only a small number of samples while significantly reducing the number of training parameters.

[0026] The following are preferred embodiments of some technical features in the method of this embodiment. They are not required to implement an independent plug-in that can be seamlessly integrated into multiple mainstream virtual try-on models and can customize and control clothing patterns. Any method within the spirit and principles of the method of this embodiment can achieve the purpose of this invention, as detailed below.

[0027] This can be used as a preferred implementation method, such as Figure 3 As shown, input a human image. Generate a mask image that does not contain the shape information of the clothing originally worn by the person in the image. and dense pose diagram .

[0028] Regarding mask image generation, to avoid leakage of the original garment outline (if leakage occurs, the model during training will tend to determine the edge position of the target garment along the edge of the mask, and cannot flexibly adjust the edge position of the target garment according to the target pattern), a garment segmentation model is not used; instead, a model based on human body key points is employed. Construct a rectangular mask image.

[0029] The sample construction module constructs binary images in the following way: Based on the set of two-dimensional key points of the upper half of the body corresponding to the target clothing in the portrait image. Calculate the horizontal span and vertical span Among them, if the target clothing in the target clothing image is a top, then the two-dimensional keypoint set... This is the set of two-dimensional key points for the shoulder, elbow, and wrist joints of a person in an image; if the target clothing in the image is a bottom garment, then the set of two-dimensional key points is... This is a set of two-dimensional key points for the hip, knee, and ankle joints of a person in an image. Construct a rectangular region. Widen its horizontal span by the same first preset amount at both ends, defining its horizontal span. Extend its vertical span by the same second preset amount at both ends, defining its vertical span. The center point of this rectangular region... and width and height Determined by the following formula: ; ; In the formula, and The horizontal widening ratio and the vertical lengthening ratio; Construct a binary image of the same size as the person's image, with pixel values ​​set to 1 within a rectangular region and 0 elsewhere.

[0030] In this embodiment, for the top-loading component, an empirical filling ratio is used. and For bottoms, an empirical fill ratio and .

[0031] As a preferred implementation, the dense pose prediction results can be combined with the DensePose model (inputting a person image). and synthetic dense pose based on human proportions and key points The resulting dense attitude simulation diagram is shown as follows: .

[0032] The sample construction module constructs dense pose simulation graphs in the following way: For the torso, quadrilaterals are constructed with the shoulder and hip joints as vertices. The shoulder and hip joints are connected by line segments, and arcs are used to connect the two shoulder joints and the two hip joints to retain a natural shape. For the arms, circles are constructed with the shoulder, elbow, and wrist joints of the figure in the image as centers, each with a specific diameter preset based on human proportions. Tangents connecting these three circles near the torso and tangents away from the torso form two cylinders to simulate the upper arm and forearm. For the legs, circles are constructed with the hip, knee, and ankle joints of the figure in the image as centers. A circle is constructed with a specific diameter based on human body proportions. Tangents connecting three circles near the human body's central axis and connecting three circles away from the human body's central axis form two cylinders to simulate the thigh and calf, thus obtaining a synthetic dense pose based on human body proportions and key points. The synthetic dense pose is then filled with color, and the color of each body part in the synthetic dense pose is consistent with the corresponding part in the dense pose predicted based on the preset model. The intersection of the color-filled synthetic dense pose and the dense pose predicted based on the model is taken as the dense pose simulation map of the person in the human image.

[0033] As a preferred implementation, when constructing circles centered on the shoulder, elbow, and wrist joints of the person in the image, the diameters are respectively 0.06, 0.048, and 0.033 times the person's height; when constructing circles centered on the hip, knee, and ankle joints of the person in the image, the diameters are respectively 0.09, 0.055, and 0.03 times the person's height. The height of a person is defined as 3.2 times the torso height (the vertical height between the shoulder and hip in the image) or 2.3 times the leg length. The vertical height between the shoulder and hip in the image is taken as the torso height. If the person in the image is standing, the vertical height between the hip and ankle is taken as the leg length. If the person in the image is sitting, twice the vertical height between the knee and ankle is taken as the leg length.

[0034] This can be used as a preferred implementation method, such as Figure 4 As shown, U-Net uses the pre-trained U-Net from Stable Diffusion and modifies its structure, including changing the input layer from 9 channels to 13 channels and the output layer from 4 channels to 3 channels. And each training sample is input to the layout generator. Previously, the VAE encoder in the Stable Diffusion was used. The mask image in the training sample Simulated dense posture diagram Target clothing image and noise graph Mapping these to the latent space yields four-channel tensors, which are then used to generate the layout generator. Input In the formula, For all-zero tensors, To splice along the channel dimension, For splicing in the spatial dimension. In other words, a layout generator. The input is a 13-channel tensor. (in ).

[0035] As a preferred implementation, the resolution alignment unit aligns the intermediate feature resolutions of the multi-scale pattern-perceiving features and the target virtual fitting model in the following way: Determine the architecture of the target virtual fitting model. If the architecture is a U-Net-like architecture, perform tensor space segmentation on the multi-scale pattern-aware features after convolution to align the multi-scale pattern-aware features with the intermediate feature resolution of the target virtual fitting model. If the architecture is a DiT-like architecture, upsample the features of each resolution in the multi-scale pattern-aware features after convolution to the resolution of the corresponding layer features in the DiT class through interpolation and flatten them into sequence tokens to align the multi-scale pattern-aware features with the intermediate feature resolution of the target virtual fitting model.

[0036] As a preferred embodiment, the semantic alignment unit is a convolutional module used for... Perform convolution operations on each scale feature; When training the multi-scale pattern injector begins, each convolutional layer in the convolutional module is a zero-initialized convolutional layer.

[0037] In this implementation, the multi-scale pattern injector utilizes zero-initialization convolutional layers to align features and matches the feature resolution required by different virtual try-on models through operations such as segmentation, interpolation, or flattening. This step is achieved through a lightweight multi-scale pattern injector. Features Adapt to the target virtual fitting model.

[0038] Specifically, the multi-scale pattern-aware features from the previous stage First, semantic alignment is performed using a zero-initialization convolutional layer, that is, for Each scale feature Perform convolution operations:

[0039] Zero initialization ensures that the injector has zero impact on the virtual try-on model at the start of training.

[0040] After convolution, it is still necessary to adjust the model according to the architecture of the target virtual fitting model. Perform resolution matching. If the target virtual fitting model has a U-Net-like architecture, then the layout generator... Both the target virtual fitting model and the virtual fitting model adopt the U-Net-like architecture, and their feature spaces are similar. Therefore, it is only necessary to integrate the multi-scale pattern-aware features. Multi-scale pattern-aware features can be achieved by performing a simple tensor space partitioning operation. Alignment with the intermediate feature resolution of the target virtual fitting model. If the target virtual fitting model has a DiT-type architecture, multi-scale pattern-aware features need to be incorporated. By upsampling the features at each resolution to the corresponding layer resolution in the DiT class through interpolation and flattening them into sequence tokens, multi-scale pattern-aware features can be achieved. Alignment with the intermediate feature resolution of the target virtual fitting model.

[0041] In training template injector At that time, layout generator The parameters of both the fitting model and the device model are frozen. Only the injector parameters are optimized. The loss function can be the mean squared error loss commonly used in target virtual try-on models:

[0042] in It is a time step The latent noise variables, Other inputs to the virtual fitting model It is a noise predictor.

[0043] Figure 5 Examples of images of people wearing different styles of clothing are provided in the training samples. These include short-sleeved shirts and trousers. The short-sleeved shirts are divided into three styles: fitted, straight, and loose. The main difference between the three styles is the fit. The trousers are divided into two styles: tapered and straight. The main difference between the two styles is whether the trouser leg width gradually narrows from the knee to the ankle.

[0044] Example 2 A virtual try-on method, comprising: Based on the received person image, target clothing image, and target pattern label, a pluggable clothing pattern control model is used to obtain multi-scale pattern perception features that are semantically and resolutionally aligned with the intermediate features of the target virtual try-on model. The original inputs required for the target virtual try-on model and the multi-scale pattern perception features are input into the target virtual try-on model to obtain the virtual try-on effect image, thus completing the virtual try-on.

[0045] This method, given a person image, a clothing image, and a target pattern label, can generate high-fidelity try-on images that match the target pattern. .

[0046] In practice, pattern features can be input into the virtual fitting model via its ControlNet interface without altering the model's network structure and parameters. In other words, the multi-scale pattern injector... Processing multi-scale pattern-aware features The obtained features are input into the target virtual fitting model through the ControlNet interface of the target virtual fitting model to generate a fitting effect image corresponding to the target pattern.

[0047] Figure 6 This is a virtual try-on model generated based on a pattern control plugin. For short-sleeved shirts, after inserting the pattern control model, the target virtual try-on model can generate try-on effects for three fit types: slim, fitted, and loose. For trousers, after inserting the pattern control model, the target virtual try-on model can generate try-on effects for two fit types: straight and tapered.

[0048] The relevant technical solutions are the same as above, and will not be repeated here.

[0049] Example 3 This application also relates to an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0050] The electronic device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory can be used to store computer programs and / or modules. The processor performs various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory.

[0051] The relevant technical solutions are the same as above, and will not be repeated here.

[0052] Example 4 This application also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0053] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0054] The relevant technical solutions are the same as above, and will not be repeated here.

[0055] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing a pluggable garment pattern control model for multi-architecture virtual try-on models, characterized in that, include: Construct a training sample set, where each training sample includes a mask image of a person image, a dense pose simulation image of the person in the person image, and a target clothing image; using a sample construction module, based on the two-dimensional key point set of the half-body corresponding to the target clothing of the person in the person image, determine a binary image that can completely occlude the clothing worn by the person in the person image, and multiply the binary image with the person image to obtain the mask image of the person image. Based on preset human body proportions and key points of the person in the image, simulate the human torso and limbs to construct a dense posture simulation map. The U-Net structure, built from ResNet blocks, is modified as follows: An affine transformation layer is added after the normalization layer in each ResNet block of the decoding module within the U-Net. Two linear layers to be learned are constructed, and their outputs are connected to each affine transformation layer. The output of each affine transformation layer is then connected to the next layer connected to the normalization layer in the corresponding ResNet block. This modified U-Net structure serves as the layout generator to be trained. ; Using the training sample set, a layout generator is generated based on the actual body-clothing spatial layout corresponding to each training sample. The calculated loss of the generated body-clothing spatial layout is related to the layout generator. Training; in this process, two linear layers are used to receive and modulate the one-hot vectors corresponding to the target pattern labels, respectively, resulting in two modulation parameters, which are then input into each affine transformation layer. This affine transformation layer is used to calculate the intermediate features generated by the ResNet block. Mean values ​​among the features of each channel with standard deviation Perform an affine transformation according to the following formula and output the result: ; Input each training sample into the trained layout generator. Extract layout generator The output of each ResNet block constitutes the multi-scale pattern-aware features corresponding to that training sample. Construct a multi-scale pattern injector, including semantic alignment units and resolution alignment units. The semantic alignment unit is used to implement... Alignment with the intermediate feature semantic hierarchy of the target virtual fitting model; resolution alignment units are used to achieve... Alignment with the intermediate feature resolution of the target virtual fitting model; freeze layout generator The parameters of the target virtual try-on model are used to calculate the loss based on the real virtual try-on effect image corresponding to each training sample and the virtual try-on effect image output by the target virtual try-on model, and the multi-scale pattern injector is trained. Layout generator after training Together with a multi-scale pattern injector and a sample construction module, they constitute a pluggable clothing pattern control model for the target virtual try-on model.

2. The pluggable garment pattern control model construction method as described in claim 1, characterized in that, The sample construction module constructs binary images in the following way: Based on the set of two-dimensional key points of the upper half of the body corresponding to the target clothing in the portrait image. Calculate the horizontal span and vertical span Among them, if the target clothing in the target clothing image is a top, then the two-dimensional keypoint set... This is the set of two-dimensional key points for the shoulder, elbow, and wrist joints of a person in an image; if the target clothing in the image is a bottom garment, then the set of two-dimensional key points is... This is a set of two-dimensional key points for the hip, knee, and ankle joints of a person in an image. Construct a rectangular region. Widen its horizontal span by the same first preset amount at both ends, defining its horizontal span. Extend its vertical span by the same second preset amount at both ends, defining its vertical span. The center point of this rectangular region... and width and height Determined by the following formula: ； ； In the formula, and The horizontal widening ratio and the vertical lengthening ratio; Construct a binary image of the same size as the person's image, with pixel values ​​set to 1 within a rectangular region and 0 elsewhere.

3. The pluggable garment pattern control model construction method as described in claim 1, characterized in that, The sample construction module constructs dense pose simulation graphs in the following way: For the torso, quadrilaterals are constructed with the shoulder and hip joints as vertices, and the shoulder and hip joints are connected by line segments. Arcs are used to connect the two shoulder joints and the two hip joints. For the arms, circles are constructed with the shoulder, elbow, and wrist joints of the person in the image as centers, each with a specific diameter preset based on human proportions. Tangents connecting these three circles near the torso and those away from the torso form two cylinders to simulate the upper arm and forearm. For the legs, circles are constructed with the hip, knee, and ankle joints of the person in the image as centers, each with a specific diameter preset based on human proportions. Tangents connecting these three circles near the human midline and those away from the human midline form two cylinders to simulate the thigh and calf. This yields a composite dense pose based on human proportions and key points. The composite dense pose is then color-filled, with the colors of each body part matching the corresponding parts predicted by the preset model. The intersection of the color-filled composite dense pose and the model-predicted dense pose is used as the simulation image of the person's dense pose in the image.

4. The pluggable garment pattern control model construction method as described in claim 3, characterized in that, When constructing circles centered on the shoulder, elbow, and wrist joints of the person in the image, the diameters are 0.06, 0.048, and 0.033 times the person's height, respectively; when constructing circles centered on the hip, knee, and ankle joints of the person in the image, the diameters are 0.09, 0.055, and 0.03 times the person's height, respectively. The height of a person is defined as 3.2 times the torso height (the vertical height between the shoulder and hip in the image) or 2.3 times the leg length. The vertical height between the shoulder and hip in the image is taken as the torso height. If the person in the image is standing, the vertical height between the hip and ankle is taken as the leg length. If the person in the image is sitting, twice the vertical height between the knee and ankle is taken as the leg length.

5. The pluggable garment pattern control model construction method as described in claim 1, characterized in that, The U-Net uses a pre-trained Stable Diffusion U-Net and modifies its structure, including changing the input layer to 13 channels and the output layer to 3 channels. And each training sample is input to the layout generator. Previously, the VAE encoder in the Stable Diffusion was used. The mask image in the training sample Simulated dense posture diagram Target clothing image and noise graph Mapping these to the latent space yields four-channel tensors, which are then used to generate the layout generator. Input In the formula, For all-zero tensors, To splice along the channel dimension, To splice together in the spatial dimension.

6. The pluggable garment pattern control model construction method as described in claim 1, characterized in that, The resolution alignment unit aligns the intermediate feature resolutions of multi-scale pattern-aware features and the target virtual fitting model as follows: Determine the architecture of the target virtual fitting model. If the architecture is a U-Net-like architecture, perform tensor space segmentation on the multi-scale pattern-aware features after convolution to align the multi-scale pattern-aware features with the intermediate feature resolution of the target virtual fitting model. If the architecture is a DiT-like architecture, upsample the features of each resolution in the multi-scale pattern-aware features after convolution to the resolution of the corresponding layer features in the DiT class through interpolation and flatten them into sequence tokens to align the multi-scale pattern-aware features with the intermediate feature resolution of the target virtual fitting model.

7. The pluggable garment pattern control model construction method as described in claim 1, characterized in that, The semantic alignment unit is a convolutional module used for... Perform convolution operations on each scale feature; When training the multi-scale pattern injector begins, each convolutional layer in the convolutional module is a zero-initialized convolutional layer.

8. A virtual try-on method, characterized in that, include: Based on the received person image, target clothing image, and target pattern label, a pluggable clothing pattern control model is used to obtain multi-scale pattern perception features that are semantically and resolutionally aligned with the intermediate features of the target virtual try-on model. The original inputs required for the target virtual try-on model and the multi-scale pattern perception features are input into the target virtual try-on model to obtain the virtual try-on effect image, thus completing the virtual try-on.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein the computer program, when executed by a processor, controls the device on which the storage medium resides to perform the steps of the method as described in any one of claims 1 to 8.

Images