Method for training a fitting model, virtual fitting method and related devices

Abstract

The embodiment of the application relates to the image processing technical field, discloses a kind of training fitting model method, virtual fitting method and related device, fitting network image generation network, clothes deformation network and fusion module, using several image groups corresponding model contour image, clothes image is trained to fitting network, and fitting model is obtained.Therein, the model contour image obtained by combining body posture text information and body torso map can reflect the model body shape contour and body posture features, so that the model contour image is input into the clothes deformation network to guide the clothes deformation, so that the clothes deformation conforms to the human body contour features and body posture features.Several image groups are trained to the fitting network in the above manner, and the loss and back propagation is based on, and the fitting network is constrained to converge.Therefore, the fitting model obtained by training can make the clothes more fit the human body contour, conform to the human body posture, and the fitting effect is more real and accurate.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method for training a virtual fitting model, a virtual fitting method, and related apparatus. Background Technology

[0002] With the continuous advancement of modern technology and the increasing scale of online shopping, users can purchase clothing on online shopping platforms via their mobile phones. However, since the information about the clothing for sale is generally presented as two-dimensional images, users cannot know how the clothing will look on them, which may lead to them buying clothes that are not suitable for them and resulting in a poor shopping experience.

[0003] With the continuous development of neural networks, they have been widely applied in the field of image generation. Therefore, researchers have applied neural networks to virtual try-on, proposing various try-on algorithms. Existing virtual try-on algorithms all first deform the clothing and then generate the skin area to achieve the try-on function. However, this method relies too heavily on human body key points and human anatomy maps. Because there are relatively few human body key points, and the body shape in the anatomy map is obscured by the original clothing, the try-on clothing cannot fit the human body contour; that is, it is difficult for the try-on clothing to conform to the contour and posture information of the human body. Summary of the Invention

[0004] The main technical problem solved by the embodiments of this application is to provide a method for training a fitting model, a virtual fitting method, and related devices. The fitting model trained by this method can make the clothes fit the human body contour more closely and conform to the human body shape, and the fitting effect is more realistic and accurate.

[0005] To address the aforementioned technical problems, in a first aspect, this application provides a method for training a virtual fitting model, wherein the virtual fitting network includes an image generation network, a clothing deformation network, and a fusion module;

[0006] The methods include:

[0007] Acquire several image groups, each including clothing images and model images. The model images contain images of the model wearing the clothing shown in the clothing images, and each model image also contains corresponding body posture text information.

[0008] The body posture text information is feature-encoded to obtain a body posture feature map; and the torso image is extracted from the model image.

[0009] The body feature map and body torso map are input into the image generation network for encoding and then decoding to obtain the model's outline image;

[0010] The model's outline image and clothing image are input into the clothing deformation network to deform the clothing in the clothing image, resulting in a deformed clothing image.

[0011] The deformed clothing image and the model's outline image are input into the fusion module for fusion processing to obtain the pre-test clothing image;

[0012] The loss function is used to calculate the loss between the pre-test clothing image and the model image. Based on the loss sum corresponding to several image groups, the clothing try-on network is iteratively trained until convergence, thus obtaining the clothing try-on model.

[0013] In some embodiments, the body posture text information corresponds to a body posture contour image, and the fitting network also includes a style transfer network, which is used to extract features from the input image and output multiple feature maps of different sizes.

[0014] The method also includes:

[0015] A style transfer network was used to extract features from the body contour image and the model contour image, respectively, to obtain multiple body contour feature maps and multiple model contour feature maps.

[0016] The aforementioned calculation of the loss between the pre-test garment image and the model image using a loss function includes:

[0017] The style transfer loss between multiple body contour feature maps and multiple model contour feature maps is calculated using a loss function.

[0018] In some embodiments, the aforementioned calculation of the loss between the pre-test garment image and the model image using a loss function further includes:

[0019] The loss function is used to calculate the clothing deformation loss between the clothing in the clothing deformation image and the clothing in the model image;

[0020] The reconstruction loss between the model image and the pre-test garment image is calculated using a loss function.

[0021] In some embodiments, the loss function includes:

[0022] Loss=λ0L style +λ1L warp +λ2L rec ;in,

[0023]

[0024]

[0025] L rec =||YR||1;

[0026] Where λ0, λ1, and λ2 are hyperparameters, L style For style transfer loss, L warp For the loss due to clothing deformation, L recThe reconstruction loss is H, where H is the body contour image. The image represents the model's silhouette. VGG is a style transfer network, where V is the total number of layers in the style transfer network, and F... i and R i These are the activation count and the number of elements in the i-th layer of the style transfer network, respectively. It's the clothes in the model's picture. It is the clothing in the distorted image of clothing. It is the outline of the clothes in the model's image. Y is the outline of the clothing in the clothing deformation image, Y is the model image, and R is the pre-test clothing image.

[0027] In some embodiments, the virtual fitting network further includes a multilayer perceptron module;

[0028] The aforementioned feature encoding of body posture text information yields a body posture feature map, including:

[0029] The bag-of-words model is used to encode each word in the body language text information to obtain the text encoding;

[0030] A multilayer perceptron module is used to extract features from the text encoding to obtain a body shape feature map.

[0031] In some embodiments, the clothing deformation network includes a Transformer module, a feature matching module, and a deformation module;

[0032] The aforementioned input of a model outline image and a clothing image into a clothing deformation network to deform the clothing in the clothing image, resulting in a deformed clothing image, includes:

[0033] Input the model outline image and clothing image into the Transformer module to obtain the first model outline feature map and the first clothing feature map, respectively.

[0034] The first model outline feature map and the first clothing feature map are input into the feature matching module for feature matching and association to obtain the association matrix. The association matrix reflects the pixel association and matching between the model and the clothing.

[0035] The deformation module deforms the clothing image based on the correlation matrix to obtain a deformed clothing image.

[0036] In some embodiments, the virtual fitting network further includes a feature extraction module. Before the step of inputting the model contour image and the clothing image into the Transformer module to obtain the first model contour feature map and the first clothing feature map, respectively, the method further includes:

[0037] The feature extraction module is used to extract features from the model's contour image to obtain a second model contour feature map;

[0038] The feature extraction module is used to extract features from the clothing image to obtain a second clothing feature map;

[0039] The aforementioned input of the model outline image and clothing image into the Transformer module yields the first model outline feature map and the first clothing feature map, respectively, including:

[0040] Input the second model outline feature map and the second clothing feature map into the Transformer module to obtain the first model outline feature map and the first clothing feature map, respectively.

[0041] To address the aforementioned technical problems, in a second aspect, this application provides a virtual try-on method, comprising:

[0042] Acquire images of the person trying on the clothes, images of the clothes to be tried on, and textual information about the person's body posture;

[0043] Feature encoding is performed on the textual information of the wearer's posture to obtain the wearer's posture feature map; and the wearer's torso image is extracted from the wearer's image.

[0044] Input the body shape feature map and the torso map of the wearer into the fitting model to generate a fitting image. The fitting model is trained using the method described in any one of claims 1-7.

[0045] To address the aforementioned technical problems, in a third aspect, this application provides an electronic device, comprising:

[0046] At least one processor, and

[0047] A memory communicatively connected to at least one processor, wherein,

[0048] The memory stores instructions that can be executed by at least one processor, such that the instructions are executed by at least one processor to enable the at least one processor to perform the method described in the first aspect above.

[0049] To address the aforementioned technical problems, in a fourth aspect, this application provides a computer-readable storage medium storing computer-executable instructions for causing a computer device to perform the method described in the first aspect above.

[0050] The beneficial effects of this application's embodiments are as follows: Unlike existing technologies, the method for training a fitting model provided in this application first constructs a fitting network including an image generation network, a clothing deformation network, and a fusion module. Then, it acquires several image groups, each including a clothing image and a model image. The model in the model image wears the clothing in the clothing image, and the model image corresponds to body posture text information (e.g., height, weight, and body shape). Next, it performs feature encoding on the body posture text information to obtain a body posture feature map; and extracts a torso image from the model image. The body posture feature map and torso image are input into the image generation network for encoding and decoding to obtain a model contour image. The model contour image and clothing image are input into the clothing deformation network to deform the clothing in the clothing image, obtaining a deformed clothing image. The deformed clothing image and model contour image are input into the fusion module for fusion processing to obtain a pre-test fitting image. Finally, a loss function is used to calculate the loss between the pre-test clothing images and the model images. After processing these several image groups in turn, the clothing try-on network is iteratively trained based on the loss sums corresponding to these image groups until convergence. The converged clothing try-on network is the clothing try-on model.

[0051] In this embodiment, the model contour image obtained by combining body posture text information with a body torso image can reflect the model's body shape and posture features. Therefore, the model contour image is input into the clothing deformation network to guide clothing deformation. The clothing in the clothing image can be deformed according to the model's body shape and posture features, making the clothing deformation conform to the human body contour and posture characteristics. Compared to using human keypoints to guide clothing deformation, the model contour image provides the clothing deformation network with the model's body shape and posture features, making the clothing in the deformed image more closely fit the human body contour. Several image groups are used to train the virtual fitting network in the above manner. Based on the backpropagation of the loss sum, the virtual fitting effect in the pre-test virtual fitting images corresponding to each image group is constrained to continuously approach the real wearing effect in the model image, making the clothing deformation more consistent with the human body contour and posture characteristics. Therefore, the trained virtual fitting model can make the virtual fitting clothes fit the human body contour more closely and conform to the human body posture, resulting in a more realistic and accurate virtual fitting effect. Attached Figure Description

[0052] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0053] Figure 1 This is a schematic diagram illustrating the application scenarios of the fitting system in some embodiments of this application;

[0054] Figure 2This is a schematic diagram of the structure of an electronic device in some embodiments of this application;

[0055] Figure 3 This is a flowchart illustrating the method for training a fitting model in some embodiments of this application;

[0056] Figure 4 This is a schematic diagram of the fitting network structure in some embodiments of this application;

[0057] Figure 5 This is a schematic diagram of the fitting network structure in some embodiments of this application;

[0058] Figure 6 This is a schematic diagram of the structure of the clothing deformation network in some embodiments of this application;

[0059] Figure 7 This is a schematic diagram of the Transformer module in some embodiments of this application;

[0060] Figure 8 This is a flowchart illustrating the virtual try-on method in some embodiments of this application. Detailed Implementation

[0061] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. In addition, the terms "first," "second," and "third" used herein do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0064] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0065] Furthermore, the technical features involved in the various embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0066] To facilitate understanding of the methods provided in the embodiments of this application, the terms used in the embodiments of this application will first be introduced:

[0067] (1) Neural Network

[0068] A neural network can be composed of neural units, specifically understood as a neural network with input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. Neural networks with many hidden layers are called deep neural networks (DNNs). The work of each layer in a neural network can be described by the mathematical expression y = a(W·x + b). From a physical perspective, the work of each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 2 and 3 are performed by "W·x", operation 4 by "+b", and operation 5 by "a()". The term "space" is used here because the objects being classified are not individual things, but a class of things; space refers to the set of all individuals within that class. W is the weight matrix of each layer in the neural network, where each value represents the weight of a neuron in that layer. This matrix W determines the spatial transformation from the input space to the output space, as described above; that is, the W of each layer of the neural network controls how the space is transformed. The purpose of training the neural network is to ultimately obtain the weight matrices of all layers of the trained neural network. Therefore, the training process of a neural network is essentially learning how to control spatial transformation, more specifically, learning the weight matrix.

[0069] It should be noted that, in the embodiments of this application, the models used for machine learning tasks are essentially neural networks. Common components in neural networks include convolutional layers, pooling layers, normalization layers, and deconvolutional layers. By assembling these common components in neural networks, a model is designed. When the model parameters (weight matrices of each layer) are determined such that the model error meets a preset condition or the number of model parameters is adjusted to reach a preset threshold, the model converges.

[0070] The convolutional layer is configured with multiple convolutional kernels, each with a corresponding stride, to perform convolution operations on the image. The purpose of convolution is to extract different features from the input image. The first convolutional layer may only extract some low-level features such as edges, lines, and corners, while deeper convolutional layers can iteratively extract more complex features from low-level features.

[0071] A deconvolutional layer is used to map a low-dimensional space to a high-dimensional space while preserving the connections / patterns between them (the connections during convolution). A deconvolutional layer is configured with multiple convolutional kernels, each with a corresponding stride, to perform deconvolution operations on the image. Generally, framework libraries used for designing neural networks (such as the PyTorch library) have a built-in `upsumple()` function, which allows for low-dimensional to high-dimensional spatial mapping.

[0072] Pooling layers mimic the human visual system, reducing the dimensionality of data and representing images with higher-level features. Common pooling operations include max pooling, mean pooling, random pooling, median pooling, and combined pooling. Typically, pooling layers are periodically inserted between convolutional layers in neural networks to achieve dimensionality reduction.

[0073] The normalization layer is used to normalize all neurons in the intermediate layer to prevent gradient explosion and gradient vanishing.

[0074] (2) Loss Function

[0075] During neural network training, to ensure the output closely approximates the desired predicted value, we compare the network's current prediction with the target value. Based on the difference, we update the weight matrix of each layer (usually an initialization process before the first update, pre-configuring parameters for each layer). For example, if the network's prediction is too high, the weight matrix is ​​adjusted to predict a lower value, and this adjustment continues until the neural network accurately predicts the target value. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the neural network becomes a process of minimizing this loss.

[0076] (3) Human body analysis

[0077] Human body analysis refers to segmenting a person captured in an image into multiple semantically consistent regions, such as body parts and clothing, or further subcategories of body parts and clothing. Essentially, it involves pixel-level recognition of the input image and labeling each pixel with its corresponding object category. For example, neural networks can be used to distinguish various elements (including hair, face, limbs, clothing, and background) in an image containing a human body.

[0078] Before introducing the embodiments of this application, a brief introduction will be given to the virtual try-on method known to the inventors of this application, so as to facilitate the understanding of the embodiments of this application later.

[0079] Generally, Generative Adversarial Networks (GANs) are used to train virtual fitting models. These models are then placed on a terminal for user access. After acquiring the user's image and the image of the clothing to be tried on, a virtual fitting image can be generated. Specifically, the fitting model first deforms the clothing based on human keypoints or a human anatomy map. Then, the deformed clothing is fused with the human torso to generate the skin area, resulting in a fitting image and enabling the fitting function. However, this method relies excessively on human keypoints and anatomy maps. Because there are relatively few human keypoints, and the body shape in the anatomy map is often obscured by the original clothing, it fails to reflect body posture information. This leads to unnatural clothing deformation results that do not conform to the human body contour. In other words, it is difficult to fit clothing in a fitting image that accurately reflects the body's contour and posture information.

[0080] To address the aforementioned issues, this application provides a method for training a virtual fitting model, a virtual fitting method, and related apparatus. A fitting network is constructed, comprising an image generation network, a clothing deformation network, and a fusion module. The fitting network is trained using model contour images and clothing images corresponding to several image groups to obtain the fitting model. Specifically, the model contour image, obtained by combining body posture text information with a body torso image, reflects the model's body shape and posture characteristics. Therefore, the model contour image is input into the clothing deformation network to guide clothing deformation. The clothing in the clothing image can be deformed according to the model's body shape and posture characteristics, making the clothing deformation conform to human body contour and posture features. Compared to using human keypoints to guide clothing deformation, the model contour image provides the clothing deformation network with the model's body shape and posture characteristics, making the clothing in the deformed images more closely fit the human body contour. By training the fitting network using several image groups in the above manner, and based on backpropagation of the loss sum, the fitting effect in the pre-test fitting images corresponding to each image group is constrained to continuously approach the actual wearing effect in the model images, making the clothing deformation more consistent with human body contour and posture features. Therefore, the trained fitting model can make the clothes fit the human body contours better and conform to the human body shape, making the fitting effect more realistic and accurate.

[0081] The following describes an exemplary application of the electronic device provided in the embodiments of this application for training a fitting model or for virtual fitting. It is understood that the electronic device can both train a fitting model and use the fitting model to perform virtual fitting and generate fitting images.

[0082] The electronic device provided in this application embodiment can be a server, such as a server deployed in the cloud. When the server is used to train a virtual fitting model, it iteratively trains the virtual fitting network using the training set provided by other devices or those skilled in the art, determines the final model parameters, and then configures the virtual fitting network with these final model parameters to obtain the virtual fitting model. The training set includes several image groups, each including a clothing image and a model image. The model image shows the model wearing the clothing shown in the clothing image, and the model image corresponds to body posture text information. When the server is used for virtual fitting, it calls the built-in virtual fitting model and performs corresponding calculations on the wearer image, the clothing image to be tried on, and the wearer's body posture text information provided by other devices or users to generate a virtual fitting image that conforms to the human body contour and posture.

[0083] The electronic devices provided in some embodiments of this application can be various types of terminals such as laptops, desktop computers, or mobile devices. When the terminal is used to train a virtual fitting model, those skilled in the art input a prepared training set into the terminal and design a virtual fitting network on the terminal. The terminal uses the training set to iteratively train the virtual fitting network to determine the final model parameters. The virtual fitting network is then configured with these final model parameters to obtain the virtual fitting model. When the terminal is used for virtual fitting, it calls the built-in virtual fitting model and performs corresponding calculations on the user-inputted image of the wearer, the image of the garment to be tried on, and the textual information about the wearer's body posture to generate a virtual fitting image that can change the attributes of the garment to be tried on.

[0084] As an example, see Figure 1 , Figure 1 This is a schematic diagram of the application scenario of the fitting system provided in the embodiment of this application. The terminal 10 is connected to the server 20 through a network, wherein the network can be a wide area network or a local area network, or a combination of the two.

[0085] Terminal 10 can be used to acquire training sets and construct a virtual fitting network. For example, those skilled in the art can download a prepared training set on the terminal and build the network structure of the virtual fitting network. It is understood that terminal 10 can also be used to acquire images of the user, images of the clothing to be tried on, and textual information about the user's posture. For example, the user inputs the image of the user, the image of the clothing to be tried on, and textual information about the user's posture through an input interface. After input, the terminal automatically acquires the image of the user, the image of the clothing to be tried on, and the textual information about the user's posture. For example, terminal 10 has a camera to capture images of the user, and terminal 10 stores a clothing image library. The user can select an image of the clothing to be tried on from the clothing image library, and then input the textual information about the user's posture through the input interface of terminal 10.

[0086] In some embodiments, terminal 10 locally executes the method for training a fitting model provided in this application to train the designed fitting network using a training set, determine the final model parameters, and then configure the fitting network with the final model parameters to obtain a fitting model. In some embodiments, terminal 10 may also send a training set and a constructed fitting network stored on the terminal by a person skilled in the art to server 20 via a network. Server 20 receives the training set and the fitting network, trains the designed fitting network using the training set, determines the final model parameters, and then sends the final model parameters to terminal 10. Terminal 10 saves the final model parameters so that the fitting network can be configured with the final model parameters to obtain a fitting model.

[0087] In some embodiments, terminal 10 locally executes the virtual try-on method provided in this application embodiment to provide virtual try-on services to users. It invokes a built-in try-on model to perform corresponding calculations on the user-inputted image of the wearer, the image of the garment to be tried, and the text information about the wearer's body posture, generating a try-on image capable of changing the attributes of the garment. In some embodiments, terminal 10 can also send the user-inputted image of the wearer, the image of the garment to be tried, and the text information about the wearer's body posture to server 20 via a network. Upon receiving these materials, server 20 invokes its built-in try-on model to perform corresponding calculations on the images, generating a try-on image capable of changing the attributes of the garment, and then sends the try-on image to terminal 10. After receiving the try-on image, terminal 10 displays it on its own display interface for the user to view the try-on effect after editing the garment's attributes.

[0088] The structure of the electronic device in the embodiments of this application is described below. Figure 2 This is a schematic diagram of the structure of an electronic device 500 in an embodiment of this application. The electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It is understood that the bus system 540 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 2 The general labeled all buses as Bus System 540.

[0089] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor.

[0090] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0091] Memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory. Memory 550 may optionally include one or more storage devices physically located away from processor 510.

[0092] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0093] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks.

[0094] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0095] Display module 553 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with user interface 530;

[0096] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0097] As can be understood from the above, the method for training a virtual fitting model and the virtual fitting method provided in the embodiments of this application can be implemented by various types of electronic devices with computing power, such as smart terminals and servers.

[0098] The following describes the method for training a virtual fitting model provided in this application, using exemplary applications and implementations of the server provided in the embodiments of this application. See also... Figure 3 , Figure 3 This is a flowchart illustrating the method for training a fitting model provided in an embodiment of this application. Please refer to... Figure 4 , Figure 4 This is a schematic diagram of the fitting room network structure. (Example) Figure 4As shown, the fitting model network structure includes an image generation network, a clothing deformation network, and a fusion module. The image generation network, clothing deformation network, and fusion module are all pre-configured neural networks with neural network components (convolutional layers, deconvolutional layers, or pooling layers, etc.). The basic structure and principles of neural networks have been explained in detail in "Glossary (1)," and will not be elaborated further here. The fitting network can be constructed by those skilled in the art on a neural network design platform on a terminal (e.g., a computer) and then sent to a server.

[0099] In some embodiments, the intra-layer structure (convolutional kernel, stride, etc.), inter-layer connection structure, and layer combination of convolutional layers, deconvolutional layers, or pooling layers in the image generation network can be configured to obtain a specific image generation network. The image generation network includes an encoder and a decoder. Those skilled in the art will understand that the encoder includes multiple downsampling convolutional layers; in the encoder, as the downsampling convolutional layers progress, the size of the output feature map decreases. The decoder includes multiple upsampling convolutional layers; in the decoder, as the upsampling convolutional layers progress, the size of the output feature map increases. Those skilled in the art can configure parameters such as the kernel size and stride of each downsampling and upsampling convolutional layer according to actual needs.

[0100] Similarly, the structure of the clothing deformation network can be set so that it can be used to deform clothing; the structure of the fusion module can also be set so that it can fuse at least two feature data so that the fusion result can reflect the features in at least two feature data.

[0101] Please refer to it again. Figure 3 The method S100 may specifically include the following steps:

[0102] S10: Obtain several image groups.

[0103] For any given image set, including clothing images and model images, the model in the model image is wearing the clothing from the clothing images. The clothing images contain clothing that the model wants to try on; for example, clothing image 1# includes a green short-sleeved shirt. The model images contain clothing from the corresponding clothing images; for example, the model in the model image corresponding to clothing image 1# is wearing that green short-sleeved shirt.

[0104] Each model image corresponds to body posture text information. In other words, the clothing image, the corresponding model image, and the corresponding body posture text information constitute a set of training data. The body posture text information describes the model's body shape and posture; for example, it may include data such as "height, weight, body shape, and build." Body shape may include apple-shaped, H-shaped, pear-shaped, hourglass-shaped, rectangular, V-shaped, round, plump hourglass-shaped, 50 / 50 inch, petite, and very tall. For example, the body posture text information could be "height 165cm, weight 50kg, apple-shaped, well-proportioned."

[0105] It is understood that those skilled in the art can set the data information included in the body posture text information themselves. It is also understood that several image sets can be collected in advance by those skilled in the art on a terminal (e.g., a computer). For example, images of clothing and corresponding images of models wearing the clothing can be crawled from some clothing sales websites, and body posture text information can be prepared based on the model's body posture in the model image. After preparing several image sets and the body posture text information corresponding to each model image (or each image set), this training data is uploaded to the server via the terminal.

[0106] In some embodiments, to avoid altering the model's posture, the collected model image is padded to achieve a 1:1 aspect ratio. For example, the width is padded proportionally using the difference between length and width, with the padded value set to 0, followed by normalization. The model image size can be set to 512*512. Similarly, clothing images can also be padded to achieve a 1:1 aspect ratio, with a size of 512*512.

[0107] In some embodiments, the number of image groups is in the tens of thousands, for example, 20,000, which is beneficial for training an accurate general model. Those skilled in the art can determine the number of image groups according to the actual situation.

[0108] S20: Perform feature encoding on the body posture text information to obtain a body posture feature map; and extract the body torso map from the model image.

[0109] It is understandable that body posture text information includes textual information such as height, weight, body shape, and build. Body posture text information is text-formatted data. To enable the virtual fitting network to learn the body contour and posture features reflected in the body posture text information, feature encoding is performed on the body posture text information to obtain a body posture feature map. In other words, the body posture text information is quantified, and the resulting two-dimensional body posture feature map is numerical data.

[0110] In some embodiments, please refer to Figure 5 The virtual fitting network also includes a multilayer perceptron module.

[0111] The aforementioned "encoding the features of body posture text information to obtain a body posture feature map" includes: encoding each word in the body posture text information using a bag-of-words model to obtain text encoding; and extracting features from the text encoding using a multilayer perceptron module to obtain a body posture feature map. In some embodiments, the size of the body posture feature map is the same as the size of the model image, which can be 512*512.

[0112] The bag-of-words model involves selecting words from the text data, placing them into a bag, counting the frequency of each word in the bag within the text data, and representing them as vectors. In this embodiment, the words in all body shape text information are first integrated to construct a dictionary. It is understood that in some embodiments, the constructed dictionary may be {'body shape': {'apple-shaped 0001', 'hourglass-shaped 0010', 'H-shaped 0100'...}, and 'fat / thin': {'thin 0011', 'slightly overweight 0110', 'fat 1100', 'well-proportioned 1001'}. The corresponding codes for each word in the body shape text information are concatenated with the original numerical data to obtain the text encoding. For example, for the body shape text information "height 165cm, weight 50kg, apple-shaped, well-proportioned", its text encoding could be "165 50 0001 1001".

[0113] A multilayer perceptron module consists of an input layer, multiple hidden layers, and an output layer. The input layer contains N neurons, the hidden layers contain Q neurons, and the output layer contains K neurons. The operation of each layer can be described by a function expression; understandably, the function expression for each layer is different.

[0114] Understandably, if the input information feature encoding is represented by x, then the input layer is fed to the hidden layer x, and the output of the hidden layer can be f(w1x+b1), where w1 is the weight, b1 is the bias, and the function f can be the commonly used sigmoid function or tanh function. The process from the hidden layer to the output layer is equivalent to a multi-class logistic regression, i.e., softmax regression. Therefore, the output of the output layer is softmax(w2x1+b2), where x1 is the output of the hidden layer f(w1x+b1).

[0115] Therefore, the multilayer perceptron module can be represented by the following formula:

[0116]

[0117] Where G represents the softmax activation function, h represents the number of hidden layers, and W... i and b i Let represent the weights and biases of the i-th hidden layer. Let x represent the encoded information features of the input. W 1 and b1 Let S represent the weights and biases of the input layer, S represent the activation function, and MLP(x) represent the target information vector.

[0118] In some embodiments, K can be 1024, so that the output layer outputs a one-dimensional vector of length 1024, that is, a target information vector of length 1024.

[0119] The multilayer perceptron module uses activation functions in each layer, introducing nonlinear factors into neurons. This allows the module to approximate any nonlinear function, thus enabling its application in a wider range of nonlinear models. The multilayer perceptron module also possesses excellent feature extraction capabilities for discrete information. Therefore, using the multilayer perceptron module to extract features from text encoding yields a body feature map that fully reflects the model's body contour and posture characteristics.

[0120] In this embodiment, a multilayer perceptron module is used to extract text encoding, so that the body feature map can more fully reflect the model's body contour features and body features. As a result, the fitting network can learn the model's body contour features and body features better. Therefore, using a multilayer perceptron module to extract body feature maps is beneficial to model convergence.

[0121] Furthermore, in step S20, a torso image is extracted from the model image. It is understandable that when changing clothes on a model in a model image, it is necessary to preserve the model's identity features and other essential characteristics. Extracting the model's identity features, i.e., obtaining the torso image, before merging the clothes and the model serves two purposes: firstly, it avoids interference from the original clothing features during fusion; secondly, it preserves the model's identity features, ensuring that the model's appearance is not distorted after trying on the clothes.

[0122] Specifically, firstly, the model image can be analyzed to obtain an analyzed image. As mentioned in the "Glossary (3)" above, human body analysis involves segmenting the various parts of the human body and assigning categories to the pixels. From the analyzed image, the category to which each part of the image belongs can be determined. Then, the torso region (e.g., face, neck, hands, feet, etc.) in the analyzed image is retained, and the pixel category corresponding to the torso region is set to 1, while other categories are set to 0, resulting in a binarized image. Finally, the corresponding positions of the pixels in the model image and the pixels in the binarized image are multiplied to obtain the torso image.

[0123] Multiplying corresponding positions can be done using the formula F′. ij =F ij ×M ij Explain, where F ij M represents the pixel value in the i-th row and j-th column of the model image. ijF′ represents the value in the i-th row and j-th column of the binarized image. ij The pixel value in the i-th row and j-th column of the body torso image.

[0124] In this embodiment, the body torso image can accurately preserve the model's identity features and remove the original clothing features that need to be replaced, so that the model's identity is not distorted after changing into the clothes to be tried on.

[0125] S30: Input the body feature map and body torso map into the image generation network for encoding and decoding to obtain the model's outline image.

[0126] It's understandable that because the original clothing obscures the torso—for example, a long-sleeved shirt covers the upper body and arms—a torso drawing without the clothing cannot reflect the body's contours and posture. For instance, it's impossible to tell from the torso drawing whether the model's upper body is thin or plump. Therefore, the clothing distortion doesn't accurately reflect the actual body shape. Applying the same distortion method to people with different body types (thin or plump) would result in clothing distortion that is inconsistent with the human form.

[0127] In order to ensure that the body torso image can reflect the body contour and posture features and guide the reasonable deformation of clothing in the clothing image, firstly, the posture feature image and the body torso image are input into the image generation network for encoding and then decoding to obtain a model contour image that can reflect the body contour and posture features.

[0128] In some embodiments, the image generation network includes an encoder and a decoder, wherein the encoder mainly consists of convolutional layers, activation function layers, and instance normalization layers, and the corresponding mathematical expression is as follows:

[0129]

[0130] in, This represents the m-th feature map of the i-th layer. Let represent the nth feature map of the (+l)th layer, W represent the convolution kernel, B represent the bias term, σ(·) represent the LealyReLU activation function, and IN represent instance normalization. Since downsampling and upsampling introduce information loss, a uniform stride convolution is used, with five different parameters set for the number of kernels in each layer: 16, 32, 64, 128, and 256. The final feature map is 8*8*256 pixels in size.

[0131] The decoder mainly consists of transposed convolutional layers, activation function layers, and instance normalization layers. The transposed convolutional layer maps the feature layer to the input size. Compared to upsampling, transposed convolution can better recover image details. In the final transposed convolutional layer, the number of filters is 3. After the 8*8*256 feature map is upsampled by the decoder, the final output is a 512*512*3 model contour image.

[0132] In other words, the body posture feature map and the torso image are concatenated and then input into the image generation network. They undergo spatial transformation processing, first downsampling and then upsampling, so that the resulting model silhouette image includes not only the body contour and posture features reflected in the body posture text information, but also the torso identity features reflected in the torso image. Therefore, the model silhouette image can not only guide clothing deformation, but also be used to fuse with the deformed clothing image, accurately preserving the model's identity features, ensuring that the model's identity is not distorted when wearing the clothes to be tried on.

[0133] S40: Input the model outline image and clothing image into the clothing deformation network to deform the clothing in the clothing image and obtain the clothing deformation image.

[0134] Among them, the clothing deformation network is used to deform clothing. It is understood that the clothing deformation network is also a type of neural network, and those skilled in the art can design the specific structure of the clothing deformation network according to actual needs.

[0135] After concatenating the model's silhouette image and the clothing image, the images are input into a clothing deformation network. Each layer or component module in the clothing deformation network (e.g., convolutional layers, pooling layers, activation function layers, or residual extraction modules) performs spatial mapping transformations on the concatenated image. The model's silhouette image provides the clothing deformation network with body contour and posture features, causing the clothing in the clothing image to deform according to these features. Therefore, in the clothing deformation image, the deformation of the clothing conforms to the human body's silhouette and posture features.

[0136] Understandably, as the network training converges, the human contour and posture features in the model's silhouette image become increasingly closer to the model's true silhouette and posture features, resulting in more accurate clothing deformation. This method of introducing body contour and posture features to guide clothing deformation, compared to using human keypoints to guide clothing deformation, provides the clothing deformation network with the model's body contour and posture features, enabling the clothing in the deformation image to better conform to the human body contour.

[0137] In some embodiments, please refer to Figure 6 The clothing deformation network includes a Transformer module, a feature matching module, and a deformation module.

[0138] The entire network structure of the Transformer module is composed of attention mechanisms. By introducing multi-head attention and removing operations such as downsampling and convolution, the Transformer module can, to some extent, compensate for the limitation of traditional neural networks that lack spatial location concepts. The Transformer module includes multiple encoders and multiple decoders, which interact with each other to encode and decode the input image, focusing attention on certain features, thereby obtaining feature maps with globally sparse features.

[0139] The feature matching module is used to match related pixels in two images, such as matching the shoulder pixels of a human body with the shoulder pixels of clothing. The deformation module is used to deform the image, causing the positions of pixels in the image to change.

[0140] Those skilled in the art will understand that the Transformer module, feature matching module, and deformation module described above are all neural networks, and have components of a neural network, such as convolutional layers, pooling layers, normalization layers, deconvolutional layers, fully connected layers, and activation functions, which will not be elaborated on here.

[0141] In this embodiment, the aforementioned step S40 specifically includes:

[0142] S41: Input the model outline image and clothing image into the Transformer module to obtain the first model outline feature map and the first clothing feature map, respectively.

[0143] Please see Figure 7 , Figure 7This is a schematic diagram of the network structure of the Transformer module. The Transformer module includes N encoders, N decoders, linear layers, and activation functions; for example, N can be 6. There is interaction between an encoder and a decoder. The encoder consists of two parts: a multi-head self-attention mechanism and a position-wise feed-forward network. Both parts are followed by residual connections and layer normalization layers. The decoder consists of three parts: a mask multi-head self-attention mechanism, a multi-head self-attention mechanism, and a position-wise feed-forward network. All three parts are followed by residual connections and layer normalization layers. It should be noted that the Transformer module involved in this application is not limited to... Figure 7 The network structure shown also includes various improved Transformer modules.

[0144] The "attention mechanism" refers to the attention that the Transformer module applies to different parts of the input x for the output y at a given time. This attention can be understood as weights, assigning greater weights to useful features. Thus, the Transformer module possesses the ability to handle globally sparse features.

[0145] The Transformer module takes a sequence as input. The image is segmented into pixel blocks, then linearly transformed and stretched into a sequence. Finally, the positional encoding information of each pixel block in the image is added to form the input sequence, which is then fed into the Transformer module. After processing by multiple encoders and decoders, the input sequence outputs a feature map.

[0146] In this embodiment, the images input to the Transformer module include a model outline image and a clothing image. Thus, after the Transformer module performs calculations on the model outline image and the clothing image respectively, it obtains a first model outline feature map and a first clothing feature map respectively.

[0147] The Transformer module employs an attention mechanism, assigning significant weights to features in the input x that influence the output y. For example, it assigns high weights to features in the model silhouette image that influence the first model silhouette feature map, and high weights to features in the clothing image that influence the first clothing feature map. This results in a first model silhouette feature map and a first clothing feature map with globally sparse features, where the sparse features in the first model silhouette feature map and the first clothing feature map are correlated. In both the first model silhouette feature map and the first clothing feature map, there are fewer non-zero pixels and more zero pixels. These non-zero pixels are the correlated pixels selected by the Transformer module. In other words, pixels in the first model silhouette feature map and the first clothing feature map are correlated. For example, shoulder pixels in the first model silhouette feature map are correlated with shoulder pixels in the first clothing feature image.

[0148] S42: Input the first model outline feature map and the first clothing feature map into the feature matching module for feature matching and association to obtain the association matrix.

[0149] The association matrix reflects the pixel-level matching between the model and the clothing.

[0150] Please refer to it again. Figure 6 After obtaining the first model outline feature map and the first clothing feature map with matching correlation of pixels, the correlation matrix is ​​obtained by calculating the degree of matching correlation between the two through the feature matching module.

[0151] In some embodiments, the feature matching module includes a correlation value calculation layer and a softmax layer; the correlation value calculation layer is used to calculate the correlation value between the corresponding pixels in the first model contour feature map and the corresponding pixels in the first clothing feature map.

[0152] Specifically, a correlation value calculation layer is used to multiply the first model's contour feature map and the first clothing feature map to obtain a correlation matrix. A softmax layer is then used to normalize the correlation matrix to obtain an association matrix.

[0153] In this embodiment, the correlation value calculation layer can use the following formula to calculate the correlation matrix;

[0154]

[0155] In this context, each element in the correlation matrix S represents the correlation value of the corresponding pixel in the first model contour feature map T1 and the first clothing feature map C1. It is a normalization coefficient, which can be set by those skilled in the art according to the actual situation.

[0156] After obtaining the correlation matrix S, the last two dimensions of S are normalized using the softmax function to obtain the correlation matrix M. It can be understood that the correlation matrix M = softmax(s) ∈ R. H*W*H*W .

[0157] In the correlation matrix M obtained after normalization using the softmax function, the value of each element is between [-1, 1]. Each element in M ​​can represent an optical flow point, and the value of the element is the weight of the optical flow point. Optical flow points with higher weights correspond to correlated pixels with higher recognizability and representativeness.

[0158] S43: The deformation module deforms the clothing image based on the correlation matrix to obtain a deformed clothing image.

[0159] Based on the correlation matrix M, which includes optical flow points with different weights, and the weight of the optical flow point corresponding to the correlated pixel, the deformation module deforms the clothing image according to the correlation matrix, so that the clothing image can be deformed according to the optical flow trajectory of the correlated pixel, thus obtaining a deformed clothing image.

[0160] In some embodiments, a standard pixel grid is taken, and multiplied by the correlation matrix to obtain a deformed pixel grid. Finally, a deformation function is used to process the deformed pixel grid and the clothing image to obtain a deformed clothing image.

[0161] The standard pixel grid is a grid image with the same size as the last two dimensions of the correlation matrix; for example, the size of the standard pixel grid is H*W. In the standard pixel grid, each element can have a pixel value of 1, indicating that the pixel has no offset.

[0162] The elements in the standard pixel grid are deformed and offset using an incidence matrix. Specifically, the standard pixel grid G ​​is multiplied by the incidence matrix M to obtain the deformed pixel grid.

[0163] It is understandable that the deformed pixel grid is a grid image obtained by deforming the standard pixel grid according to the correlation matrix, and the pixel value of each pixel represents the offset of the corresponding pixel in the clothing image. Here, the deformation function f is used to calculate and process the deformed pixel grid and the clothing image to obtain the deformed clothing image. This allows the clothing deformation image to be deformed according to the correlation matrix, that is, according to the optical flow trajectory of the correlated pixels with large weights in the correlation matrix.

[0164] In this embodiment, the attention mechanism of the Transformer module is used to establish feature relationships between the model contour image and the clothing image, resulting in a first model contour feature map and a first clothing feature map with globally sparse features. The sparse features in the first model contour feature map and the sparse features in the first clothing feature map are matched and correlated. Then, the feature matching module is used to calculate the correlation matching degree between pixels in the first model contour feature map and the first clothing feature map, obtaining a correlation matrix, which allows for the selection of more recognizable and representative correlated pixels. The deformation module deforms the clothing image based on the optical flow trajectories of these correlated pixels. Compared to deforming by tracking the optical flow trajectories of all pixels (over-deformation), this embodiment uses the optical flow trajectories of correlated pixels to achieve deformation, making the clothing texture more natural.

[0165] It is understood that in this embodiment, "clothing texture" refers to the natural wrinkles formed when clothing is worn on the human body. Clothing fabric is soft and, when covering the human body, conforms to the body's shape, forming natural wavy wrinkles. Clothing laid flat has no wrinkles; that is, the clothing in the original image does not have wrinkles that conform to the body's shape. When deformed to simulate a try-on effect, natural wrinkles are generated because the overall pixel positions of the clothing change. Compared to deforming by tracking the optical flow trajectory of all pixels, this embodiment tracks the optical flow trajectory of associated pixels, resulting in fewer deformation parameters and a more natural clothing texture.

[0166] In some embodiments, please refer to this document. Figure 5 The virtual fitting network also includes a feature extraction module, which consists of multiple convolutional layers used to downsample and extract features from the image to obtain a feature map with a larger receptive field.

[0167] Prior to the aforementioned step S40, the method S100 further includes:

[0168] The feature extraction module is used to extract features from the model's contour image to obtain a second model contour feature map; the feature extraction module is also used to extract features from the clothing image to obtain a second clothing feature map.

[0169] Understandably, please refer to Figure 5 The feature extraction module is a dual-channel convolutional neural network that performs downsampling feature extraction on the combined image to obtain a smaller-resolution second model contour feature map, and performs downsampling feature extraction on the clothing image to obtain a smaller-resolution second clothing feature map. It is understandable that the smaller-resolution second model contour feature map and the second clothing feature map have a larger receptive field.

[0170] In this embodiment, the aforementioned step S40 specifically includes: inputting the second model outline feature map and the second clothing feature map into the Transformer module to obtain the first model outline feature map and the first clothing feature map, respectively.

[0171] The input to the Transformer module is a second model contour feature map and a second clothing feature map with smaller resolution and larger receptive field, which allows the Transformer module to better focus on the feature distribution of the global image and is more conducive to learning.

[0172] S50: Input the clothing deformation image and the model outline image into the fusion module for fusion processing to obtain the pre-test clothing image.

[0173] The fusion module is used to fuse the clothing deformation image and the model contour image, so that the pre-test clothing image (fusion result) can include features from both the clothing deformation image and the model contour image. In some embodiments, the fusion module can be a neural network including an encoder and a decoder. In some embodiments, the fusion module can also be a function layer, using a function to fuse the clothing deformation image and the model contour image. Here, no restrictions are placed on the structure of the fusion module, as long as it can achieve feature fusion.

[0174] The clothing deformation image is based on the outline and posture features of the model in the clothing deformation image. The model outline image also includes the model's identity features. Therefore, after fusing the clothing deformation image and the model outline image, the resulting pre-test clothing image can show the effect of the model trying on the clothes.

[0175] S60: The loss function is used to calculate the loss between the pre-test clothing image and the model image. Based on the loss sum corresponding to several image groups, the clothing try-on network is iteratively trained until convergence, and the clothing try-on model is obtained.

[0176] Here, the loss function can be configured by those skilled in the art in the terminal. The configured loss function is sent to the server along with the virtual try-on network. After the server processes and obtains the pre-test images corresponding to each image group, it uses the loss function to calculate the loss between each model image and the pre-test image. Based on the loss, the virtual try-on network is iteratively trained until the virtual try-on network converges, and the virtual try-on model is obtained.

[0177] Understandably, the smaller the difference between each model image and the pre-fitting garment image, the more similar the model image and the pre-fitting garment image are, indicating that the pre-fitting garment image can accurately reproduce the real fitting effect. Therefore, the model parameters of the aforementioned fitting network can be adjusted based on the differences between each model image and the pre-fitting garment image, and the fitting network can be iteratively trained. That is, the differences are backpropagated, causing the pre-fitting garment image output by the fitting network to continuously approximate the model image until the fitting network converges, resulting in a fitting model. Understandably, in some embodiments, the fitting network includes an image generation network, a garment deformation network, a fusion module, or a multilayer perceptron module. The model parameters then include the model parameters of the image generation network, the garment deformation network, the fusion module, or the multilayer perceptron module, thereby achieving end-to-end training.

[0178] In some embodiments, the Adam algorithm is used to optimize the model parameters. For example, the number of iterations is set to 100,000, the initial learning rate is set to 0.001, the learning rate weight decay is set to 0.0005, and the learning rate decays to 1 / 10 of its original value every 1,000 iterations. The learning rate, loss, and input can be fed into the Adam algorithm to obtain the adjusted model parameters output by the Adam algorithm. The adjusted model parameters are used for the next training until the training is completed. Then, the model parameters of the converged virtual fitting network are output, which is the virtual fitting model.

[0179] Understandably, after the server obtains the model parameters (i.e., the final model parameters) of the converged virtual try-on network, it can send these final model parameters to the terminal. The virtual try-on network within the terminal is configured with these final model parameters, thus obtaining the virtual try-on model. In some embodiments, the server can also save the virtual try-on network and the final model parameters to obtain the virtual try-on model.

[0180] In this embodiment, a virtual fitting network is constructed, comprising an image generation network, a clothing deformation network, and a fusion module. The network is trained using model contour images and clothing images corresponding to several image groups to obtain a virtual fitting model. The model contour image, obtained by combining body posture text information with a torso image, reflects the model's body shape and posture features. Therefore, the model contour image is input into the clothing deformation network to guide clothing deformation. The clothing in the clothing image can be deformed according to the model's body shape and posture features, ensuring that the clothing deformation conforms to human body contour and posture characteristics. Compared to using human keypoints to guide clothing deformation, the model contour image provides the clothing deformation network with the model's body shape and posture features, making the clothing in the deformed images more closely fit the human body contour. By training the virtual fitting network using several image groups in the above manner, and based on backpropagation of the loss sum, the virtual fitting effect in the pre-test clothing images corresponding to each image group is constrained to continuously approach the actual wearing effect in the model images, making the clothing deformation more consistent with human body contour and posture characteristics. Therefore, the trained fitting model can make the clothes fit the human body contours better and conform to the human body shape, making the fitting effect more realistic and accurate.

[0181] In some embodiments, body posture text information corresponds to a body posture contour image. It is understood that each type of body posture text information corresponds to a specific body posture contour image. A body posture contour image is a real image that can represent the corresponding body posture text information. For example, for the body posture text information "height 165cm, weight 50kg, apple-shaped, well-proportioned," the corresponding body posture contour image could be a model with a height of approximately 165cm, a weight of approximately 50kg, an apple-shaped, well-proportioned figure, wearing clothing that can display the body contour, such as swimwear, bodysuit, or shaping clothing.

[0182] It is understandable that the model in the body contour image corresponding to the body posture text information and the model in the model image do not need to be the same person; both only need to have the body contour features or body posture features described in the body posture text information.

[0183] Please refer to it again. Figure 5 The virtual fitting network also includes a style transfer network, which is used to extract features from the input image and output multiple feature maps of different sizes.

[0184] In some embodiments, the style transfer network can be a VGG network. VGG networks are an existing type of convolutional neural network, which will not be described in detail here.

[0185] The method S100 further includes: using a style transfer network to extract features from the body contour image and the model contour image respectively, to obtain multiple body contour feature maps and multiple model contour feature maps.

[0186] In this embodiment, convolutional neural networks (style transfer networks) such as VGG can be used to downsample the body contour image to extract multiple body contour feature maps. Similarly, convolutional neural networks (style transfer networks) such as VGG can be used to downsample the model contour image to extract multiple model contour feature maps. It is understood that the feature map size output by each layer of the style transfer network is different, and the feature map size decreases as the layers progress. The sizes of the multiple body contour feature maps correspond one-to-one with the sizes of the multiple model contour feature maps.

[0187] In this embodiment, the aforementioned "calculating the loss between the pre-test garment image and the model image using a loss function" includes: calculating the style transfer loss between multiple body contour feature maps and multiple model contour feature maps using a loss function.

[0188] Style transfer loss is calculated by comparing the various levels of body contour feature maps obtained from convolving the body contour image (the real image of the body contour text information) with the various levels of model contour feature maps obtained from convolving the model contour image. The smaller the style transfer loss, the closer the body contour image is to the model contour image, and the better the model contour image can reconstruct the real body contour described by the body contour text information.

[0189] Therefore, based on backpropagation using style transfer loss, as the network converges, the constrained model contour image becomes increasingly closer to the body contour image. The model contour image can reconstruct the true body contour described by the body contour text information. Consequently, the image generation network can generate model contour images that conform to the body contour text information description. This is beneficial for subsequently guiding the clothing deformation network to achieve clothing deformation that adapts to the body contour and body features. Furthermore, it is beneficial for generating realistic preview clothing images. Once the clothing try-on network converges, the resulting try-on model allows the clothing to fit the human body contour more closely and conform to the human body posture, resulting in a more realistic and accurate try-on effect.

[0190] In some embodiments, the aforementioned "calculating the loss between the pre-test garment image and the model image using a loss function" further includes: calculating the garment deformation loss between the garment in the garment deformation image and the garment in the model image using a loss function; and calculating the reconstruction loss between the model image and the pre-test garment image using a loss function.

[0191] The clothing deformation loss reflects the contour difference between the clothing in the deformed image and the clothing in the model image (the actual clothing deformation). A greater difference in contour between the clothing in the deformed image and the clothing in the model image indicates that the clothing deformation reflected in the deformed image deviates from the realistic wearing effect, and the deformation effect of the clothing deformation algorithm is poor. Conversely, a smaller difference between the clothing area in the deformed image and the clothing area in the model image indicates that the clothing deformation reflected in the deformed image is closer to the realistic wearing effect, and the deformation effect of the clothing deformation algorithm is good.

[0192] Reconstruction loss is the loss of determining whether the predicted fitting image corresponds to the model image (real fitting image). A large reconstruction loss indicates a significant difference between the distribution of the predicted fitting image and the model image (real fitting image). Conversely, a small reconstruction loss indicates a smaller difference or similarity between the distribution of the predicted fitting image and the model image (real fitting image). Here, the distribution of the fitting image refers to the distribution of various parts of the image, such as the clothing, head, and limbs.

[0193] In some embodiments, the loss function includes:

[0194] Loss=λ0L style +λ1L warp +λ2L rec ;in,

[0195]

[0196]

[0197] L rec =||YR||1;

[0198] Where λ0, λ1, and λ2 are hyperparameters, L style For style transfer loss, L warp For the loss due to clothing deformation, L rec The reconstruction loss is H, where H is the body contour image. The image represents the model's silhouette. VGG is a style transfer network, where V is the total number of layers in the style transfer network, and F... i and R i These are the activation count and the number of elements in the i-th layer of the style transfer network, respectively. It's the clothes in the model's picture. It is the clothing in the distorted image of clothing. It is the outline of the clothes in the model's image. Y is the outline of the clothing in the clothing deformation image, Y is the model image, and R is the pre-test clothing image.

[0199] In some embodiments, convolutional neural networks such as VGG can be used as style transfer networks to downsample the body contour images and extract V body contour feature maps F. i (H). Similarly, VGG or other convolutional neural networks can be used to downsample the model contour image and extract V model contour feature maps.

[0200] In some embodiments, when i = 1, F i (H) and The average size is 8*8; when i = 2, F i (H) and The size is 16*16; when i=3, F i (H) and The size is 32*32; when i=4, F i (H) and The size is 64*64; when i=5, F i (H) and The size is 128*128; when i=6, F i (H) and The size is 256*256; when i=7, F i (H) and The size is 512*512.

[0201] Style transfer loss compares body contour feature maps of different sizes with model contour feature maps and calculates the differences at different granularities. This can quickly constrain the model contour image to become closer and closer to the body contour image, which is beneficial for the convergence of the fitting network and improves training efficiency.

[0202] In some embodiments, convolutional neural networks such as VGG can be used to downsample the clothing in the model image to extract V real clothing deformation feature maps. Similarly, convolutional neural networks such as VGG can be used to downsample the clothing in the clothing deformation image to extract V predicted clothing deformation feature maps.

[0203] In some embodiments, when i = 1 and The size is 8*8; when i=2 and The size is 16*16; when i=3 and The size is 32*32; when i=4 and The size is 64*64; when i=5 and The size is 128*128; when i=6 and The size is 256*256; when i=7 and The size is 512*512.

[0204] Therefore, by iteratively training the virtual fitting network based on the differences calculated by loss functions including style transfer loss, clothing deformation loss, and reconstruction loss, the pre-test virtual fitting images can be constrained to continuously approach the model images (real virtual fitting images), which can accelerate network convergence and improve the virtual fitting effect of the trained virtual fitting model.

[0205] In summary, by designing the above-mentioned virtual fitting network structure, which includes an image generation network, a clothing deformation network, and a fusion module, the virtual fitting network is trained using model contour images and clothing images corresponding to several image groups to obtain a virtual fitting model. Specifically, the model contour image, obtained by combining body posture text information with a torso image, reflects the model's body shape and posture features. Therefore, the model contour image is input into the clothing deformation network to guide clothing deformation. The clothing in the clothing image can be deformed according to the model's body shape and posture features, making the clothing deformation conform to the human body contour and posture features. Compared to using human body keypoints to guide clothing deformation, the model contour image provides the clothing deformation network with the model's body shape and posture features, making the clothing in the deformed images more closely fit the human body contour. By training the virtual fitting network using several image groups in the above manner, and based on backpropagation of the loss sum, the virtual fitting effect in the pre-test clothing images corresponding to each image group is constrained to continuously approach the actual wearing effect in the model images, making the clothing deformation more consistent with the human body contour and posture features. Therefore, the trained fitting model can make the clothes fit the human body contours better and conform to the human body shape, making the fitting effect more realistic and accurate.

[0206] After training a virtual try-on model using the method provided in this application, the model can be applied to virtual try-on to generate a try-on image. The virtual try-on method provided in this application can be implemented by various types of electronic devices with computing capabilities, such as smart terminals and servers.

[0207] The virtual try-on method provided in this application is described below with reference to exemplary applications and implementations of the terminal provided in the embodiments of this application. See also Figure 8 , Figure 8 This is a flowchart illustrating the virtual try-on method provided in this application embodiment. Method S200 includes the following steps:

[0208] S201: Obtain the image of the person trying on the clothes, the image of the clothes to be tried on, and the text information of the person trying on the clothes.

[0209] A fitting assistant (application software) built into a terminal (such as a smartphone or smart fitting mirror) acquires an image of the person trying on the clothes, an image of the clothes to be tried on, and text information about the person's posture. The image of the person trying on can be taken by the terminal or entered by the user. The image of the clothes to be tried on can be selected by the user from the fitting assistant, and the text information about the person's posture can be entered by the user.

[0210] Understandably, the image of the wearer includes the wearer's body, the image of the clothes to be tried on includes the clothes, and the text information describing the wearer's posture describes the wearer's posture.

[0211] S202: Perform feature encoding on the textual information of the wearer's body posture to obtain the wearer's body posture feature map; and extract the wearer's torso image from the wearer's image.

[0212] It is understood that the try-on user's body posture text information may include text information such as height, weight, body shape, and build. This body posture text information is in text format. To enable the fitting model to understand the attribute characteristics reflected in the body posture text information, the body posture text information is encoded to obtain a body posture feature map. For specific encoding implementation methods, please refer to the encoding method in step S20 of the above-described training fitting model embodiment.

[0213] To preserve the wearer's identity features, a torso image is extracted from the wearer's image. For specific extraction implementation details, please refer to step S20 of the above-described training fitting model method embodiment.

[0214] S203: Input the body shape feature map and the torso map of the wearer into the fitting model to generate the fitting image.

[0215] The fitting model was trained using any of the methods mentioned above.

[0216] The terminal's built-in virtual try-on assistant includes a virtual try-on model. This model is used to virtually try on clothes and generate virtual try-on images. It is understood that this virtual try-on model is trained using the same method as the virtual try-on model training method described in the above embodiments, and has the same structure and function as the virtual try-on model described in the above embodiments; therefore, it will not be described in detail here.

[0217] This application also provides a computer-readable storage medium storing computer-executable instructions for causing an electronic device to execute the method for training a fitting model provided in this application, for example... Figure 3-7 The method for training the virtual fitting model shown, or the virtual fitting method provided in the embodiments of this application, for example, Figure 8 The virtual try-on method is shown.

[0218] In some embodiments, the storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0219] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0220] As an example, executable instructions may, but do not necessarily, correspond to files in the file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborative files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0221] As an example, executable instructions can be deployed to execute on a single computing device (including devices such as smart terminals and servers), or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0222] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method for training a fitting model or the virtual fitting method as described in the foregoing embodiments.

[0223] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0224] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0225] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for training a fitting model, characterized in that, The virtual fitting network includes an image generation network, a clothing deformation network, and a fusion module; The method includes: Acquire several image groups, each image group including clothing images and model images, wherein the model in the model image is wearing the clothing in the clothing image, and the model image corresponds to body posture text information; The body posture text information is feature-encoded to obtain a body posture feature map; and a torso image is extracted from the model image. The body feature map and the body torso map are input into the image generation network for encoding and decoding to obtain the model outline image; The model outline image and the clothing image are input into the clothing deformation network to deform the clothing in the clothing image, thereby obtaining a clothing deformation image; The deformed clothing image and the model silhouette image are input into the fusion module for fusion processing to obtain a pre-test clothing image; The loss function is used to calculate the loss between the pre-test clothing image and the model image, and the clothing try-on network is iteratively trained according to the loss sum corresponding to the several image groups until convergence, thus obtaining the clothing try-on model.

2. The method according to claim 1, characterized in that, The body posture text information corresponds to a body posture contour image, and the fitting network also includes a style transfer network, which is used to extract features from the input image and output multiple feature maps of different sizes. The method further includes: The style transfer network is used to extract features from the body contour image and the model contour image respectively, resulting in multiple body contour feature maps and multiple model contour feature maps; The step of calculating the loss between the pre-test garment image and the model image using a loss function includes: The style transfer loss between the plurality of body contour feature maps and the plurality of model contour feature maps is calculated using the loss function.

3. The method according to claim 2, characterized in that, The step of calculating the loss between the pre-test garment image and the model image using a loss function further includes: The loss function is used to calculate the clothing deformation loss between the clothing in the clothing deformation image and the clothing in the model image; The reconstruction loss between the model image and the pre-test garment image is calculated using the loss function.

4. The method according to claim 3, characterized in that, The loss function includes: Loss=λ0L style +λ1L warp +λ2L rec ; among which, L rec =||Y-R||1; Where λ0, λ1, and λ2 are hyperparameters, L style For the style transfer loss, L warp L represents the deformation loss of the clothing. rec Let H be the reconstruction loss, and H be the body contour image. The model's silhouette image is given, VGG is the style transfer network, where V is the total number of layers in the style transfer network, and F is the model's silhouette image. i and R i These are the activation count and the number of elements in the i-th layer of the style transfer network, respectively. The clothes in the model image, It is the clothing in the deformed image of the clothing. It is the outline of the clothing in the model image. Y is the outline of the clothing in the clothing deformation image, Y is the model image, and R is the pre-test clothing image.

5. The method according to any one of claims 1-4, characterized in that, The virtual fitting network also includes a multilayer sensor module; The step of encoding the body posture text information to obtain a body posture feature map includes: The bag-of-words model is used to encode each word in the body text information to obtain the text encoding; The multilayer perceptron module is used to extract features from the text encoding to obtain the body feature map.

6. The method according to any one of claims 1-4, characterized in that, The clothing deformation network includes a Transformer module, a feature matching module, and a deformation module; The model silhouette image and the clothing image are input into a clothing deformation network to deform the clothing in the clothing image, resulting in a deformed clothing image, including: The model outline image and the clothing image are input into the Transformer module to obtain the first model outline feature map and the first clothing feature map, respectively. The first model outline feature map and the first clothing feature map are input into the feature matching module for feature matching and association to obtain an association matrix. The association matrix reflects the pixel association and matching between the model and the clothing. The deformation module deforms the clothing image according to the correlation matrix to obtain the deformed clothing image.

7. The method according to claim 6, characterized in that, The virtual fitting network further includes a feature extraction module. Before the step of inputting the model outline image and the clothing image into the Transformer module to obtain the first model outline feature map and the first clothing feature map, respectively, the method further includes: The feature extraction module is used to extract features from the model contour image to obtain a second model contour feature map; The feature extraction module is used to extract features from the clothing image to obtain a second clothing feature map; The step of inputting the model outline image and the clothing image into the Transformer module to obtain the first model outline feature map and the first clothing feature map respectively includes: The second model outline feature map and the second clothing feature map are input into the Transformer module to obtain the first model outline feature map and the first clothing feature map, respectively.

8. A virtual try-on method, characterized in that, include: Acquire images of the person trying on the clothes, images of the clothes to be tried on, and textual information about the person's body posture; The body posture text information of the try-on user is feature-encoded to obtain the body posture feature map of the try-on user; And extract the torso image of the wearer from the wearer's image; The body feature map and the torso map of the wearer are input into the fitting model to generate a fitting image. The fitting model is trained using the method described in any one of claims 1-7.

9. An electronic device, characterized in that, include: At least one processor, and The memory communicatively connected to the at least one processor, wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer device to perform the method as described in any one of claims 1-8.

Images