Method for training a fitting model, virtual fitting method and related devices

Abstract

The embodiment of the application relates to the technical field of image processing, and discloses a method for training a fitting model, a virtual fitting method and related devices, clothes in each clothes image of a plurality of image groups belong to the same clothes category, clothes of the same category have the same characteristics (for example, the positions relative to a human body are approximately the same), which is conducive to fitting network learning fitting features of the clothes, thereby, the fitting network converges, the obtained fitting model is used for fitting the clothes of the same category, and the fitting effect is more real and natural. In addition, the clothes image is deformed, and the model image is preserved, so that the spliced image can preserve the identity features of the model without distortion, and the fitting clothes and the model are preliminarily attached. The obtained fitting model can make the fitting clothes conform to the human body posture, and the fitting effect is real and natural and has reference value.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method for training a virtual fitting model, a virtual fitting method, and related apparatus. Background Technology

[0002] With the continuous advancement of modern technology and the increasing scale of online shopping, users can purchase clothing on online shopping platforms via their mobile phones. However, since the information about the clothing for sale is generally presented as two-dimensional images, users cannot know how the clothing will look on them, which may lead to them buying clothes that are not suitable for them and resulting in a poor shopping experience.

[0003] Online virtual fitting typically involves taking a user's image, selecting a target garment provided by the system, and then automatically replacing it with the desired garment. Specifically, most of these methods involve collecting human body data and information about the garment to be tried on, and then using 3D modeling to reshape the user's image and the target garment. However, the combination of the target garment and the user's image is often not very accurate and lacks a natural feel. Summary of the Invention

[0004] The main technical problem solved by the embodiments of this application is to provide a method for training a fitting model, a virtual fitting method and related devices. The fitting model trained by this method can make the clothes fit the human body shape, the fitting effect is realistic and natural and has reference value, and the fitting process is simple.

[0005] In a first aspect, embodiments of this application provide a method for training a clothing try-on model, wherein the try-on network includes a clothing deformation network and an image generation network, comprising:

[0006] Acquire several image groups, each containing clothing images and model images. The model images are wearing the clothing from the clothing images, and the clothing in each clothing image of the several image groups belongs to the same clothing category.

[0007] Human body analysis is performed on the model image to obtain an analyzed image, and the preserved region image is obtained based on the analyzed image and the model image;

[0008] Human key point detection is performed on the model image to obtain the key point image;

[0009] A clothing deformation network is used to deform the clothing in a clothing image to obtain a deformed clothing image.

[0010] The image of the preserved region and the image of the deformed clothing are stitched together to obtain a stitched image. The stitched image, the key point image and the model image are then input into an image generation network to obtain a pre-test clothing image.

[0011] The loss function is used to calculate the loss between the pre-test clothing image and the model image. Based on the loss sum corresponding to several image groups, the clothing try-on network is iteratively trained until convergence, thus obtaining the clothing try-on model.

[0012] In some embodiments, the clothing deformation network includes a dual-channel feature extraction module, an optical flow module, and a deformation module.

[0013] The aforementioned method uses a clothing deformation network to deform clothing in a clothing image, resulting in a deformed clothing image, including:

[0014] The model image, clothing image, and key point image are input into a dual-channel feature extraction module for feature extraction to obtain the target feature map.

[0015] The target feature map is input into the optical flow module to calculate the target optical flow map;

[0016] The clothing image is deformed according to the target optical flow map to obtain a deformed clothing image.

[0017] In some embodiments, the optical flow module includes N flow networks, and the feature extraction module includes N convolutional layers. The N convolutional layers correspond one-to-one with the N flow networks. The feature map output by a convolutional layer is input into the corresponding flow network to obtain an optical flow map.

[0018] The loss function includes a second-order smoothing loss, which is used to constrain the collinearity of adjacent outflows in each optical flow map.

[0019] In some embodiments, the second-order smoothing loss includes:

[0020]

[0021] in, It is the t-th pixel on the i-th optical flow map, N t It is the set of horizontal, vertical, and two diagonal neighborhoods around the t-th pixel, and P is the generalized Charbonnier loss function.

[0022] In some embodiments, the image generation network includes an input convolutional layer, a U-NET module, and a fusion module;

[0023] The aforementioned input of the stitched image, key point image, and model image into the image generation network yields a pre-test garment image, including:

[0024] After combining the stitched image, key point image, and model image by channel, the images are input into the convolutional layer to obtain local feature maps.

[0025] The local feature map is input into the U-NET module for multi-scale feature learning to obtain a multi-scale feature map;

[0026] The local feature map and the multi-scale feature map are input into the fusion module for fusion processing to obtain the pre-test garment image.

[0027] In some embodiments, the U-NET module includes an encoder and a decoder, the encoder including multiple cascaded downsampled convolutional layers, the decoder including multiple cascaded upsampled convolutional layers, and cross-layer connections between downsampled and upsampled convolutional layers at the same level.

[0028] In some embodiments, the fusion module performs fusion processing in the following manner:

[0029] The corresponding pixel positions of the local feature map and the multi-scale feature map are added together to obtain the preliminary test garment image.

[0030] Secondly, embodiments of this application provide a virtual try-on method, including:

[0031] Obtain the image of the person trying on the clothes, the image of the clothes to be tried on, and the clothing category to which the clothes belong in the image of the clothes to be tried on;

[0032] The target fitting model is determined based on the clothing category. The target fitting model is trained using the method described in the first aspect above. The clothing category of the clothing to be tried on is the same as the clothing category of the clothing in each clothing image of several image groups.

[0033] Human body analysis is performed on the image of the wearer to obtain the analyzed image of the wearer, and the preserved region image of the wearer is obtained based on the analyzed image of the wearer and the image of the wearer;

[0034] Human key point detection is performed on the image of the try-on person to obtain the key point image of the try-on person;

[0035] Input the image of the user's reserved area, the image of the user's key points, the image of the user, and the image of the clothes to be tried on into the target clothing model to obtain the clothing image.

[0036] Thirdly, embodiments of this application provide an electronic device, including:

[0037] At least one processor, and

[0038] A memory that is communicatively connected to at least one processor, wherein,

[0039] The memory stores instructions that can be executed by at least one processor, such that the instructions are executed by at least one processor to enable the at least one processor to perform the method of the first aspect.

[0040] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer device to perform the method of the first aspect.

[0041] The beneficial effects of this application's embodiments are as follows: Unlike existing technologies, the training method, virtual fitting method, and related apparatus for fitting models provided in this application first construct a fitting network including a clothing deformation network and an image generation network. Several image groups are obtained, each including a clothing image and a model image. The model image wears the clothing from the clothing images, meaning the model image can serve as a real label. The clothing in each of these image groups belongs to the same clothing category (tops, bottoms, suits, or dresses, etc.). Then, human body analysis is performed on the model image to obtain an analyzed image, and a region-preserving image is obtained based on the analyzed image and the model image. Human body keypoint detection is performed on the model image to obtain a keypoint image. The clothing deformation network is used to deform the clothing in the clothing image to obtain a deformed clothing image. Finally, the region-preserving image and the deformed clothing image are stitched together to obtain a stitched image. The stitched image, keypoint image, and model image are input into the image generation network to obtain a preliminary fitting image. A loss function is used to calculate the loss between the preliminary fitting image and the model image, and the fitting network is iteratively trained based on the sum of losses corresponding to several image groups until convergence, resulting in a fitting model.

[0042] In this embodiment, the clothing images in several image groups belong to the same clothing category. Since clothing of the same category shares similar characteristics (e.g., roughly the same position relative to the human body), it facilitates the virtual try-on network learning the try-on features of this type of clothing. This, in turn, promotes network convergence, and the resulting virtual try-on model performs on clothing of the same category, resulting in a more realistic and natural try-on effect. Furthermore, by deforming the clothing images, the clothing in the deformed images takes on a three-dimensional form, adapting to the human body structure. By preserving the model image, the preserved region retains the model's identity features and other necessary characteristics, while obscuring the original clothing features that need to be replaced during the try-on process. Thus, the stitched image preserves the model's identity features without distortion and allows for a preliminary fit between the try-on clothing and the model. Finally, the stitched image, keypoint image, and model image are input into the image generation network to obtain a preliminary virtual try-on image. The keypoint image provides further guidance for clothing deformation, and the model image provides a reference for the torso skin features around the try-on clothing, making the preliminary virtual try-on image realistic and natural. The virtual try-on network is trained using several image sets in the manner described above. Based on backpropagation of the loss sum, the try-on effect in the pre-test images of each image set is constrained to continuously approach the real try-on effect in the model images, making the clothing deformation more consistent with the human body contour and posture features. Therefore, the trained virtual try-on model can make the clothes fit the human body shape, and the try-on effect is realistic, natural, and referential. It can be understood that this virtual try-on model is a general model that can try on clothes of the same category as those in the training data. Virtual try-on can be achieved simply by processing the images of the try-on user and the clothes to be tried on, without the need for modeling, and the try-on process is simple. Attached Figure Description

[0043] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0044] Figure 1 This is a schematic diagram of the structure of the virtual fitting system in some embodiments of this application;

[0045] Figure 2 This is a schematic diagram of the structure of an electronic device in some embodiments of this application;

[0046] Figure 3 This is a flowchart illustrating the method for training a fitting model in some embodiments of this application;

[0047] Figure 4 This is a schematic diagram of an image group in some embodiments of this application;

[0048] Figure 5 This is a schematic diagram of the reserved area image in some embodiments of this application;

[0049] Figure 6 These are key point images from some embodiments of this application;

[0050] Figure 7 These are stitched images in some embodiments of this application;

[0051] Figure 8 This is a schematic diagram of the structure of the image generation network in some embodiments of this application;

[0052] Figure 9 This is a schematic diagram showing the connection between the feature extraction module and the optical flow module in some embodiments of this application;

[0053] Figure 10 This is a schematic diagram of the structure of a portable network (PNG network) in some embodiments of this application;

[0054] Figure 11 This is a flowchart illustrating the virtual try-on method in some embodiments of this application. Detailed Implementation

[0055] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. In addition, the terms "first," "second," and "third" used herein do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0058] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0059] Furthermore, the technical features involved in the various embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0060] To facilitate understanding of the methods provided in the embodiments of this application, the terms used in the embodiments of this application will first be introduced:

[0061] (1) Neural Network

[0062] A neural network can be composed of neural units, specifically understood as a neural network with input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. Neural networks with many hidden layers are called deep neural networks (DNNs). The work of each layer in a neural network can be described by the mathematical expression y = a(W·x + b). From a physical perspective, the work of each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 2 and 3 are performed by "W·x", operation 4 by "+b", and operation 5 by "a()". The term "space" is used here because the objects being classified are not individual things, but a class of things; space refers to the set of all individuals within that class. W is the weight matrix of each layer in the neural network, where each value represents the weight of a neuron in that layer. This matrix W determines the spatial transformation from the input space to the output space, as described above; that is, the W of each layer of the neural network controls how the space is transformed. The purpose of training the neural network is to ultimately obtain the weight matrices of all layers of the trained neural network. Therefore, the training process of a neural network is essentially learning how to control spatial transformation, more specifically, learning the weight matrix.

[0063] It should be noted that, in the embodiments of this application, the models used for machine learning tasks are essentially neural networks. Common components in neural networks include convolutional layers, pooling layers, normalization layers, and deconvolutional layers. By assembling these common components in neural networks, a model is designed. When the model parameters (weight matrices of each layer) are determined such that the model error meets a preset condition or the number of model parameters is adjusted to reach a preset threshold, the model converges.

[0064] The convolutional layer is configured with multiple convolutional kernels, each with a corresponding stride, to perform convolution operations on the image. The purpose of convolution is to extract different features from the input image. The first convolutional layer may only extract some low-level features such as edges, lines, and corners, while deeper convolutional layers can iteratively extract more complex features from low-level features.

[0065] A deconvolutional layer is used to map a low-dimensional space to a high-dimensional space while preserving the connections / patterns between them (the connections during convolution). A deconvolutional layer is configured with multiple convolutional kernels, each with a corresponding stride, to perform deconvolution operations on the image. Generally, framework libraries used for designing neural networks (such as the PyTorch library) have a built-in `upsumple()` function, which allows for low-dimensional to high-dimensional spatial mapping.

[0066] Pooling layers mimic the human visual system, reducing the dimensionality of data and representing images with higher-level features. Common pooling operations include max pooling, mean pooling, random pooling, median pooling, and combined pooling. Typically, pooling layers are periodically inserted between convolutional layers in neural networks to achieve dimensionality reduction.

[0067] The normalization layer is used to normalize all neurons in the intermediate layer to prevent gradient explosion and gradient vanishing.

[0068] (2) Loss Function

[0069] During neural network training, to ensure the output closely approximates the desired predicted value, we compare the network's current prediction with the target value. Based on the difference, we update the weight matrix of each layer (usually an initialization process before the first update, pre-configuring parameters for each layer). For example, if the network's prediction is too high, the weight matrix is ​​adjusted to predict a lower value, and this adjustment continues until the neural network accurately predicts the target value. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the neural network becomes a process of minimizing this loss.

[0070] (3) Human body analysis

[0071] Human body analysis refers to segmenting a person captured in an image into multiple semantically consistent regions, such as body parts and clothing, or further subcategories of body parts and clothing. Essentially, it involves pixel-level recognition of the input image and labeling each pixel with its corresponding object category. For example, neural networks can be used to distinguish various elements (including hair, face, limbs, clothing, and background) in an image containing a human body.

[0072] Before introducing the embodiments of this application, a brief introduction will be given to the virtual try-on method known to the inventors of this application, so as to facilitate the understanding of the embodiments of this application later.

[0073] In some solutions known to the inventors of this application, the process involves acquiring a captured facial image; matching the captured facial image with a pre-stored facial image; when a match is successful, acquiring a virtual fitting model corresponding to the pre-stored facial image; and performing a virtual fitting operation based on the virtual fitting model. This solution uses a 3D model for fitting, which has high 3D reconstruction costs. When the person's body shape changes, it needs to be reconstructed again, resulting in low reusability and usability, and making it impossible to perform virtual fittings instantly. Furthermore, the model reconstructed from the 3D model lacks realism, making the virtual fitting experience unrealistic.

[0074] In some known solutions, a real-life image of the user is obtained by photographing the user, and a virtual image containing an avatar corresponding to the user is obtained based on the user's body shape and movement. The real-life and virtual images are analyzed to determine the composite regions corresponding to the real-life and virtual images, respectively, and then the composite regions of the real-life and virtual images are combined to output a virtual try-on image. This method uses image synthesis to perform virtual try-on, compositing the user's appearance with the generated avatar. The synthesized image lacks realism, and the user cannot accurately reference the effect. Furthermore, due to the influence of the shooting environment and differences in the user's face, there are issues with unnatural blending. Additionally, this method generates an avatar with the same body shape as the user, which may not accurately match some of the user's body shape and facial details, resulting in significant deviations in the try-on effect.

[0075] To address the aforementioned issues, this application provides a method for training a virtual fitting model, a virtual fitting method, and related apparatus. A fitting network comprising a clothing deformation network and an image generation network is constructed. The fitting network is trained using stitched images corresponding to several image groups, keypoint images, and model images to obtain the fitting model. The clothing in each image group belongs to the same clothing category. Clothing of the same category shares similar characteristics (e.g., roughly the same position relative to the human body), which facilitates the fitting network learning the fitting features of this type of clothing. This, in turn, promotes network convergence, and the resulting fitting model allows for fitting of clothing of the same category, resulting in a more realistic and natural fitting effect. Furthermore, by deforming the clothing images, the clothing in the deformed images takes on a three-dimensional form, adapting to the human body structure. By preserving the model image, the preserved region retains the model's identity features and other necessary characteristics, while obscuring the original clothing features that need to be replaced during the fitting process. Therefore, the stitched image preserves the model's identity features without distortion and allows for an initial fit between the fitting clothes and the model. Finally, the stitched image, keypoint image, and model image are input into the image generation network to obtain the pre-test clothing image. The keypoint image provides further guidance for clothing deformation during the try-on process, while the model image provides a reference for the torso skin features surrounding the clothing, making the pre-test clothing image realistic and natural. Several image groups are used to train the try-on network in the above manner. Based on backpropagation of the loss sum, the try-on effect in the corresponding pre-test clothing images of each image group is constrained to continuously approach the real try-on effect in the model image, making the clothing deformation more consistent with human contour and body shape features. Therefore, the trained try-on model can make the try-on clothing conform to the human body shape, and the try-on effect is realistic, natural, and referential. It can be understood that this try-on model is a general model that can try on clothing of the same category as the clothing in the training data. Virtual try-on can be achieved simply by processing the try-on image and the clothing to be tried on, without the need for modeling, and the try-on process is simple.

[0076] The following describes an exemplary application of the electronic device provided in the embodiments of this application for training a fitting model or for virtual fitting. It is understood that the electronic device can both train a fitting model and use the fitting model to perform virtual fitting and generate fitting images.

[0077] The electronic device provided in this application embodiment can be a server, such as a server deployed in the cloud. When the server is used to train a virtual fitting model, it iteratively trains the virtual fitting network using the training set provided by other devices or those skilled in the art, determines the final model parameters, and then configures the virtual fitting network with these final model parameters to obtain the virtual fitting model. The training set includes several image groups, each including a clothing image and a model image. The model in the model image wears the clothing in the clothing image. The clothing in each of the several image groups belongs to the same clothing category, such as tops, bottoms, suits, or dresses. When the server is used for virtual fitting, it calls the built-in virtual fitting model and performs corresponding calculations on the images of the wearer and the clothing to be tried on provided by other devices or users to generate virtual fitting images that conform to the human body contour and posture.

[0078] The electronic devices provided in some embodiments of this application can be various types of terminals such as laptops, desktop computers, or mobile devices. When the terminal is used to train a virtual fitting model, those skilled in the art input a prepared training set into the terminal and design a virtual fitting network on the terminal. The terminal uses the training set to iteratively train the virtual fitting network to determine the final model parameters. The virtual fitting network is then configured with these final model parameters to obtain the virtual fitting model. When the terminal is used for virtual fitting, it calls the built-in virtual fitting model and performs corresponding calculations on the user-input image of the wearer and the image of the clothing to be tried on, generating a virtual fitting image with a realistic and natural effect.

[0079] As an example, see Figure 1 , Figure 1 This is a schematic diagram of the application scenario of the fitting system provided in the embodiment of this application. The terminal 10 is connected to the server 20 through a network, wherein the network can be a wide area network or a local area network, or a combination of the two.

[0080] Terminal 10 can be used to acquire training sets and construct a virtual fitting network. For example, those skilled in the art can download a prepared training set on the terminal and construct the network structure of the virtual fitting network. It is understood that terminal 10 can also be used to acquire images of the user and the clothes to be tried on. For example, the user inputs the user image and the clothes image through an input interface; after input, the terminal automatically acquires the user image and the clothes image. Alternatively, terminal 10 may have a camera to capture user images, and terminal 10 may have a clothing image library that can be used to select images of clothes to be tried on.

[0081] In some embodiments, terminal 10 locally executes the method for training a fitting model provided in this application to train the designed fitting network using a training set, determine the final model parameters, and then configure the fitting network with the final model parameters to obtain a fitting model. In some embodiments, terminal 10 may also send a training set and a constructed fitting network stored on the terminal by a person skilled in the art to server 20 via a network. Server 20 receives the training set and the fitting network, trains the designed fitting network using the training set, determines the final model parameters, and then sends the final model parameters to terminal 10. Terminal 10 saves the final model parameters so that the fitting network can be configured with the final model parameters to obtain a fitting model.

[0082] In some embodiments, terminal 10 locally executes the virtual try-on method provided in this application embodiment to provide virtual try-on services to users. It invokes a built-in try-on model to perform corresponding calculations on the user-inputted image of the wearer and the image of the garment to be tried, generating a try-on image with a realistic and natural effect. In some embodiments, terminal 10 can also send the user-inputted image of the wearer and the image of the garment to be tried to server 20 via a network. Upon receiving the images, server 20 invokes its built-in try-on model to perform corresponding calculations on the images of the wearer and the garment to be tried, generating a try-on image with a realistic and natural effect, and then sends the try-on image to terminal 10. After receiving the try-on image, terminal 10 displays the try-on image on its own display interface for the user to view the try-on effect.

[0083] The structure of the electronic device in the embodiments of this application is described below. Figure 2This is a schematic diagram of the structure of an electronic device 500 in an embodiment of this application. The electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It is understood that the bus system 540 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 2 The general labeled all buses as Bus System 540.

[0084] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor.

[0085] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0086] Memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory. Memory 550 may optionally include one or more storage devices physically located away from processor 510.

[0087] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0088] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks.

[0089] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0090] Display module 553 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with user interface 530;

[0091] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0092] As can be understood from the above, the method for training a virtual fitting model and the virtual fitting method provided in the embodiments of this application can be implemented by various types of electronic devices with computing power, such as smart terminals and servers.

[0093] The following describes the method for training a virtual fitting model provided in this application, using exemplary applications and implementations of the server provided in the embodiments of this application. See also... Figure 3 , Figure 3 This is a flowchart illustrating the method for training a fitting model provided in this application embodiment. The fitting model network, which serves as the network structure of the fitting model, includes a clothing deformation network and an image generation network. Both the clothing deformation network and the image generation network are pre-set neural networks with components of the neural network (convolutional layers, deconvolutional layers, or pooling layers, etc.). The basic structure and principles of the neural network have been described in detail in "Glossary (1)" and will not be described in detail here. The fitting network can be constructed by those skilled in the art on a neural network design platform on a terminal (e.g., a computer) and then sent to a server.

[0094] In some embodiments, the intra-layer structure (convolutional kernel, stride, etc.), inter-layer connection structure, and layer combination of convolutional layers, deconvolutional layers, or pooling layers in the image generation network can be configured to obtain a specific image generation network. The image generation network includes an encoder and a decoder. Those skilled in the art will understand that the encoder includes multiple downsampling convolutional layers; in the encoder, as the downsampling convolutional layers progress, the size of the output feature map decreases. The decoder includes multiple upsampling convolutional layers; in the decoder, as the upsampling convolutional layers progress, the size of the output feature map increases. Those skilled in the art can configure parameters such as the kernel size and stride of each downsampling and upsampling convolutional layer according to actual needs. Similarly, the structure of a clothing deformation network can also be configured so that the clothing deformation network is used to deform clothing.

[0095] Please refer to it again. Figure 3 The method S100 may specifically include the following steps:

[0096] S10: Obtain several image groups.

[0097] For any given image set, including clothing images and model images, the model in the model images is wearing the clothing in the clothing images. The clothing images include clothes that the model wants to try on, such as... Figure 4 As shown, for example, a clothing image includes a pair of denim shorts. A model image shows a model wearing the clothing corresponding to the clothes in the clothing image, such as... Figure 4 As shown, the model in the image corresponding to the clothing image is wearing the denim shorts.

[0098] The clothing images in several image groups belong to the same clothing category, such as tops, bottoms, suits, or dresses. When the clothing images in these image groups are all pants, the trained clothing model can try on pants; when the clothing images in these image groups are all shirts, the trained clothing model can try on shirts.

[0099] It is understood that those skilled in the art can set the clothing category of each clothing image in several image groups. It is also understood that these image groups can be collected in advance by those skilled in the art on a terminal (e.g., a computer), for example, by crawling clothing images and corresponding images of models wearing those clothes from clothing sales websites. After preparing several image groups, this training data is uploaded to the server via the terminal.

[0100] In some embodiments, the number of image groups is in the tens of thousands, for example, 20,000, which is beneficial for training an accurate general model. Those skilled in the art can determine the number of image groups according to the actual situation.

[0101] S20: Perform human body analysis on the model image to obtain the analyzed image, and obtain the preserved region image based on the analyzed image and the model image.

[0102] Understandably, when changing clothes for a model in an image, it's necessary to preserve essential features such as the model's identity. For example, when trying on pants, the upper body clothing and torso must be retained. Figure 5 As shown, Figure 5 This is a schematic diagram of a preserved region image. Before the clothing and the model are combined for a try-on, the model's preserved features are extracted, i.e., a preserved region image is obtained. On the one hand, this avoids interference from the original features of the old clothing on the combined try-on; on the other hand, it preserves the model's identity features, ensuring that the model does not lose authenticity when wearing the clothes to be tried on.

[0103] Specifically, firstly, the model image can be analyzed to obtain an analyzed image. As can be seen from the aforementioned "Glossary (3)", human body analysis involves segmenting the various parts of the human body and assigning categories to the pixels.

[0104] In some embodiments, the human body segmentation algorithm can be the existing Graphomanomy algorithm. The Graphomanomy algorithm segments the image into 20 categories, which can be distinguished by different colors, classifying each body part. In some embodiments, the 20 categories can also be classified using numbers 0-19, for example, 0 represents background, 1 represents hat, 2 represents hair, 3 represents gloves, 4 represents sunglasses, 5 represents top, 6 represents dress, 7 represents coat, 8 represents socks, 9 represents pants, 10 represents torso skin, 11 represents scarf, 12 represents skirt, 13 represents face, 14 represents left arm, 15 represents right arm, 16 represents left leg, 17 represents right leg, 18 represents left shoe, and 19 represents right shoe. From the segmented graph, the category to which each body part belongs in the image can be determined.

[0105] Then, the body torso and other regions (such as face, neck, hands, feet, and upper clothing) in the parsed image are preserved. The pixel category corresponding to the preserved region is set to 1, and the regions of other categories are set to 0, thus obtaining the binarized parsed image.

[0106] Finally, the preserved region image is obtained based on the analyzed image and the model image. Specifically, the corresponding pixel positions in the model image and the corresponding pixel positions in the binarized analyzed image are multiplied to obtain the preserved region image.

[0107] The product of corresponding positions can be expressed using the formula F′. ij =F ij ×M ij Explain, where F ij M represents the pixel value in the i-th row and j-th column of the model image.ij F′ represents the value in the i-th row and j-th column of the binarized analytical image. ij To preserve the pixel value in the i-th row and j-th column of the region image.

[0108] In this embodiment, the preserved area image can accurately retain the model's identity features and remove the original clothing features that need to be replaced, so that the model's identity is not distorted after changing into the clothes to be tried on.

[0109] S30: Perform human body key point detection on the model image to obtain a key point image.

[0110] Human keypoint detection algorithms are used to detect human keypoints in model images, which can locate keypoint information (i.e., several key points on the human body), such as... Figure 6 As shown, key points can be coordinates of the nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. In some embodiments, the human key point detection algorithm can use the OpenPose algorithm for detection. In some embodiments, the human key point detection algorithm can use a 2D key point detection algorithm, such as Convolutional Pose Machine (CPM) or Stacked Hourglass Network (Hourglass).

[0111] Understandably, each keypoint has its own index, and the position of each keypoint is represented by coordinates. The OpenPose algorithm originally defined 18 keypoints, with the index representing human joints. The indexes are: 0 (nose), 1 (neck), 2 (right shoulder), 3 (right elbow), 4 (right wrist), 5 (left shoulder), 6 (left elbow), 7 (left wrist), 8 (right hip), 9 (right knee), 10 (right ankle), 11 (left hip), 12 (left knee), 13 (left ankle), 14 (right eye), 15 (left eye), 16 (right ear), and 17 (left ear). For different human bodies, the OpenPose algorithm will detect these 18 keypoints. Due to the different body shapes of the models, the coordinates of these 18 keypoints are also different for each model.

[0112] Therefore, human keypoint detection is performed on the model image to obtain a keypoint image. The keypoint image includes multiple keypoints, each of which includes a sequence number and its corresponding coordinates; the sequence number represents a human joint.

[0113] S40: Use a clothing deformation network to deform the clothing in the clothing image to obtain a deformed clothing image.

[0114] Among them, the clothing deformation network is used to deform clothing. It is understood that the clothing deformation network is also a type of neural network, and those skilled in the art can design the specific structure of the clothing deformation network according to actual needs.

[0115] In some embodiments, the clothing image, keypoint image, and model image are concatenated through channels and then input into a clothing deformation network. The layers or components of the clothing deformation network (e.g., convolutional layers, pooling layers, activation function layers, or residual extraction modules) perform spatial mapping transformations on the concatenated image to obtain a deformed clothing image. The model image and keypoint image provide the clothing deformation network with the model's body contour and structure, guiding the network to deform the clothing towards the model's body contour and structure, making it approximate the body contour at the corresponding location. Thus, in the deformed clothing image, the clothing has a three-dimensional shape that conforms to human body characteristics.

[0116] In some embodiments, the clothing deformation network includes a dual-channel feature extraction module, an optical flow module, and a deformation module. The feature extraction module extracts features from the input image to obtain a feature map. The optical flow module calculates an output optical flow map based on the input feature map. Each point in the optical flow map is a two-dimensional vector, recording the displacement of each pixel in the clothing image in the x and y directions. The deformation module performs deformation processing on the image based on the optical flow map.

[0117] The aforementioned step S40 includes: inputting the model image, clothing image, and key point image into a dual-channel feature extraction module for feature extraction to obtain a target feature map; inputting the target feature map into an optical flow module to calculate a target optical flow map; and deforming the clothing image according to the target optical flow map to obtain a deformed clothing image.

[0118] In this embodiment, the model image, clothing image, and key point image are concatenated by channel and then input into a dual-channel feature extraction module for feature extraction. The dual-channel feature extraction module consists of two separate feature extraction modules. These two modules can have the same convolutional layer structure but different parameter configurations. In some embodiments, the feature extraction module can have a pyramid network structure, meaning the resolution of the output feature maps decreases progressively.

[0119] From multiple feature maps of different resolutions output by the feature extraction module, one is selected as the target feature map. The two target feature maps are then input into the optical flow module to calculate the target optical flow map. The optical flow module utilizes the principle of optical flow, which frequently appears in video or image sequences (multiple images), to characterize the instantaneous motion transitions (direction and offset) of a moving object (camera or observed object). The magnitude of the offset of each pixel in the image is represented by the (x, y) value of the corresponding pixel in the optical flow map, while the direction of the offset is indicated by the positive or negative sign of the (x, y) value of the corresponding pixel in the optical flow map.

[0120] In the x-direction, positive values ​​indicate that the pixel moves to the left, while negative values ​​indicate that the pixel moves to the right; in the y-direction, positive values ​​indicate that the pixel moves upward, while negative values ​​indicate that the pixel moves downward. Here, the target optical flow map can describe the state change from the original clothing image to the deformed clothing image (trying on a person).

[0121] In other words, in the optical flow image, each pixel in the clothing image has a corresponding direction of motion and offset to describe its deformation. Therefore, the deformation module can use the direction and offset of each pixel in the target optical flow image to guide the deformation of the original clothing image, resulting in a deformed clothing image.

[0122] In this embodiment, the clothing deformation network uses optical flow deformation to deform the original clothing image, which enables the deformed clothing outline to adapt to the human body and the shape to conform to human characteristics.

[0123] In some embodiments, the densepose image corresponding to the model image is also stitched together with the model image, clothing image, and keypoint image, and then input into the optical flow module to calculate the optical flow map. The densepose image reflects the model's pose, thus contributing to the generation of a more accurate optical flow map.

[0124] S50: The preserved region image and the clothing deformation image are stitched together to obtain a stitched image. The stitched image, key point image and model image are then input into the image generation network to obtain a pre-test clothing image.

[0125] The preserved region image and the distorted clothing image are pixel-wise stitched together to obtain a stitched image. For example, the distorted pants are pixel-wise stitched together with the preserved region image (including the upper body and shoes) to obtain an image like this. Figure 7 The stitched image shown lacks visible leg features.

[0126] Then, after channel stitching of the stitched image, key point image, and model image, the data is input into the image generation network for downsampling encoding followed by upsampling decoding. The features of the stitched image, key point image, and model image are fused to generate missing torso features (such as the aforementioned leg features) in the stitched image. The shape of the try-on clothes (such as pants) is adjusted so that the try-on clothes in the generated pre-test clothing image fit the human body contour better and can also retain the model's human body features without distortion.

[0127] In some embodiments, the image generation network includes an input convolutional layer, a U-NET module, and a fusion module. The input convolutional layer extracts features from the input image, the U-NET module encodes and then decodes the input image, and the fusion module fuses features from two input feature maps.

[0128] The aforementioned step S50 includes: combining the stitched image, key point image, and model image into channels, and then inputting them into the input convolutional layer to obtain a local feature map; inputting the local feature map into the U-NET module for multi-scale feature learning to obtain a multi-scale feature map; and inputting the local feature map and the multi-scale feature map into the fusion module for fusion processing to obtain a pre-test garment image.

[0129] Please see Figure 8 After combining the stitched image, key point image, and model image by channels, we obtain H×W×C. in A stitched image x of size is input, and after feature extraction by the input convolutional layer, a local feature map F1(x) is obtained. It can be understood that the input convolutional layer can be a regular convolutional layer used for extracting local features. The local feature map F1(x) can restore the color and texture of the skin.

[0130] Then, the local feature map F1(x) is input into the U-NET module for encoding and decoding. Multi-scale feature learning is performed during the encoding and decoding process, and multi-scale feature map U(F1(x)) is output.

[0131] In some embodiments, the aforementioned U-NET module includes an encoder and a decoder, the encoder including multiple cascaded downsampled convolutional layers and the decoder including multiple cascaded upsampled convolutional layers, with cross-layer connections between downsampled and upsampled convolutional layers at the same level.

[0132] Please refer to it again. Figure 8 L is the number of encoder or decoder layers ( Figure 8 (Using L=7 as an example for illustrative purposes, C) in Indicates the number of input channels, C out This indicates the output channel, and M represents the number of channels in the convolutional layer inside the encoder or decoder.

[0133] Understandably, the larger L is, the deeper the network, the more pooling operations, the larger the receptive field, and the richer the local and global features the network can extract. Configuring this parameter can extract multi-scale features from local feature maps F1(x) with arbitrary spatial resolution.

[0134] In this embodiment, the upsampling convolutional layers of the encoder correspond one-to-one with the downsampling convolutional layers of the decoder, and there are cross-layer connections between downsampling and upsampling convolutional layers at the same level. For example, the downsampling feature map output by the i-th downsampling convolutional layer is input to the (i+1)-th downsampling convolutional layer for downsampling, and also input to the i-th upsampling convolutional layer which is connected across layers to the i-th downsampling convolutional layer. This is then fused with the upsampling feature map output by the (i-1)-th upsampling convolutional layer above, which can reduce feature loss.

[0135] The encoder extracts multi-scale features from the gradient downsampled feature map, and the decoder decodes it into a high-resolution feature map through progressive upsampling, fusion, and convolution. This process reduces the loss of detailed features in the downsampled feature map during the progressive upsampling process.

[0136] Finally, the multi-scale feature map U(F1(x)) and the local feature map F1(x) output by the decoder are fused in the fusion module to output H×W×C. out Preview image of the size of the test garment.

[0137] In some embodiments, the aforementioned fusion module performs fusion processing in the following manner: adding the corresponding pixel positions of the local feature map and the multi-scale feature map to obtain the pre-test garment image.

[0138] Here, the corresponding pixel positions are added together, which can be done using the formula Y′. ij =F1(x) ij ×U(F1(x)) ij Explain, where F1(x) ij Let U(F1(x)) represent the pixel value in the i-th row and j-th column of the local feature map. ij Y′ represents the value in the i-th row and j-th column of the global feature map. ij Let be the pixel value in the i-th row and j-th column of the test garment image.

[0139] As can be seen, the fusion module can effectively fuse local and global feature maps through the above-mentioned addition and fusion method, making the pre-test clothing image more realistic and without distortion.

[0140] In this embodiment, an image generation network with the above-described structure is used to simulate the generation of human skin regions. The U-NET module has excellent capture capabilities for both local and global features. The local minimum receptive field is used to mimic skin details, and the global filter is used to extract human torso contour features. Thus, it is possible to generate accurate images in stitched images that lack torso features (such as the aforementioned leg features). Furthermore, it can mimic the realistic try-on effect in model images by adjusting the shape of the clothes (such as pants) to make the clothes in the generated pre-test clothing images fit the human body contour more closely, while also preserving the model's human features without distortion.

[0141] S60: The loss function is used to calculate the loss between the pre-test clothing image and the model image. Based on the loss sum corresponding to several image groups, the clothing try-on network is iteratively trained until convergence, and the clothing try-on model is obtained.

[0142] Here, the loss function can be configured by those skilled in the art in the terminal. The configured loss function is sent to the server along with the virtual try-on network. After the server processes and obtains the pre-test images corresponding to each image group, it uses the loss function to calculate the loss between each model image and the pre-test image. Based on the loss, the virtual try-on network is iteratively trained until the virtual try-on network converges, and the virtual try-on model is obtained.

[0143] Understandably, the smaller the difference between each model image and the pre-fitting garment image, the more similar the model image and the pre-fitting garment image are, indicating that the pre-fitting garment image can accurately reproduce the real fitting effect. Therefore, the model parameters of the aforementioned fitting network can be adjusted based on the differences between each model image and the pre-fitting garment image, and the fitting network can be iteratively trained. That is, the differences are backpropagated, causing the pre-fitting garment image output by the fitting network to continuously approximate the model image until the fitting network converges, resulting in a fitting model. Understandably, in some embodiments, the fitting network includes a clothing deformation network and an image generation network; the model parameters then include the model parameters of the clothing deformation network and the image generation network, thereby achieving end-to-end training.

[0144] In some embodiments, the Adam algorithm is used to optimize the model parameters. For example, the number of iterations is set to 100,000, the initial learning rate is set to 0.001, the learning rate weight decay is set to 0.0005, and the learning rate decays to 1 / 10 of its original value every 1,000 iterations. The learning rate, loss, and input can be fed into the Adam algorithm to obtain the adjusted model parameters output by the Adam algorithm. The adjusted model parameters are used for the next training until the training is completed. Then, the model parameters of the converged virtual fitting network are output, which is the virtual fitting model.

[0145] Understandably, after the server obtains the model parameters (i.e., the final model parameters) of the converged virtual try-on network, it can send these final model parameters to the terminal. The virtual try-on network within the terminal is configured with these final model parameters, thus obtaining the virtual try-on model. In some embodiments, the server can also save the virtual try-on network and the final model parameters to obtain the virtual try-on model.

[0146] In this embodiment, a virtual fitting network is constructed, comprising a clothing deformation network and an image generation network. The network is trained using stitched images, keypoint images, and model images corresponding to several image groups to obtain a virtual fitting model. The clothing in each image group belongs to the same clothing category. Clothing of the same category shares similar characteristics (e.g., roughly the same position relative to the human body), which helps the virtual fitting network learn the fitting features of this type of clothing. This facilitates network convergence, and the resulting virtual fitting model performs fitting on clothing of the same category, resulting in a more realistic and natural fitting effect. Furthermore, clothing image deformation makes the clothing in the deformed image appear three-dimensional, adapting to the human body structure. By preserving the model image, the preserved region retains the model's identity features and other necessary characteristics, while obscuring the original clothing features that need to be replaced during the fitting process. Therefore, the stitched image preserves the model's identity features without distortion and allows for an initial fit between the fitting clothes and the model. Finally, the stitched image, keypoint image, and model image are input into the image generation network to obtain the pre-test clothing image. The keypoint image provides further guidance for clothing deformation during the try-on process, while the model image provides a reference for the torso skin features surrounding the clothing, making the pre-test clothing image realistic and natural. Several image groups are used to train the try-on network in the above manner. Based on backpropagation of the loss sum, the try-on effect in the corresponding pre-test clothing images of each image group is constrained to continuously approach the real try-on effect in the model image, making the clothing deformation more consistent with human contour and body shape features. Therefore, the trained try-on model can make the try-on clothing conform to the human body shape, and the try-on effect is realistic, natural, and referential. It can be understood that this try-on model is a general model that can try on clothing of the same category as the clothing in the training data. Virtual try-on can be achieved simply by processing the try-on image and the clothing to be tried on, without the need for modeling, and the try-on process is simple.

[0147] In some embodiments, please refer to Figure 9 The optical flow module includes N flow networks, and the feature extraction module includes N convolutional layers. The N convolutional layers correspond one-to-one with the N flow networks. The feature map output by a convolutional layer is input into the corresponding flow network to obtain an optical flow map.

[0148] It is understandable that the N convolutional layers of the feature extraction module form a pyramid network structure, and the resolution of the N output feature maps f1 to fn gradually decreases. The feature map f1 with the highest resolution is input into the FN-1 flow network, the feature map f2 is input into the FN-2 flow network, and so on, with the feature map fn being input into the FN-n flow network.

[0149] By setting up an optical flow module with the above structure, optical flow maps of different granularities can be generated, that is, optical flow maps with different resolutions.

[0150] In this embodiment, the loss function includes a second-order smoothing loss, which is used to constrain the collinearity of adjacent outflows in each optical flow graph.

[0151] Understandably, clothing often contains closely spaced text and repeating patterns (e.g., stripes). The appearance flow between the person image and the clothing image needs to be accurately predicted; otherwise, subtle differences can lead to very unnatural graphic pattern distortions. To better preserve clothing features, a second-order smoothing loss is introduced to encourage collinearity of adjacent external optical flows in the optical flow map. Collinearity of adjacent appearance flows refers to small displacement differences reflected by adjacent pixels in the optical flow map, i.e., small displacement differences between adjacent pixels in the clothing image.

[0152] In some embodiments, the aforementioned second-order smoothing loss includes:

[0153]

[0154] in, It is the t-th pixel on the i-th optical flow map, N t It is the set of horizontal, vertical, and two diagonal neighborhoods around the t-th pixel, and P is the generalized Charbonnier loss function.

[0155] In this embodiment, by setting the aforementioned second-order smoothing loss, the displacement differences reflected by pixels within a range of π in the optical flow map at various resolutions can be calculated. Backpropagation based on the second-order smoothing loss can constrain the collinearity of adjacent appearance flows in the optical flow maps generated by the optical flow module at all granularities. Therefore, it is beneficial for pre-test clothing images to retain detailed clothing features, resulting in a natural and realistic appearance.

[0156] In summary, the method for training a fitting model provided in this application constructs a fitting network comprising a clothing deformation network and an image generation network. The fitting network is trained using stitched images, keypoint images, and model images corresponding to several image groups to obtain the fitting model. The clothing in each image group belongs to the same clothing category. Since clothing of the same category shares similar characteristics (e.g., roughly the same position relative to the human body), it facilitates the fitting network learning the fitting features of this type of clothing, thus promoting network convergence. The resulting fitting model performs fitting on clothing of the same category, resulting in a more realistic and natural fitting effect. Furthermore, by deforming the clothing images, the clothing in the deformed images takes on a three-dimensional form, adapting to the human body structure. By preserving the model images, the preserved region retains the model's identity features and other necessary characteristics, while obscuring the original clothing features that need to be replaced during the fitting process. Therefore, the stitched image preserves the model's identity features without distortion and allows for an initial fit between the fitting clothes and the model. Finally, the stitched image, keypoint image, and model image are input into the image generation network to obtain the pre-test clothing image. The keypoint image provides further guidance for clothing deformation during the try-on process, while the model image provides a reference for the torso skin features surrounding the clothing, making the pre-test clothing image realistic and natural. Several image groups are used to train the try-on network in the above manner. Based on backpropagation of the loss sum, the try-on effect in the corresponding pre-test clothing images of each image group is constrained to continuously approach the real try-on effect in the model image, making the clothing deformation more consistent with human contour and body shape features. Therefore, the trained try-on model can make the try-on clothing conform to the human body shape, and the try-on effect is realistic, natural, and referential. It can be understood that this try-on model is a general model that can try on clothing of the same category as the clothing in the training data. Virtual try-on can be achieved simply by processing the try-on image and the clothing to be tried on, without the need for modeling, and the try-on process is simple.

[0157] In some embodiments, training data for different clothing categories can be prepared to train a fitting model for a specific clothing category. For example, training data for clothing categories such as tops, bottoms, suits, or dresses can be collected separately, and the fitting model can be trained using any of the above methods. This will yield fitting models for tops, bottoms, suits, and dresses, respectively.

[0158] In some embodiments, a human semantic parsing model is trained using Portable Network Graphics (PNG) to classify the human body into 20 different semantic types, thereby allowing clothing to be extracted and categorized.

[0159] Please see Figure 10For the input human image, a ResNet network is first used for feature extraction. Then, feature maps of multiple resolutions are input into two branches to capture partial regions and human boundary regions, simultaneously generating a segmentation map and a human boundary map. The segmentation map includes the category of each pixel. The human boundary map reflects the completeness of the human edge and is calculated by subtracting it from the ground truth value; Euclidean distance or other similar subtraction methods can be used.

[0160] Finally, a refinement branch is executed, which refines the predicted segmentation map and edge map by integrating the segmentation map and the human body boundary map.

[0161] In this embodiment, the PGN network redefines instance-level human parsing as two twin sub-tasks that can be learned and improved by a unified network: 1) semantic segmentation of each pixel as a human part (such as face, arm); 2) instance-aware edge detection, assigning semantic parts to different human instances. Therefore, the shared intermediate representation can not only represent fine-grained parts, but also infer which instance each part belongs to.

[0162] Therefore, the human semantic parsing model trained by the PNG network can be used as part of a virtual fitting system to extract the top, bottom, or background image as needed.

[0163] Understandably, after extracting the top or bottom garment from the input image, it can be recorded and saved to the clothing image library, so that other people can try on the saved top or bottom garment, thus realizing the function of trying on clothes worn by others.

[0164] Understandably, after extracting the background of the input image, it can be recorded and saved to the background image library, which can then be used by others when virtual try-on.

[0165] In this embodiment, the virtual try-on model is combined with a human semantic parsing model (cutting out clothes and background), which not only improves the realism of the virtual try-on effect but also reduces the cost of the entire try-on process. While ensuring the experience and effect, it also provides a variety of auxiliary functions to improve the overall virtual try-on experience.

[0166] After training a virtual try-on model using the method provided in this application, the model can be applied to virtual try-on to generate a try-on image. The virtual try-on method provided in this application can be implemented by various types of electronic devices with computing capabilities, such as smart terminals and servers.

[0167] The virtual try-on method provided in this application is described below with reference to exemplary applications and implementations of the terminal provided in the embodiments of this application. See also Figure 11 , Figure 11This is a flowchart illustrating the virtual try-on method provided in this application embodiment. Method S200 includes the following steps:

[0168] S201: Obtain the image of the person trying on the clothes, the image of the clothes to be tried on, and the clothing category to which the clothes to be tried on belong in the image of the clothes to be tried on.

[0169] The fitting assistant (application software) built into the terminal (such as a smartphone or smart fitting mirror) acquires an image of the person trying on the clothes, an image of the clothes to be tried on, and the clothing category (e.g., top, bottom, suit, or dress) of the clothes in the image of the clothes to be tried on. The image of the person trying on can be taken by the terminal or input by the user. The image of the clothes to be tried on can be selected by the user from the fitting assistant, and the clothing category can be input by the user or obtained by the terminal based on the image of the clothes to be tried on.

[0170] Understandably, the image of the person trying on the clothes includes the person's body, and the image of the clothes to be tried on includes the clothes.

[0171] S202: Determine the target fitting model based on the clothing category. The target fitting model is trained using any of the above methods for training fitting models. The clothing category of the clothing to be tried on is the same as the clothing category of the clothing in each of the clothing images in several image groups.

[0172] For example, if the garment to be tried on is a top, then the top fitting model is selected as the target fitting model; if the garment to be tried on is a bottom, then the bottom fitting model is selected as the target fitting model; if the garment to be tried on is a suit, then the suit fitting model is selected as the target fitting model; if the garment to be tried on is a dress, then the dress fitting model is selected as the target fitting model.

[0173] S203: Perform human body analysis on the image of the wearer to obtain the analyzed image of the wearer, and obtain the image of the retained area of ​​the wearer based on the analyzed image of the wearer and the image of the wearer.

[0174] In this embodiment, the image of the wearer can be processed to obtain the image of the retained area of ​​the wearer, referring to the method of parsing and obtaining the retained area image in step S20. This will not be repeated here.

[0175] S204: Perform human key point detection on the image of the wearer to obtain the key point image of the wearer.

[0176] In this embodiment, the human body key point detection method in step S30 can be used to perform human body key point detection on the wearer image to obtain the wearer key point image. This will not be repeated here.

[0177] S205: Input the image of the preserved area of ​​the wearer, the key point image of the wearer, the image of the wearer, and the image of the clothes to be tried on into the target clothing model to obtain the clothing image.

[0178] The fitting model was trained using any of the methods mentioned above.

[0179] The terminal's built-in virtual try-on assistant includes a virtual try-on model. This model is used to virtually try on clothes and generate virtual try-on images. It is understood that this virtual try-on model is trained using the same method as the virtual try-on model training method described in the above embodiments, and has the same structure and function as the virtual try-on model described in the above embodiments; therefore, it will not be described in detail here.

[0180] This application also provides a computer-readable storage medium storing computer-executable instructions for causing an electronic device to execute the method for training a fitting model provided in this application, for example... Figure 3-9 The method for training the virtual fitting model shown, or the virtual fitting method provided in the embodiments of this application, for example, Figure 11 The virtual try-on method is shown.

[0181] In some embodiments, the storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0182] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0183] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborative files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0184] As an example, executable instructions can be deployed to execute on a single computing device (including devices such as smart terminals and servers), or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0185] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method for training a fitting model or the virtual fitting method as described in the foregoing embodiments.

[0186] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for training a fitting model, characterized in that, The virtual fitting network includes a clothing deformation network and an image generation network, including: Acquire several image groups, each image group including clothing images and model images, wherein the model images are wearing the clothing in the clothing images, and the clothing in each clothing image of the several image groups belongs to the same clothing category; Human body analysis is performed on the model image to obtain an analyzed image, and a preserved region image is obtained based on the analyzed image and the model image. The preserved region image includes other regions in the model image except for the torso region corresponding to the clothing category. Perform human body key point detection on the model image to obtain a key point image; The clothing deformation network is used to deform the clothing in the clothing image to obtain a deformed clothing image; The preserved region image and the clothing deformation image are stitched together to obtain a stitched image. The stitched image, the key point image, and the model image are then input into the image generation network to obtain a pre-test clothing image. The loss function is used to calculate the loss between the pre-test clothing image and the model image, and the clothing try-on network is iteratively trained according to the loss sum corresponding to the several image groups until convergence, thus obtaining the clothing try-on model.

2. The method according to claim 1, characterized in that, The clothing deformation network includes a dual-channel feature extraction module, an optical flow module, and a deformation module. The step of using the clothing deformation network to deform the clothing in the clothing image to obtain a deformed clothing image includes: The model image, the clothing image, and the key point image are input into the dual-channel feature extraction module for feature extraction to obtain the target feature map; The target feature map is input into the optical flow module to calculate the target optical flow map; The clothing image is deformed according to the target optical flow map to obtain the deformed clothing image.

3. The method according to claim 2, characterized in that, The optical flow module includes N flow networks, and the feature extraction module includes N convolutional layers. The N convolutional layers correspond one-to-one with the N flow networks. The feature map output by each convolutional layer is input into the corresponding flow network to obtain an optical flow map. The loss function includes a second-order smoothing loss, which is used to constrain the collinearity of adjacent external flows in each of the optical flow maps.

4. The method according to claim 3, characterized in that, The second-order smoothing loss includes: in, It is the t-th pixel on the i-th optical flow map. It is the set of horizontal, vertical, and two diagonal neighborhoods around the t-th pixel, and P is the generalized Charbonnier loss function.

5. The method according to any one of claims 1-4, characterized in that, The image generation network includes an input convolutional layer, a U-NET module, and a fusion module; The step of inputting the stitched image, the key point image, and the model image into the image generation network to obtain the pre-test garment image includes: After combining the stitched image, the key point image, and the model image by channels, the results are input into the input convolutional layer to obtain a local feature map. The local feature map is input into the U-NET module for multi-scale feature learning to obtain a multi-scale feature map. The local feature map and the multi-scale feature map are input into the fusion module for fusion processing to obtain the pre-test garment image.

6. The method according to claim 5, characterized in that, The U-NET module includes an encoder and a decoder. The encoder includes multiple cascaded downsampled convolutional layers, and the decoder includes multiple cascaded upsampled convolutional layers. There are cross-layer connections between the downsampled convolutional layers and the upsampled convolutional layers at the same level.

7. The method according to claim 5, characterized in that, The fusion module performs fusion processing in the following manner: The corresponding pixel positions of the local feature map and the multi-scale feature map are added together to obtain the pre-test garment image.

8. A virtual try-on method, characterized in that, include: Obtain an image of the person trying on the clothes, an image of the clothes to be tried on, and the clothing category to which the clothes to be tried on belong in the image of the clothes to be tried on; The target fitting model is determined based on the clothing category. The target fitting model is trained using the method for training fitting models as described in any one of claims 1-7, wherein the clothing category to which the fitting clothes belong is the same as the clothing category of the clothes in each of the clothing images in the plurality of image groups. Human body analysis is performed on the image of the wearer to obtain an analyzed image of the wearer, and an image of the retained area of ​​the wearer is obtained based on the analyzed image of the wearer and the image of the wearer; Human body key point detection is performed on the image of the wearer to obtain the key point image of the wearer; The image of the user's reserved area, the image of the user's key points, the image of the user, and the image of the clothes to be tried on are input into the target fitting model to obtain the fitting image.

9. An electronic device, characterized in that, include: At least one processor, and The memory communicatively connected to the at least one processor, wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer device to perform the method as described in any one of claims 1-8.

Images