Method for training a parsing model, virtual fitting method and related devices

Abstract

Embodiments of the present application relate to the technical field of image processing, and particularly relate to a method for training a parsing model, a virtual fitting method and related devices. The method comprises training an image generation network by using clothes deformation images and local parsing images corresponding to a plurality of image groups to obtain a parsing model. The clothes deformation images serve as prior conditions, effectively guiding the image generation network to generate a predicted parsing image with an accurate clothes contour. In addition, the local parsing images retain the torso features of the model, guiding the image generation network to generate a predicted parsing image with an accurate torso contour. Based on the guiding effect of the clothes deformation images and the fact that the local parsing images are not affected by the original clothes on the model, the predicted parsing image can accurately adapt to the clothes to be tried on. The above method trains the image generation network, and the trained parsing model can generate a more stable parsing image with a self-adaptive clothes style.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method for training an analytical model, a virtual try-on method, and related apparatus. Background Technology

[0002] With the continuous advancement of modern technology and the increasing scale of online shopping, users can purchase clothing on online shopping platforms via their mobile phones. However, since the information about the clothing available for sale is generally presented as two-dimensional images, users cannot know how the clothing would look on them. This has led to a growing demand for online try-on, making clothing display an important area in the field of modern computer vision. Online try-on typically involves capturing a user's image, performing human anatomy analysis on the image to obtain a human anatomy image, selecting clothing provided by the system, deforming the clothing according to the corresponding clothing area in the human anatomy image, and automatically replacing the user's original clothing with the deformed clothing.

[0003] However, because the style of the clothes being tried on differs from the style of the clothes the user is wearing, the outline of the clothing area in the human body analysis image differs significantly from the outline of the clothes being tried on. Distorting the clothes to match the corresponding clothing area in the human body analysis image will ruin the try-on effect. For example, if the clothes being tried on are a loose-fitting coat, and the user is wearing a tight-fitting sweater, distorting the loose coat to match the outline of the tight-fitting sweater will result in the loose coat appearing as a tight-fitting style, leading to a poor try-on effect. Summary of the Invention

[0004] The main technical problem solved by the embodiments of this application is to provide a method for training an analytical model, a virtual try-on method, and related devices. The analytical model trained by this method can generate more stable analytical images that adapt to clothing styles, so that the deformation effect of the clothes to be tried on is realistic after deformation according to the clothing area in the analytical image, which is beneficial to improving the try-on effect.

[0005] To address the aforementioned technical problems, in a first aspect, embodiments of this application provide a method for training an analytical model, comprising:

[0006] Acquire several image groups, each image group including clothing images and model images, where the model in the model image is wearing the clothing in the clothing image;

[0007] The clothing image is initially deformed according to the human body structure of the model in the model image to obtain a deformed clothing image;

[0008] Human body analysis is performed on the model image to obtain a real analysis image, and the model's torso region is extracted from the real analysis image to obtain a local analysis image;

[0009] The local analytical image and the clothing deformation image are input into the image generation network to obtain the predicted analytical image;

[0010] The loss function is used to calculate the loss between the predicted and real parsed images. Based on the loss sums corresponding to several image groups, the image generation network is iteratively trained until convergence, thus obtaining the parsed model.

[0011] In some embodiments, the aforementioned preliminary deformation of the clothing image according to the human anatomy of the model in the model image to obtain a deformed clothing image includes:

[0012] Keypoint detection is performed on the model image to obtain a keypoint image;

[0013] The geometric matching module is used to calculate and process the key point image and the clothing image respectively to generate transformation parameters;

[0014] The clothing image is deformed by using transformation parameters to obtain a deformed clothing image.

[0015] In some embodiments, the method further includes:

[0016] Each keypoint in the keypoint image is converted into a 1-channel heatmap, and the heatmaps corresponding to each keypoint are stitched together by channel to obtain a pose heatmap.

[0017] The geometric matching module is used to process the keypoint image and the clothing image separately to generate transformation parameters, including:

[0018] The geometric matching module is used to calculate and process the pose heatmap and clothing image respectively to generate transformation parameters.

[0019] In some embodiments, the aforementioned image generation network includes a connected encoder, a fusion module, and a decoder;

[0020] The method also includes:

[0021] Obtain the clothing category text information corresponding to the clothing image;

[0022] Feature encoding is performed on the textual information of clothing categories to obtain clothing category vectors;

[0023] The local analytical image and the clothing deformation image are input into the image generation network to obtain the predicted analytical image, including:

[0024] After the locally analyzed image and the clothing deformation image are stitched together, they are input into the encoder for downsampling feature extraction, and the output is an encoded feature map.

[0025] The encoded feature map and the clothing category vector are input to the fusion module for fusion processing, and the fused feature map is output.

[0026] The fused feature map is input into the decoder for upsampling feature extraction, and the output is a predicted parsed image.

[0027] In some embodiments, the aforementioned fusion module fuses the encoded feature map and the clothing category vector in the following manner:

[0028] The dimensions of the clothing category vector are transformed to convert it into a clothing category feature map of the same size as the encoded feature map.

[0029] The encoded feature map and the clothing category feature map are multiplied at corresponding positions to obtain the fused feature map.

[0030] In some embodiments, the loss function includes resolution loss, deformable region loss, and deformable feature loss, wherein resolution loss reflects the difference between the real resolution image and the predicted resolution image, deformable region loss reflects the difference between the clothing region in the deformed clothing image and the clothing region in the predicted resolution image, and deformable feature loss reflects the difference between the clothing features in the model image and the clothing features in the deformed clothing image.

[0031] In some embodiments, the aforementioned loss function includes:

[0032]

[0033]

[0034]

[0035] in, For the loss in the deformed area, For deformation feature loss, To analyze the loss, For the clothing region in the deformed clothing image, To predict the clothing region in the parsed image, To accurately analyze the image, For the predicted resolved image, M*L is the resolution of either the true resolved image or the predicted resolved image. Image of clothing Image of clothing deformation The image is of the clothing in the model's picture. To use VGG network for The i-th layer feature map obtained by feature extraction, To use VGG network for The i-th layer feature map obtained by feature extraction, where V is the layer number of the VGG network.

[0036] To address the aforementioned technical problems, in a second aspect, this application provides a virtual try-on method, comprising:

[0037] Obtain images of the clothes to be tried on and the user image;

[0038] An analytical model is used to perform human body analysis based on the image of the clothing to be tried on and the user image to obtain an analytical image. The analytical model is trained using any of the methods described in the first aspect.

[0039] The user image and the image of the clothes to be tried on are fused together. During the fusion process, the image of the clothes to be tried on is deformed based on the clothing region in the parsed image to obtain the try-on image.

[0040] To address the aforementioned technical problems, in a third aspect, this application provides an electronic device, comprising:

[0041] At least one processor, and

[0042] A memory that is communicatively connected to at least one processor, wherein,

[0043] The memory stores instructions that can be executed by at least one processor, such that the at least one processor is able to perform the method as described in the first aspect or the method as described in the second aspect above.

[0044] To address the aforementioned technical problems, in a fourth aspect, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer device to perform the methods described in the first aspect or the second aspect above.

[0045] The beneficial effects of this application's embodiments are as follows: Unlike existing technologies, the method for training a parsing model provided in this application first acquires several image groups, each including a clothing image and a model image, where the model in the model image wears the clothing shown in the clothing image. Then, the clothing image is initially deformed according to the human body structure of the model in the model image, resulting in a deformed clothing image. Human body analysis is performed on the model image to obtain a true parsing image (which can be used as a label), and the model's torso region is extracted from the true parsing image to obtain a local parsing image. This local parsing image and the deformed clothing image are input into an image generation network to obtain a predicted parsing image. Finally, a loss function is used to calculate the loss between the predicted parsing image and the true parsing image. After processing these image groups in turn, the image generation network is iteratively trained based on the sum of the losses corresponding to these image groups until convergence. The converged image generation network is the parsing model.

[0046] In this embodiment, the clothing deformation image not only preserves the clothing features but also initially reflects the approximate deformed shape of the clothing. Therefore, it can serve as a priori condition, effectively guiding the image generation network to ensure accurate clothing contours when generating predictive analytical images. This allows the predictive analytical images to adapt to the style of the clothing being tried on, making them more stable. Furthermore, the local analytical images preserve the model's torso features. On one hand, this guides the image generation network to ensure accurate torso contours when generating predictive analytical images; on the other hand, it is unaffected by the original clothing on the model, helping the clothing region in the predictive analytical images to adapt to the style of the clothing being tried on. Based on the guiding role of the clothing deformation image and the fact that the local analytical images are unaffected by the original clothing on the model, the predictive analytical images can accurately adapt to the clothing being tried on, thus effectively reducing the network's dependence on training data. Several image groups are used to train the image generation network in the above manner. Based on the backpropagation of the loss sum calculated by the loss function, the predictive analytical images corresponding to each image group are constrained to continuously approach the true predictive image. Therefore, the trained analytical model can generate more stable analytical images that adapt to clothing styles, making the deformation effect of the clothes to be tried on realistic and lifelike after deformation according to the clothing area in the corresponding analytical image, which is beneficial to improving the fitting effect, and also has a strong adaptability to new styles of clothing. Attached Figure Description

[0047] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0048] Figure 1 This is a schematic diagram illustrating the application scenarios of virtual try-on in some embodiments of this application;

[0049] Figure 2 This is a schematic diagram of the structure of an electronic device in some embodiments of this application;

[0050] Figure 3 This is a flowchart illustrating the method for training an analytical model in some embodiments of this application;

[0051] Figure 4 for Figure 3 A schematic diagram of a sub-process of step S20 in the method shown;

[0052] Figure 5 These are key point images from some embodiments of this application;

[0053] Figure 6 Human body analysis images are provided in some embodiments of this application;

[0054] Figure 7This is a schematic diagram of the image generation network in some embodiments of this application;

[0055] Figure 8 This is a flowchart illustrating the method for training an analytical model in some embodiments of this application;

[0056] Figure 9 for Figure 3 A schematic diagram of a sub-process of step S40 in the method shown;

[0057] Figure 10 This is a view of the process of generating predicted parsed images in some embodiments of this application;

[0058] Figure 11 This is a flowchart illustrating the virtual try-on method in some embodiments of this application. Detailed Implementation

[0059] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. In addition, the terms "first," "second," and "third" used herein do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0062] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0063] Furthermore, the technical features involved in the various embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0064] To facilitate understanding of the methods provided in the embodiments of this application, the terms used in the embodiments of this application will first be introduced:

[0065] (1) Neural Network

[0066] A neural network can be composed of neural units, specifically understood as a neural network with input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. Neural networks with many hidden layers are called deep neural networks (DNNs). The work of each layer in a neural network can be described by the mathematical expression y=a(W·x+b). From a physical perspective, the work of each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 2 and 3 are performed by "W·x", operation 4 by "+b", and operation 5 by "a()". The term "space" is used here because the objects being classified are not individual things, but a class of things; space refers to the set of all individuals within that class. W is the weight matrix of each layer in the neural network, where each value represents the weight of a neuron in that layer. This matrix W determines the spatial transformation from the input space to the output space, as described above; that is, the W of each layer of the neural network controls how the space is transformed. The purpose of training the neural network is to ultimately obtain the weight matrices of all layers of the trained neural network. Therefore, the training process of a neural network is essentially learning how to control spatial transformation, more specifically, learning the weight matrix.

[0067] It should be noted that, in the embodiments of this application, the models used for machine learning tasks are essentially neural networks. Common components in neural networks include convolutional layers, pooling layers, normalization layers, and deconvolutional layers. By assembling these common components in neural networks, a model is designed. When the model parameters (weight matrices of each layer) are determined such that the model error meets a preset condition or the number of model parameters is adjusted to reach a preset threshold, the model converges.

[0068] The convolutional layer is configured with multiple convolutional kernels, each with a corresponding stride, to perform convolution operations on the image. The purpose of convolution is to extract different features from the input image. The first convolutional layer may only extract some low-level features such as edges, lines, and corners, while deeper convolutional layers can iteratively extract more complex features from low-level features.

[0069] Deconvolutional layers are used to map a low-dimensional space to a high-dimensional space while preserving the connections / patterns between them (the connections from the convolution process). A deconvolutional layer has multiple convolutional kernels, each with a corresponding stride, to perform deconvolution operations on the image. Generally, framework libraries used for designing neural networks (such as the PyTorch library) have a built-in `upsumple()` function, which allows for low-dimensional to high-dimensional spatial mapping.

[0070] Pooling layers mimic the human visual system's ability to reduce the dimensionality of data or represent images with higher-level features. Common pooling operations include max pooling, mean pooling, random pooling, median pooling, and combined pooling. Typically, pooling layers are periodically inserted between convolutional layers in neural networks to achieve dimensionality reduction.

[0071] The normalization layer is used to normalize all neurons in the intermediate layer to prevent gradient explosion and gradient vanishing.

[0072] (2) Loss function

[0073] During neural network training, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight matrix of each layer (usually initialized before the first update, pre-configuring parameters for each layer). For example, if the prediction is too high, the weight matrix is ​​adjusted to predict a lower value, and this process continues until the neural network accurately predicts the target value. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are crucial equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the neural network becomes a process of minimizing this loss.

[0074] (3) Human body analysis

[0075] Human body analysis refers to segmenting a person captured in an image into multiple semantically consistent regions, such as body parts and clothing, or further subcategories of body parts and clothing. Essentially, it involves pixel-level identification of the input image and labeling each pixel with its corresponding object category. For example, neural networks can be used to distinguish various elements (such as hair, face, limbs, clothing, and background) in an image containing a human body.

[0076] Before introducing the embodiments of this application, a brief introduction will be given to the virtual try-on method known to the inventors of this application, so as to facilitate the understanding of the embodiments of this application later.

[0077] In some solutions, after acquiring the user image and the image of the clothes to try on, the clothes in the image of the clothes to try on are first deformed according to the human body structure in the user image, resulting in a deformed image of the clothes to try on. Then, the corresponding clothing area in the user image is masked, and the masked user image is fused with the deformed image of the clothes to obtain a fitting image. The fitting image shows how the clothes look on the user. In this embodiment, the clothes to try on are simply deformed according to key points of the human body structure, which can lead to problems such as blurred boundaries, resulting in a poor fitting effect.

[0078] In some solutions, after acquiring the user image and the image of the clothes to try on, the user image is first subjected to human body analysis to obtain a human body analysis image. The clothes to try on are then deformed according to the corresponding clothing area in the human body analysis image, and the deformed clothes are used to automatically replace the original clothes on the user. In this embodiment, the human body analysis image can help to define the boundary information of the clothes to try on, making the boundaries of the clothes to try on clear in the generated clothing image, and the displayed clothing effect is more reasonable.

[0079] However, because the style of the clothes being tried on differs from the style of the clothes the user is wearing, the outline of the clothing area in the human body analysis image differs significantly from the outline of the clothes being tried on. Distorting the clothes to match the corresponding clothing area in the human body analysis image will ruin the try-on effect. For example, if the clothes being tried on are a loose-fitting coat, and the user is wearing a tight-fitting sweater, distorting the loose coat to match the outline of the tight-fitting sweater will result in the loose coat appearing as a tight-fitting style, leading to a poor try-on effect.

[0080] To address the aforementioned issues, this application provides a method for training an analytical model, a virtual try-on method, and related apparatus. The method for training the analytical model involves training an image generation network using deformed clothing images and local analytical images corresponding to several image groups to obtain the analytical model. The deformed clothing images not only preserve clothing features but also preliminarily reflect the approximate deformed shape of the clothing. Therefore, they can serve as prior conditions, effectively guiding the image generation network to ensure accurate clothing contours when generating predicted analytical images. This allows the predicted analytical images to adapt to the style of the tried-on clothing and are more stable. Furthermore, the local analytical images preserve the model's torso features. On one hand, this guides the image generation network to ensure accurate torso contours when generating predicted analytical images. On the other hand, they are unaffected by the original clothing on the model, helping the clothing area in the predicted analytical images to adapt to the style of the tried-on clothing. Based on the guiding role of the deformed clothing images and the fact that the local analytical images are unaffected by the original clothing on the model, the predicted analytical images can accurately adapt to the tried-on clothing, thereby effectively reducing the network's dependence on training data. The image generation network is trained using several image groups in the manner described above. Backpropagation, based on the loss sum calculated by the loss function, constrains the predicted analytical images corresponding to each image group to continuously approximate the true predicted image. Thus, the trained analytical model can generate more stable analytical images that adapt to clothing styles. This ensures that the deformation effect of the clothing to be tried on, based on the clothing region in the corresponding analytical image, is realistic and lifelike, improving the fitting experience. Furthermore, it exhibits strong adaptability to new clothing styles.

[0081] The following describes an exemplary application of the electronic device provided in this application for training a parsing model or for virtual try-on. It is understood that the electronic device can train a parsing model, or it can use the parsing model to parse the image of the clothes to be tried on and the user image to generate a parsing image. Then, the clothes in the image of the clothes to be tried on are deformed according to the outline of the clothes area in the parsing image. The deformed image of the clothes to be tried on and the user image are fused to obtain a try-on image.

[0082] The electronic device provided in this application embodiment can be a server, such as a server deployed in the cloud. When the server is used to train the analytical model, it iteratively trains the neural network using the training set provided by other devices or those skilled in the art, determines the final model parameters, and then configures the neural network with the final model parameters to obtain the analytical model. When the server is used for virtual try-on, it calls the built-in analytical model to perform corresponding calculations on the images of clothes to be tried on and user images provided by other devices or users to obtain an analytical image. Then, it deforms the clothes in the image of clothes to be tried on according to the contour of the clothes area in the analytical image. The deformed image of clothes to be tried on and the user image are then fused to obtain a try-on image.

[0083] The electronic devices provided in some embodiments of this application can be various types of terminals such as laptops, desktop computers, or mobile devices. When the terminal is used to train an analytical model, those skilled in the art input a prepared training set into the terminal and design a neural network on the terminal. The terminal uses the training set to iteratively train the neural network to determine the final model parameters. The neural network is then configured with these final model parameters to obtain the analytical model. When the terminal is used for virtual try-on, the built-in analytical model is invoked to perform corresponding calculations on the user-input user image and the image of the clothing to be tried on, resulting in an analytical image. The contour of the clothing area in this analytical image conforms to the contour of the actual try-on effect of the clothing. Then, the clothing in the image of the clothing to be tried on is deformed according to the contour of the clothing area in the analytical image. The deformed image of the clothing to be tried on and the user image are fused to obtain a try-on image.

[0084] As an example, see Figure 1 , Figure 1 This is a schematic diagram of the application scenario of the virtual fitting system provided in the embodiments of this application. The terminal 10 is connected to the server 20 through a network, wherein the network can be a wide area network or a local area network, or a combination of both.

[0085] Terminal 10 can be used to acquire training sets and construct neural networks. For example, those skilled in the art can download prepared training sets on the terminal and build the network structure of the neural network. It is understood that terminal 10 can also be used to acquire user images and images of clothing to be tried on. For example, the user inputs a user image and an image of the clothing to be tried on through an input interface; after input, the terminal automatically acquires the user image and the image of the clothing to be tried on. Alternatively, terminal 10 may have a camera to capture user images, and terminal 10 may store a clothing image library from which the user can select an image of clothing to try on.

[0086] In some embodiments, terminal 10 locally executes the method for training an analytical model provided in this application to train a designed neural network using a training set, determine the final model parameters, and then configure the neural network with the final model parameters to obtain an analytical model. In some embodiments, terminal 10 may also send a training set and a constructed neural network stored on the terminal by a person skilled in the art to server 20 via a network. Server 20 receives the training set and the neural network, trains the designed neural network using the training set, determines the final model parameters, and then sends the final model parameters to terminal 10. Terminal 10 saves the final model parameters so that the neural network can be configured with the final model parameters to obtain an analytical model.

[0087] In some embodiments, the terminal 10 locally executes the virtual try-on method provided in this application embodiment to provide virtual clothing try-on service to the user. It calls the built-in parsing model to perform corresponding calculations on the user image and the clothing image to be tried on input by the user, generates a parsing image, and then deforms the clothing in the clothing image according to the outline of the clothing area in the parsing image. The deformed clothing image to be tried on and the user image are then fused to obtain a try-on image.

[0088] In some embodiments, terminal 10 can also send the user image and the image of the clothing to be tried on, input by the user on the terminal, to server 20 via the network. Upon receiving the user image and the image of the clothing to be tried on, server 20 calls its built-in analytical model to perform corresponding calculations on the user image and the image of the clothing to be tried on, obtaining an analytical image. The clothing in the image of the clothing to be tried on is then deformed according to the contour of the clothing area in the analytical image. The deformed image of the clothing to be tried on is then merged with the user image to obtain a fitting image. This fitting image is then sent to terminal 10. After receiving the fitting image, terminal 10 displays it on its interface for the user to view.

[0089] The structure of the electronic device in the embodiments of this application is described below. Figure 2 This is a schematic diagram of the structure of an electronic device 500 in an embodiment of this application. The electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It is understood that the bus system 540 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 2 The general labeled all buses as Bus System 540.

[0090] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor.

[0091] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0092] Memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory. Memory 550 may optionally include one or more storage devices physically located away from processor 510.

[0093] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0094] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks.

[0095] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, such as Bluetooth, WiFi, and Universal Serial Bus (USB).

[0096] Display module 553 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with user interface 530;

[0097] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0098] As can be understood from the above, the training and virtual try-on methods for analytical models provided in this application can be implemented by various types of electronic devices with computing capabilities, such as smart terminals and servers.

[0099] The following describes the method for training a parsing model provided in this application, using exemplary applications and implementations of the server provided in the embodiments of this application. See also... Figure 3 , Figure 3 This is a flowchart illustrating the method for training a parsing model provided in an embodiment of this application.

[0100] Please refer to it again. Figure 3The method S100 may specifically include the following steps:

[0101] S10: Obtain several image groups.

[0102] For any given image set, including clothing images and model images, the model in the model image is wearing the clothing in the clothing image. The clothing image includes the clothing to be tried on; for example, clothing image 1# includes a green short-sleeved shirt. The model in the model image is wearing the corresponding clothing in the clothing image; for example, in the actual fitting image corresponding to clothing image 1#, the model is wearing the green short-sleeved shirt.

[0103] It is understandable that several sets of images can be collected in advance by those skilled in the art, for example, images of clothing and corresponding images of models wearing the clothing can be crawled from some clothing sales websites.

[0104] In some embodiments, those skilled in the art can determine the number of image groups according to the actual situation, and train an accurate general model.

[0105] S20: Perform preliminary deformation on the clothing image according to the human body structure of the model in the model image to obtain a deformed clothing image.

[0106] This step deforms the clothing in the clothing image according to the human body structure of the model in the model image, resulting in a deformed clothing image. It can be understood that the clothing in the original clothing image is in a two-dimensional plane; compared to the original clothing image, the outline of the clothing in the deformed clothing image is closer to the outline of the corresponding torso of the model's body.

[0107] This embodiment does not impose any limitations on the deformation method; any method capable of deforming the clothing in the clothing image is acceptable. In some embodiments, a thin plate spline (TPS) can be used to simulate the deformation of the clothing. In some embodiments, the optical flow transformed from the clothing pixels in the clothing image to the clothing pixels in the model image can be calculated based on the clothing image and the model image. Then, the clothing pixels in the clothing image are deformed according to this optical flow to obtain a deformed clothing image.

[0108] To make the clothing outline in the deformed image more closely resemble the outline of the model's torso, in some embodiments, please refer to... Figure 4 The aforementioned step S20 specifically includes:

[0109] S21: Perform key point detection on the model image to obtain a key point image.

[0110] Human key point detection algorithms are used to detect human key points in model images, which can locate human key point information (i.e., several key points on the human body), such as... Figure 5As shown, key points can be coordinates of the nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. In some embodiments, the human key point detection algorithm can use the OpenPose algorithm for detection. In some embodiments, the human key point detection algorithm can use a 2D key point detection algorithm, such as Convolutional Pose Machine (CPM) or Stacked Hourglass Network (Hourglass).

[0111] As is understandable, each keypoint has its own serial number, and the location of each keypoint is represented by coordinates. For example, please refer again... Figure 5 The OpenPose algorithm originally defined 18 keypoints, with the keypoint numbers representing human joints. These keypoint numbers are: 0 (nose), 1 (neck), 2 (right shoulder), 3 (right elbow), 4 (right wrist), 5 (left shoulder), 6 (left elbow), 7 (left wrist), 8 (right hip), 9 (right knee), 10 (right ankle), 11 (left hip), 12 (left knee), 13 (left ankle), 14 (right eye), 15 (left eye), 16 (right ear), and 17 (left ear). The OpenPose algorithm detects these 18 keypoints for different human bodies. However, due to variations in model body shape, the coordinates of these 18 keypoints differ for each model.

[0112] Therefore, human keypoint detection is performed on the model image to obtain a keypoint image. The keypoint image includes multiple keypoints, each of which includes a sequence number and corresponding coordinates, where the sequence number represents a human joint.

[0113] S22: The geometric matching module is used to calculate and process the key point image and the clothing image respectively to generate transformation parameters.

[0114] The geometric matching module includes a dual-channel convolutional block, a correlation merging layer, and a regression network. The convolutional block comprises multiple convolutional layers used to downsample the input image and extract features. Specifically, after the keypoint image undergoes downsampling and feature extraction through a convolutional block, a keypoint feature map is output; similarly, after the clothing image undergoes downsampling and feature extraction through a convolutional block, a clothing feature map is output. Those skilled in the art can configure the structure of the convolutional block and the parameters of the convolutional layers according to actual needs; no restrictions are imposed here.

[0115] The correlation merging layer is used to merge two feature maps. Specifically, the keypoint feature map and the clothing feature map are merged through the correlation merging layer to obtain a merged feature map. This merged feature map includes both keypoint features and clothing features. In some embodiments, the correlation merging layer can use channel concatenation to merge the keypoint feature map and the clothing feature map.

[0116] Understandably, a regression network is a neural network used to perform regression analysis. Regression analysis determines the quantitative relationship between variables from a set of data. In this embodiment, the regression network is able to determine the quantitative relationship between the merged feature map and the transformation parameter. Specifically, the merged image is input into the regression network, which learns the quantitative relationship between the merged image and the transformation parameter, and estimates the transformation parameter θ.

[0117] Understandably, in this embodiment, the geometric matching module is also trained along with the image generation network. As the network converges, the geometric matching module can output more accurate transformation parameters θ.

[0118] S23: Use transformation parameters to deform the clothing image to obtain a deformed clothing image.

[0119] The clothing image is deformed according to the transformation parameter θ (e.g., interpolation deformation), resulting in a deformed clothing image. In some embodiments, thin plate spline interpolation can be used for interpolation deformation. Using thin plate spline interpolation for image deformation requires specifying the coordinates of control points, then interpolating all pixels in the clothing image using the TPS function to obtain the interpolated positions, and finally mapping the pixel values ​​to obtain the deformed clothing image. In this embodiment, the transformation parameter is used as the parameter of the TPS function. For the coordinates of each pixel in the clothing image, the TPS function is used to calculate the new coordinates, achieving positional change, i.e., deformation, resulting in the deformed clothing image.

[0120] In this embodiment, a geometric matching module is used to calculate the transformation parameters required for deformation. Then, the clothing pixels in the clothing image are interpolated and deformed according to the transformation parameters, resulting in an accurate deformed clothing image. This makes the clothing outline in the deformed image closer to the outline of the model's torso, which helps guide the image generation network to generate accurate clothing outlines when generating predictive parsing images. Furthermore, the geometric matching module is trained together with the image generation network, achieving end-to-end training, making the training process cleaner and facilitating model convergence.

[0121] In some embodiments, the method further includes:

[0122] S24: Convert each keypoint in the keypoint image into a 1-channel heatmap, and stitch the heatmaps corresponding to each keypoint together by channel to obtain a pose heatmap.

[0123] In some embodiments, for any keypoint in the keypoint image, the pixel values ​​of an 11x11 neighborhood surrounding the keypoint are set to 1, while the pixel values ​​of other areas are set to 0, thereby obtaining a 1-channel heatmap corresponding to that keypoint. It is understood that other sizes of neighborhoods around the keypoint can also be selected, such as 10x10 or 14x14.

[0124] like Figure 5 The 18 key points shown correspond to 18 one-channel heatmaps. In this embodiment, the heatmaps corresponding to these 18 key points are stitched together by channel to obtain an 18-channel pose heatmap.

[0125] This pose heatmap can reflect the shape characteristics of the model's torso, thus guiding the generation of transformation parameters θ in the geometry matching module to deform the clothing according to the body shape.

[0126] In this embodiment, the aforementioned step S22 specifically includes:

[0127] S221: The geometric matching module is used to perform feature calculation processing on the pose heatmap and clothing image respectively to generate transformation parameters.

[0128] For the structure and calculation process of the geometric matching module, please refer to the description in step S20. The key point image can be replaced with the pose heatmap, which will not be repeated here.

[0129] In this embodiment, since the pose heatmap expands each key point, it can better reflect the shape features of the model's torso compared to the key point image, which is beneficial for the geometric matching module to generate accurate transformation parameters.

[0130] S30: Perform human body analysis on the model image to obtain a real analysis image, and extract the model's torso region from the real analysis image to obtain a local analysis image.

[0131] Specifically, as can be seen from the aforementioned "Glossary (3)", human body analysis involves dividing the human body into different parts, such as... Figure 6 As shown, different body parts, such as hair, face, top, pants, arms, hat, and shoes, are identified and segmented, and represented by different colors to obtain a human body analysis image.

[0132] Here, existing analytical algorithms, such as the Graphiconomy algorithm, can be used to perform human body analysis on the model image, obtaining a human body analysis image (realistic analysis image). In the human body analysis image, pixels are divided into 20 categories, which can be labeled using 0-19. For example, 0 represents the background, 1 represents the hat, 2 represents the hair, 3 represents the gloves, 4 represents the sunglasses, 5 represents the top, 6 represents the dress, 7 represents the coat, 8 represents the socks, 9 represents the pants, 10 represents the torso skin, 11 represents the scarf, 12 represents the skirt, 13 represents the face, 14 represents the left arm, 15 represents the right arm, 16 represents the left leg, 17 represents the right leg, 18 represents the left shoe, and 19 represents the right shoe. From the human body analysis image, the category to which each region in the human body image belongs can be determined.

[0133] It is understandable that, based on the fact that the model in the model image is wearing the clothes in the corresponding clothing image, that is, the model image can represent the real wearing effect of the clothes, the human body image parsed from the model image can be used as the real parsed image.

[0134] To ensure that the torso region, reflecting the model's identity, remains unchanged during the generation of new predictive analytical images, the model's torso region is extracted from the real analytical image based on pixel categories to obtain a local analytical image. In some embodiments, the model's torso region may include areas such as the head, arms, and legs. Retaining the model's torso region and removing the clothing region in the local analytical image reduces the risk of distortion in the model's torso region and ensures that the clothing on the model does not affect the distortion of the tried-on clothing in the clothing image.

[0135] S40: Input the local analytical image and the clothing deformation image into the image generation network to obtain the predicted analytical image.

[0136] The image generation network is a pre-configured neural network with neural network components (convolutional layers, deconvolutional layers, or pooling layers, etc.). The basic structure and principles of neural networks have been explained in detail in "Glossary (1)" and will not be described in detail here. The image generation network can be constructed by those skilled in the art on a neural network design platform on a terminal (e.g., a computer) and then sent to a server. In some embodiments, the intra-layer structure (convolutional kernel, stride, etc.), inter-layer connection structure, and layer combination of convolutional layers, deconvolutional layers, or pooling layers in the image generation network can be set to obtain a specific image generation network.

[0137] The image generation network performs feature extraction, spatial transformation, and other computational processing on the input local parsing image and clothing deformation image to obtain the predicted parsing image. In some embodiments, the local parsing image and clothing deformation image are channel-stitched and then input into the image generation network. The image generation network performs computational processing on the stitched image (e.g., convolution operation, deconvolution operation, etc.) to achieve human body parsing and obtain the predicted parsing image. Among them, "human body parsing" has been described in detail in "Glossary (3)" and will not be described in detail here. It can be understood that the predicted parsing image can reflect the category to which each pixel belongs in the stitched image (local parsing image and clothing deformation image). Therefore, from the predicted parsing image, the shape regions of each part of the stitched image (local parsing image and clothing deformation image) (e.g., clothing region, limb region, and background, etc.) can be obtained.

[0138] Understandably, the image generation network performs human body parsing on several image groups in the same way as in S40, and obtains the corresponding predicted parsed images.

[0139] In some embodiments, please refer to Figure 7 The image generation network consists of a connected encoder, a fusion module, and a decoder.

[0140] Those skilled in the art will understand that the encoder is used to downsample the input image, and the size of the output downsampled feature map decreases as the convolutional layers progress. The fusion module is used to fuse at least two data points. The decoder is used to upsample the input image, and the size of the output upsampled feature map increases as the layers (which may include deconvolutional layers) progress.

[0141] Please see Figure 7 The encoder consists of multiple cascaded convolutional layers. Figure 7 The encoder, consisting of seven convolutional layers, is used as an example for illustration. The decoder consists of multiple cascaded deconvolutional layers. Figure 7 The decoder is illustrated by comprising seven deconvolutional layers.

[0142] After the encoder performs downsampling feature extraction on the input image, the downsampling feature map output from the last convolutional layer in the encoder is then input into the fusion module for fusion with other feature maps to obtain a fused feature map. This fused feature map is then input into the decoder for upsampling feature extraction layer by layer. The upsampling feature map output from the last deconvolutional layer in the decoder is the image that the generator network needs to produce.

[0143] In this embodiment, please refer to Figure 8 The aforementioned methods also include:

[0144] S60: Obtain the clothing category text information corresponding to the clothing image.

[0145] S70: Perform feature encoding on the textual information of clothing categories to obtain clothing category vectors.

[0146] The clothing category text information refers to text information that reflects the category of the clothing being tried on in the clothing image. For example, clothing category text information could be "loose short-sleeved shirt," "drop-shoulder shirt," "fitted sweater," "loose coat," "straight-leg trousers," or "A-line skirt," etc. It is understandable that clothing category text information reflects the category and style of the clothing. It is also understandable that clothing in several clothing images can cover multiple clothing categories.

[0147] It is understandable that the clothing category text information corresponding to the clothing images in several image groups can be collected in advance by those skilled in the art. For example, clothing images, corresponding model images wearing the clothing, and corresponding clothing category text information can be crawled from some clothing sales websites. The terminal sends these corresponding clothing images, model images, and clothing category text information to the server, thereby enabling the server to obtain the clothing category text information corresponding to the clothing images.

[0148] It's understandable that clothing category text information is text-based data. To enable the image generation network to learn the features reflected in this text information, the clothing category text information is encoded to obtain a clothing category vector. In other words, the clothing category text information is digitized, and the resulting clothing category vector is in digital format.

[0149] In some embodiments, a bag-of-words model can be used to encode clothing category text information, transforming it into a clothing category vector. The bag-of-words model involves selecting all words from the clothing category text information and placing them into a bag, counting the frequency of each word in the bag within the clothing category text information, and representing this count as a vector. In this embodiment, a dictionary can be constructed by integrating clothing category text information corresponding to several clothing images. This dictionary includes the words contained in the clothing category text information corresponding to each clothing image, including word position information and the corresponding word. Words can be represented numerically. Thus, the clothing category text information is encoded into a clothing category vector of length N*1, where N is the maximum length of the dictionary.

[0150] In other embodiments, a word embedding algorithm can be used to encode the clothing category text information into a clothing category vector. This word embedding algorithm is a commonly used algorithm in this field and will not be described in detail here.

[0151] In this embodiment, please refer to Figure 9 The aforementioned step S40 specifically includes:

[0152] S41: After the local analytical image and the clothing deformation image are stitched together, they are input into the encoder for downsampling feature extraction, and the encoded feature map is output.

[0153] It is understood that the locally resolved image is a 1-channel grayscale image, while the clothing deformation image is a 3-channel image. In some embodiments, the resolution of the locally resolved image is 512*384*1, and the resolution of the clothing deformation image is 512*384*3. In this embodiment, "stitching" can be done channel-wise; for example, stitching the locally resolved image and the clothing deformation image channel-wise yields a combined image of size 512*384*4. Thus, the combined image includes the torso features of the locally resolved image and the clothing features of the clothing image.

[0154] The combined image is input into the encoder for downsampling feature extraction, and the output is an encoded feature map. For details, please refer to [link to relevant documentation]. Figure 10 The combined image of size 512*384 is input into the encoder, and then downsampled features are extracted through seven convolutional layers, generating downsampled feature maps of sizes 256*192, 128*96, 64*48, 32*24, 16*12, 8*6, and 4*3 in sequence. The 4*3 downsampled feature map is the encoded feature map.

[0155] S42: The encoded feature map and the clothing category vector are input to the fusion module for fusion processing, and the fused feature map is output.

[0156] The fusion module is used to fuse at least two datasets. Therefore, the fused feature map includes not only the clothing image features reflected in the encoded feature map, but also the clothing category reflected in the clothing category vector.

[0157] In some embodiments, the fusion module fuses the encoded feature map and the clothing category vector in the following manner:

[0158] The dimensions of the clothing category vector are transformed to convert it into a clothing category feature map of the same size as the encoded feature map.

[0159] The encoded feature map and the clothing category feature map are multiplied at corresponding positions to obtain the fused feature map.

[0160] It is understood that the clothing category vector is an N*1 vector. In some embodiments, the fusion module may use at least one deconvolution layer to upsample the N*1 clothing category vector to extract features, resulting in a clothing category feature map of the same size as the encoded feature map. For example, if the encoded feature map is 4*3 in size, then the upsampled feature extraction will yield a 4*3 clothing category feature map.

[0161] Then, the encoded feature map and the clothing category feature map are multiplied at corresponding positions to obtain the fused feature map. Here, corresponding position multiplication means multiplying the pixel values ​​in the same row and column of the encoded feature map and the clothing category feature map.

[0162] In this embodiment, the clothing category vector and the encoded feature map are fused in the above manner, which enriches the weight of the image features related to clothing category in the encoded feature map, so that the image features related to clothing category in the fused feature map are highlighted.

[0163] S43: The fused feature map is input to the decoder for upsampling feature extraction, and the output is a predicted parsed image.

[0164] Specifically, please refer to Figure 10 After the 4x3 fused feature map is input into the decoder, it undergoes upsampling feature extraction through seven deconvolutional layers, successively generating upsampling feature maps of sizes 8x6, 16x12, 32x24, 64x48, 128x96, 256x192, and 512x384. The 512x384 upsampling feature map is the predicted parsed image.

[0165] In this embodiment, since the image features related to clothing category are highlighted in the fused feature map, the image generation network can be informed of the clothing category of the clothes being tried on. As training progresses, the shape of the clothing region in the predicted parsing image continuously approaches the shape under that clothing category, which helps to constrain the accuracy of the predicted clothing region in the predicted parsing image.

[0166] S50: The loss function is used to calculate the loss between the predicted parsed image and the real parsed image. Based on the loss sums corresponding to several image groups, the image generation network is iteratively trained until convergence, and the parsed model is obtained.

[0167] Here, the loss function can be configured by those skilled in the art on the terminal, and the configured loss function is sent to the server along with the image generation network. The loss function has been described in detail in the aforementioned "Glossary (2)" section, and will not be repeated here. It is understood that, based on different network structures and training methods, the structure of the loss function can be set according to the actual situation.

[0168] After processing and obtaining the predicted parsed image corresponding to a group of images, the server uses a loss function to calculate the loss corresponding to the group of images based on the predicted parsed image and the real parsed image (this loss reflects the parsing loss). After processing these groups of images in turn, the image generation network is iteratively trained based on the sum of the losses corresponding to these groups of images until convergence. The converged image generation network is then used as the parsing model.

[0169] Understandably, a smaller loss sum indicates that the shape of the clothing region in the predicted analytical image is similar to the actual shape of the clothing in the true analytical image. Therefore, a loss function can be used to calculate the loss corresponding to each image group. After processing these image groups in turn, the model parameters of the image generation network are adjusted based on the sum of losses for these image groups. That is, the loss sum is backpropagated, so that the shape of the clothing region in the predicted analytical image output by the image generation network continuously approximates the shape of the clothing region in the true analytical image. After multiple iterations of adjusting the model parameters until the image generation network converges, the converged image generation network is used as the analytical model.

[0170] It is understandable that convergence here can refer to the loss being less than a preset threshold or fluctuating within a certain range under certain model parameters. In some embodiments, the Adam algorithm is used to optimize the model parameters. For example, the number of iterations is set to 100,000, the initial learning rate is set to 0.001, the learning rate weight decay is set to 0.0005, and the learning rate decays to 1 / 10 of its original value every 1,000 iterations. The learning rate, loss, and sum can be input into the Adam algorithm to obtain the adjusted model parameters output by the Adam algorithm. These adjusted model parameters are used for the next training iteration until training is complete. The converged model parameters of the image generation network are then output, and the converged model parameters of the image generation network can be saved to obtain the analytical model.

[0171] Understandably, after the server obtains the converged model parameters of the image generation network (i.e., the final model parameters), it can send these final model parameters to the terminal. The image generation network within the terminal is configured with these final model parameters, resulting in a parsed model. In some embodiments, the server can also save the image generation network and the final model parameters to obtain a parsed model.

[0172] It should be noted that in this embodiment, multiple image groups cover various clothing categories, encompassing most types of clothing available on the market. Therefore, the trained analytical model is a general-purpose model that can be widely used for human body analysis in virtual try-on.

[0173] In this embodiment, the image generation network is trained using clothing deformation images and local analytical images corresponding to several image groups to obtain an analytical model. The clothing deformation images not only preserve clothing features but also preliminarily reflect the approximate deformed shape of the clothing. Therefore, they can serve as prior conditions, effectively guiding the image generation network to ensure accurate clothing contours when generating predicted analytical images. This allows the predicted analytical images to adapt to the style of the clothing being tried on, making them more stable. Furthermore, the local analytical images preserve the model's torso features. On one hand, this guides the image generation network to ensure accurate torso contours when generating predicted analytical images. On the other hand, they are unaffected by the original clothing on the model, helping the clothing area in the predicted analytical images to adapt to the style of the clothing being tried on. Based on the guiding role of the clothing deformation images and the fact that the local analytical images are unaffected by the original clothing on the model, the predicted analytical images can accurately adapt to the clothing being tried on, thus effectively reducing the network's dependence on training data. The image generation network is trained using several image groups in the above manner. Based on the backpropagation of the loss sum calculated by the loss function, the predicted analytical images corresponding to each image group are constrained to continuously approach the true predicted image. Therefore, the trained analytical model can generate more stable analytical images that adapt to clothing styles, making the deformation effect of the clothes to be tried on realistic and lifelike after deformation according to the clothing area in the corresponding analytical image, which is beneficial to improving the fitting effect, and also has a strong adaptability to new styles of clothing.

[0174] In some embodiments, the aforementioned loss function includes analytical loss, deformable region loss, and deformable feature loss.

[0175] The resolution loss reflects the difference between the ground truth resolution image and the predicted resolution image. A larger difference indicates a greater difference between the shapes of features in the predicted and ground truth resolution images. Conversely, a smaller difference indicates a smaller difference between the shapes of features in the predicted and ground truth resolution images. In other words, the resolution loss compares the predicted and ground truth resolution images, striving to make the shapes of features in the predicted image increasingly approximate the shapes of features in the ground truth resolution image.

[0176] The deformation region loss reflects the difference between the clothing region in the deformed image and the clothing region in the predicted analytical image. A larger difference indicates that the clothing deformation reflected in the deformed image deviates from the actual wearing effect, and the deformation effect of the geometric matching module is poor. Conversely, a smaller difference indicates that the clothing deformation reflected in the deformed image is closer to the actual wearing effect, and the deformation effect of the geometric matching module is good.

[0177] Deformation feature loss reflects the difference between clothing features in the model image and clothing features in the deformed clothing image. By comparing clothing features in the model image and clothing features in the deformed clothing image at different resolutions, the clothing in the deformed clothing image can approximate the realistic wearing effect in the model image, thereby improving the deformation effect of the geometric matching module.

[0178] In this embodiment, the accuracy of the image generation network is improved by constraining the predicted analytical image to closely approximate the real analytical image through analytical loss. Furthermore, the deformation region loss and deformation feature loss are used to ensure that the clothing in the deformed image closely resembles the realistic wearing effect in the model image, thus improving the deformation effect of the geometric matching module. Therefore, an analytical model with good deformation performance can be obtained while achieving end-to-end training.

[0179] In some embodiments, the loss function includes:

[0180]

[0181]

[0182]

[0183] in, For the loss in the deformed area, For deformation feature loss, To analyze the loss, For the clothing region in the deformed clothing image, To predict the clothing region in the parsed image, To accurately analyze the image, For the predicted resolved image, M*L is the resolution of either the true resolved image or the predicted resolved image. Image of clothing Image of clothing deformation The image is of the clothing in the model's picture. To use VGG network for The i-th layer feature map obtained by feature extraction, To use VGG network for The i-th layer feature map obtained by feature extraction, where V is the layer number of the VGG network.

[0184] Among them, the VGG network is an existing convolutional neural network, which will not be described in detail here.

[0185] In this embodiment, the loss of the image group is calculated based on the aforementioned loss function, and the sum of the losses corresponding to several image groups (i.e., the loss sum) is obtained. Then, the geometric matching module and the image generation network are iteratively trained based on the loss sum, which can accelerate the network convergence.

[0186] Furthermore, the aforementioned loss function, by constraining the clothing region in the clothing deformation image to predict the clothing region in the parsed image, allows the image generation network to focus more on the outline of the clothing region and less on the deformation of clothing texture or pattern. This effectively reduces the influence of new clothing styles on the parsed image generated by the parsing model, making the generated parsed image more stable. For example, two loose-fitting short-sleeved shirts with different styles (different textures or patterns) will produce identical parsed images of the same person.

[0187] In summary, the method for training the analytical model provided in this application trains the image generation network using deformed clothing images and local analytical images corresponding to several image groups, resulting in an analytical model. The deformed clothing images not only preserve clothing features but also preliminarily reflect the approximate deformed shape of the clothing. Therefore, they can serve as prior conditions, effectively guiding the image generation network to ensure accurate clothing contours when generating predicted analytical images. This allows the predicted analytical images to adapt to the style of the clothing being tried on, making them more stable. Furthermore, the local analytical images preserve the model's torso features. On the one hand, this guides the image generation network to ensure accurate torso contours when generating predicted analytical images; on the other hand, they are unaffected by the original clothing on the model, helping the clothing region in the predicted analytical images to adapt to the style of the clothing being tried on. Based on the guiding role of the deformed clothing images and the fact that the local analytical images are unaffected by the original clothing on the model, the predicted analytical images can accurately adapt to the clothing being tried on, thus effectively reducing the network's dependence on training data. By training the image generation network with several image groups in the above manner, and using backpropagation based on the loss sum calculated by the loss function, the predicted analytical images corresponding to each image group are constrained to continuously approach the true predicted image. Therefore, the trained analytical model can generate more stable analytical images that adapt to clothing styles, making the deformation effect of the clothes to be tried on realistic and lifelike after deformation according to the clothing area in the corresponding analytical image, which is beneficial to improving the fitting effect, and also has a strong adaptability to new styles of clothing.

[0188] After training the analytical model using the method provided in this application, the analytical model can be applied to virtual try-on. The virtual try-on provided in this application can be implemented by various types of electronic devices with computing capabilities, such as smart terminals and servers.

[0189] The virtual try-on method provided in this application is described below with reference to exemplary applications and implementations of the terminal provided in the embodiments of this application. See also Figure 11 , Figure 11 This is a flowchart illustrating the virtual try-on method provided in this application embodiment. Method S200 includes the following steps:

[0190] S201: Obtain the image of the clothes to be tried on and the user image.

[0191] A fitting assistant (application software) built into a terminal (such as a smartphone or smart fitting mirror) acquires an image of the user and an image of the clothing to be tried on. The user image can be taken by the terminal or input by the user. The image of the clothing to be tried on can be selected by the user from the fitting assistant.

[0192] Understandably, user images include the user's body, and images of clothes to be tried on include the clothes.

[0193] S202: Using an analytical model, based on the image of the clothing to be tried on and the user image, perform human body analysis to obtain an analytical image.

[0194] The analytical model is trained using any of the methods described in the aforementioned training examples.

[0195] The built-in virtual try-on assistant includes a parsing model that can be used for human body analysis. The model takes an image of the clothing to be tried on and the user's image, and performs human body analysis on the user's image based on the clothing category in the clothing image, resulting in a parsed image corresponding to that clothing category.

[0196] It is understood that the analytical model is trained using the method described in the above embodiments, and has the same structure and function as the analytical model described in the above embodiments, so it will not be described in detail here.

[0197] It is understandable that the shape of the clothing area in the parsed image matches the category of the clothing to be tried on, thus the shape of the clothing area in the parsed image conforms to the actual wearing effect of the clothing.

[0198] S203: Fuse the user image and the image of the clothes to be tried on, and during the fusion process, deform the image of the clothes to be tried on based on the clothing region in the parsed image to obtain the try-on image.

[0199] In some embodiments, the parsed image, the image of the garment to be tried on, and the user image are input into a generative adversarial network. The user image and the image of the garment to be tried on are fused to generate a fitting image. During the fusion process, the parsed image constrains the boundary information of the fitting garment, causing the fitting garment to be deformed according to the garment region in the parsed image. Thus, the fused fitting image conforms to the realistic fitting effect of the garment to be tried on.

[0200] Generative adversarial networks (GANs) are a common method known to those skilled in the art for their application in image generation, and will not be described in detail here.

[0201] In this embodiment, the analytical model can generate more stable analytical images that adapt to clothing styles. Therefore, by using this analytical model to analyze and deform the images of the clothes to be tried on and the user image, the deformation effect of the clothes to be tried on after being deformed according to the clothing area in the corresponding analytical image is realistic and lifelike, which is beneficial to improving the fitting effect and also has strong adaptability to new styles of clothing.

[0202] This application also provides a computer-readable storage medium storing computer-executable instructions for causing an electronic device to execute the method for training a parsing model provided in this application, for example... Figures 3-10 The method for training the analytical model shown, or the virtual try-on method provided in the embodiments of this application, for example, Figure 11 The virtual try-on method is shown.

[0203] In some embodiments, the storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0204] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0205] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborative files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0206] As an example, executable instructions can be deployed to execute on a single computing device (including devices such as smart terminals and servers), or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0207] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method for training an analytical model or the virtual try-on method as described in the foregoing embodiments.

[0208] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0209] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0210] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for training an analytical model, characterized in that, include: Acquire several image groups, each image group including clothing images and model images, wherein the model in the model image is wearing the clothing in the clothing image; The clothing image is initially deformed according to the human body structure of the model in the model image to obtain a deformed clothing image; Human body analysis is performed on the model image to obtain a real analysis image, and the model's torso region is extracted from the real analysis image to obtain a local analysis image; The local analytical image and the clothing deformation image are input into an image generation network to obtain a predicted analytical image; The loss between the predicted parsed image and the real parsed image is calculated using a loss function, and the image generation network is iteratively trained based on the sum of losses corresponding to the several image groups until convergence, thus obtaining the parsing model; The image generation network includes a connected encoder, a fusion module, and a decoder; The method further includes: Obtain the clothing category text information corresponding to the clothing image; The clothing category text information is feature-encoded to obtain a clothing category vector; The step of inputting the local parsed image and the clothing deformation image into an image generation network to obtain a predicted parsed image includes: After the local analytical image and the clothing deformation image are stitched together, they are input into the encoder for downsampling feature extraction, and the encoded feature map is output. The encoded feature map and the clothing category vector are input into the fusion module for fusion processing, and a fused feature map is output. The fused feature map is input into the decoder for upsampling feature extraction, and the predicted parsed image is output.

2. The method according to claim 1, characterized in that, The preliminary deformation of the clothing image according to the human anatomy of the model in the model image to obtain a deformed clothing image includes: Key point detection is performed on the model image to obtain a key point image; The geometric matching module is used to calculate and process the key point image and the clothing image respectively to generate transformation parameters; The clothing image is deformed using the transformation parameters to obtain the deformed clothing image.

3. The method according to claim 2, characterized in that, The method further includes: Each key point in the key point image is converted into a 1-channel heatmap, and the heatmaps corresponding to each key point are stitched together by channel to obtain a pose heatmap. The geometric matching module performs calculations on the key point image and the clothing image respectively to generate transformation parameters, including: The geometric matching module is used to calculate and process the pose heatmap and the clothing image respectively to generate the transformation parameters.

4. The method according to claim 1, characterized in that, The fusion module fuses the encoded feature map and the clothing category vector in the following manner: The clothing category vector is transformed in dimension to become a clothing category feature map of the same size as the encoded feature map. The fused feature map is obtained by multiplying the encoded feature map and the clothing category feature map at corresponding positions.

5. The method according to claim 3, characterized in that, The loss function includes analytical loss, deformed region loss, and deformed feature loss. The analytical loss reflects the difference between the real analytical image and the predicted analytical image. The deformed region loss reflects the difference between the clothing region in the deformed clothing image and the clothing region in the predicted analytical image. The deformed feature loss reflects the difference between the clothing features in the model image and the clothing features in the deformed clothing image.

6. The method according to claim 5, characterized in that, The loss function includes: in, For the loss in the deformed region, For the loss of the deformation features, For the analytical loss, The clothing region in the deformed clothing image. For the clothing region in the predicted parsed image, For the actual parsed image, For the predicted parsed image, M L is the resolution of the actual resolved image or the predicted resolved image. The image of the clothing, The image of the deformed clothing. The clothing image in the model image. To use VGG network for the The i-th layer feature map obtained by feature extraction, To use the VGG network for the The i-th layer feature map obtained by feature extraction, where V is the layer number of the VGG network.

7. A virtual try-on method, characterized in that, include: Obtain images of the clothes to be tried on and the user image; An analytical model is used to perform human body analysis based on the image of the clothing to be tried on and the user image to obtain an analytical image. The analytical model is trained using the method described in any one of claims 1-6. The user image and the image of the clothes to be tried on are fused together, and during the fusion process, the image of the clothes to be tried on is deformed based on the clothing area in the parsed image to obtain the fitting image.

8. An electronic device, characterized in that, include: At least one processor, and A memory communicatively connected to the at least one processor, wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer device to perform the method as described in any one of claims 1-7.

Images