Virtual fitting model training method, virtual fitting method and electronic equipment

Abstract

The embodiment of the invention relates to the technical field of image processing, and discloses a training method of a virtual fitting model, a virtual fitting method and electronic device.According to the training method of the virtual fitting model, a two-channel attention network is constructed, the two-channel attention network comprises an upper channel network and a lower channel network, the upper channel network is used for extracting texture features of a clothes image, and the lower channel network combines a clothes texture feature map and a model reserved area map by using a space attention mechanism to generate a space attention map, and performs feature coding on a description text corresponding to the clothes image; the space attention graph and the text features are fused through a cross attention mechanism to generate a fused feature graph, the fused feature graph is denoised for multiple times to obtain a final fitting effect graph, the stability of the clothes texture can be enhanced, and then the virtual fitting effect is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a training method for a virtual fitting model, a virtual fitting method, and an electronic device. Background Art

[0002] Virtual fitting refers to the use of virtual technology to automatically display a three-dimensional image of what a new garment will look like after trying it on. Virtual fitting technology allows customers to put on new garments and see the effect without having to take off their clothes.

[0003] At present, offline smart fitting mostly uses interactive fitting mirror equipment. Shoppers stand in front of the fitting mirror and choose to try on clothes. The fitting mirror will display the overall clothing according to the user's basic characteristics and clothing style, and the user can switch clothes freely. Online fitting generally takes the user's image and selects the target clothing provided by the system for automatic replacement. However, whether it is a smart fitting mirror or an online virtual fitting, most of the existing technologies use the method of collecting human body data and reshaping the user's image through 3D modeling. The combination of clothing and user image is often not appropriate enough and lacks a sense of naturalness, resulting in a poor user experience. At the same time, collecting 3D information of the human body and clothing is usually costly and cumbersome. Therefore, virtual fitting based on 2D images has become the main research direction in recent years.

[0004] The existing virtual fitting algorithm uses the corresponding fitting relationship between the human torso area and the clothing area reflected by the preset matching rule library to split the tried-on clothes into at least two clothing areas, and deforms them according to the corresponding key points in the human image, so that the deformation of each clothing area conforms to the corresponding human torso area and can fit the human torso. However, it only deforms according to the key points in the human image, which easily ignores the details of the clothing texture, resulting in unsatisfactory fitting effects. Summary of the invention

[0005] The embodiments of the present application provide a training method for a virtual fitting model, a virtual fitting method and an electronic device to solve the problem of poor texture details of clothes during the fitting process, and can enhance the clarity of the texture of the clothing image and the virtual fitting effect.

[0006] In a first aspect, an embodiment of the present application provides a training method for a virtual fitting model, wherein the virtual fitting model includes a dual-channel attention network, the dual-channel attention network includes an upper channel network and a lower channel network, the lower channel network includes a spatial attention mechanism layer, a cross attention mechanism layer, and an upsampling layer, and the training method of the virtual fitting model includes:

[0007] Obtaining a training data set, wherein the training data set includes clothing images, description texts of the clothing images, and model images matching the clothing images;

[0008] Perform occlusion processing on the top area of ​​the model image to obtain a model reserved area map;

[0009] The feature extraction of clothing image is performed through the upper channel network to obtain the clothing texture feature map;

[0010] Perform feature encoding on the description text to generate a text feature map corresponding to the description text;

[0011] The clothing texture feature map and the model reserved area map are fused through the spatial attention mechanism layer to obtain a spatial attention map.

[0012] The spatial attention map and the text feature map are fused through the cross attention mechanism layer to obtain a fused feature map;

[0013] The fused feature map is decoded through the upsampling layer of the lower channel network to obtain a denoised feature map, and after several denoising steps, a fitting effect map is obtained;

[0014] A loss function is constructed, and the lower channel network is trained based on the training data set until the loss function converges to generate a trained virtual fitting model.

[0015] In some embodiments, the upper channel network includes at least two sequentially connected downsampling layers, each of which is used to perform a downsampling operation on the clothing image to obtain at least two clothing texture feature maps of different scales;

[0016] The upper channel network also includes at least two upsampling layers connected in sequence, each of which is used to perform an upsampling operation on the clothing texture feature map to obtain at least two clothing texture feature maps of different scales.

[0017] In some embodiments, the down-channel network further comprises at least two sequentially connected down-sampling layers, each down-sampling layer being used to down-sample the model-retaining region map to obtain at least two model-retaining region maps of different scales;

[0018] The clothing texture feature map and the model reserved area map are fused through the spatial attention mechanism layer to obtain a spatial attention map, including:

[0019] The model retention area map and clothing texture feature map of the same scale are spliced ​​to obtain the spatial attention map.

[0020] In some embodiments, the model reserved region map and the clothing texture feature map of the same scale are spliced ​​to obtain a spatial attention map, including:

[0021] The model reserved area map and clothing texture feature map of the same scale are spliced ​​to obtain a spliced ​​feature map;

[0022] Perform average pooling and maximum pooling operations on the concatenated feature maps to generate an average pooling two-dimensional map and a maximum pooling two-dimensional map;

[0023] Generate a spatial attention map based on the average pooled 2D map and the maximum pooled 2D map, including:

[0024] M s (F) = σ(f a*a ([AvgPool(F); MaxPool(F)]))

[0025] Among them, M s (F) is the spatial attention map, F is the concatenated feature map, AvgPool is the average pooling two-dimensional map, MaxPool(F) is the maximum pooling two-dimensional map, and f a*a ([AvgPool(F); MaxPool(F)]) is a convolution operation with a filter size of a*a, where a is a positive integer.

[0026] In some embodiments, the downsampling layer includes an attention mechanism layer, the attention mechanism layer includes an attention mechanism module and an embedding layer, the attention mechanism module includes a spatial attention mechanism layer, a cross attention mechanism layer and a convolution layer; wherein,

[0027] The embedding layer is used to convert the feature map input to the next channel network into a high-dimensional vector.

[0028] In some embodiments, performing noise processing on the model image includes:

[0029] The model image is subjected to t noise adding steps, including:

[0030]

[0031] Among them, β t is the variance used in each t step, I is the mean, N is the normal distribution, X t is the t-th step based on β t Model image after adding Gaussian noise, X t-1 is the t-1 step based on β t-1 The model image obtained after adding Gaussian noise, where t is the step size, the value of t ranges from 1 to T, T is the maximum step size, β t The value of is between 0 and 1, β1<β2<…<β T .

[0032] In some embodiments, the method further comprises:

[0033] Extract the key points of the human body in the model image and obtain the human body posture diagram;

[0034] The human pose graph is input to the lower channel network.

[0035] In some embodiments, the model image after adding noise is denoised by the lower channel network to generate a fitting effect image, including:

[0036] The model image after adding noise is subjected to t denoising steps, including:

[0037] p θ (X t-1 |X t )=N(X t-1 |μ θ (X t , P, R, C), 0)

[0038] Among them, X t is the model image after adding Gaussian noise in step t, P is the human body posture map, R is the model reserved area map, C is the clothing image, μ θ (X t , P, R, C) are the estimated X in the denoising process t-1 The mean function, N(X t-1 |μ θ (X t , P, R, C), 0) is X t-1 At a given X t Under the normal distribution, For X t-1 Noise, t is the step length, the value of t ranges from 1 to T, and T is the maximum step length;

[0039]

[0040] Among them, X t is the model image after adding Gaussian noise in step t, P is the human body posture map, R is the model reserved area map, C is the clothing image, For X t-1 The noise is, where t is the step length, the value of t ranges from 1 to T, T is the maximum step length, α t is a hyperparameter that adjusts the strength of the denoising process.

[0041] In some embodiments, a loss function is constructed to train the lower channel network based on the image dataset, including:

[0042] Obtain sample data from a training data set, where the sample data includes clothing images, description texts of the clothing images, and model images matching the clothing images;

[0043] Add random noise to the model image to obtain a noisy model image;

[0044] The noisy model image, the model reserved area map, the clothing texture feature map, the text feature map, and the human posture map are input into the lower channel network to generate prediction noise through the lower channel network;

[0045] Calculate the loss between predicted noise and random noise;

[0046] Repeatedly obtain sample data for training until the calculated loss is less than the loss threshold to obtain the trained lower channel network.

[0047] In some embodiments, the loss function includes:

[0048]

[0049] in, is the mean square error loss of random noise and the noise predicted by the lower channel network, ∈ is random noise, is a hyperparameter, Y o is the sample data, The noise predicted for the lower channel network, is the mean squared error between random noise and the noise predicted by the lower channel network.

[0050] In some embodiments, feature fusion is performed on the spatial attention map and the text feature map through a cross attention mechanism layer to obtain a fused feature map, including:

[0051] The spatial attention map and the text feature map are fused through the cross attention mechanism, including:

[0052]

[0053] Among them, x1 is the spatial attention map, x2 is the text feature map, Q is the query vector corresponding to x1, K is the key vector corresponding to x2, V is the value vector corresponding to x2, and T is the time step.

[0054] In some embodiments, the virtual fitting model further includes:

[0055] An encoder, used for encoding the clothing image to obtain a feature map corresponding to the clothing image; and, used for encoding the model reserved area map to obtain a feature map corresponding to the model reserved area map;

[0056] A decoder, used to decode the denoised model image to obtain a fitting effect image;

[0057] The text encoder is used to perform feature encoding on the description text and generate a text feature map corresponding to the description text.

[0058] In a second aspect, an embodiment of the present application provides a virtual fitting method, the virtual fitting method comprising:

[0059] Obtaining an image of clothes to be tried on, an image of a user, and a description text of the image of clothes to be tried on;

[0060] The image of clothes to be tried on, the user image, and the description text of the image of clothes to be tried on are input into a virtual fitting model to obtain a fitting effect diagram corresponding to the image of clothes to be tried on and the image of the model, wherein the virtual fitting model is trained based on the method of the first aspect.

[0061] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0062] At least one processor. And

[0063] a memory communicatively coupled to at least one processor, wherein:

[0064] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the first aspect or the second aspect.

[0065] Beneficial effects of the embodiments of the present application: Different from the prior art, the embodiments of the present application provide a training method for a virtual fitting model, wherein the virtual fitting model includes a dual-channel attention network, the dual-channel attention network includes an upper channel network and a lower channel network, the lower channel network includes a spatial attention mechanism layer, a cross attention mechanism layer, and an upsampling layer, and the method includes: obtaining a training data set, wherein the training data set includes a clothing image, a description text of the clothing image, and a model image matching the clothing image; performing occlusion processing on the upper clothing area of ​​the model image to obtain a model reserved area map; performing feature extraction on the clothing image through the upper channel network to obtain a clothing Texture feature map; feature encode the description text to generate a text feature map corresponding to the description text; perform feature fusion on the clothing texture feature map and the model reserved area map through the spatial attention mechanism layer to obtain a spatial attention map; perform feature fusion on the spatial attention map and the text feature map through the cross attention mechanism layer to obtain a fused feature map; decode the fused feature map through the upsampling layer of the lower channel network to obtain a denoised feature map, and after several denoising steps, obtain a fitting effect map; construct a loss function, and train the lower channel network based on the training data set until the loss function converges to generate a trained virtual fitting model.

[0066] On the one hand, by constructing a dual-channel network to extract features of clothing images, model images, and description texts of clothing images and perform denoising processing to restore the fitting effect images, the clarity of the effect images can be improved.

[0067] On the other hand, by constructing a loss function for training and training the virtual fitting network based on the training data set to generate a virtual fitting model, the fitting effect can be enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0069] Figure 1 It is a schematic diagram of an application environment of a training method for a virtual fitting model provided in an embodiment of the present application;

[0070] Figure 2 It is a flowchart of a training method for a virtual fitting model provided in an embodiment of the present application;

[0071] Figure 3 It is a schematic diagram of a process of adding noise to a model image provided by an embodiment of the present application;

[0072] Figure 4 yes Figure 2 A detailed flow chart of step S208 in FIG.

[0073] Figure 5 This is a schematic diagram of the overall structure of generating a fitting effect diagram provided by an embodiment of the present application;

[0074] Figure 6 It is a structural diagram of an attention mechanism layer provided in an embodiment of the present application;

[0075] Figure 7 It is a flowchart of a virtual fitting method provided in an embodiment of the present application;

[0076] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0077] Description of Figure Numbers:

[0078]

[0079] DETAILED DESCRIPTION

[0080] The present application is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements can also be made without departing from the concept of the present application. These all belong to the protection scope of the present application.

[0081] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0082] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other, all within the scope of protection of the present application. In addition, although the functional module division is performed in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a sequence different from the module division in the device or the flow chart. In addition, the words "first", "second", "third", etc. used herein do not limit the data and execution order, but only distinguish the same items or similar items with basically the same functions and effects.

[0083] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in this specification and in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.

[0084] In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0085] Before introducing the embodiments of the present application, a brief introduction is first given to the portrait semantic segmentation method known to the inventors of the present application to facilitate subsequent understanding of the embodiments of the present application.

[0086] In some virtual fitting algorithms, for example: referring to the patent document with publication number CN116703747A, the above patent discloses a virtual fitting image restoration method based on a diffusion conditional generation algorithm. The method uses an improved U-Net as the backbone network and introduces virtual fitting related picture conditional coding through a cross-attention fusion method to perform image restoration and reconstruction, thereby improving the clothing fitting effect. However, due to the characteristics of the diffusion model, there are random seeds. When the same picture is input, each generated effect has randomness and uncertainty, so the stability of the generated effect cannot be guaranteed.

[0087] Referring to the patent document with publication number CN115293958A, the above patent discloses a clothing deformation method, which divides the clothing to be tried on into at least two clothing areas according to the corresponding fitting relationship between the human torso area and the clothing area reflected by the preset matching rule library, and deforms them respectively according to the corresponding key points in the human body image, so that the deformation of each clothing area conforms to the corresponding human torso area and can fit the human torso. However, it only deforms according to the key points in the human body image, which is easy to ignore details, and it cannot adapt to complex postures, resulting in unsatisfactory fitting effect and inability to meet user needs.

[0088] The above scheme cannot guarantee the stability of the fitting effect, and the posture adaptability of the fitting effect and the clothing texture effect are not good.

[0089] Based on this, in order to enhance the clarity of the texture of clothing images and improve the fit between the human body and clothing to enhance the fitting effect, an embodiment of the present application provides a training method for a virtual fitting model, wherein the virtual fitting model includes a dual-channel attention network, the dual-channel attention network includes an upper channel network and a lower channel network, the lower channel network includes a spatial attention mechanism layer, a cross attention mechanism layer, and an upsampling layer, and the method includes: obtaining a training data set, wherein the training data set includes a clothing image, a description text of the clothing image, and a model image matching the clothing image; performing occlusion processing on the upper clothing area of ​​the model image to obtain a model reserved area map; performing occlusion processing on the clothing image through the upper channel network Feature extraction is performed to obtain a clothing texture feature map; feature encoding is performed on the description text to generate a text feature map corresponding to the description text; the clothing texture feature map and the model retention area map are feature fused through the spatial attention mechanism layer to obtain a spatial attention map; the spatial attention map and the text feature map are feature fused through the cross attention mechanism layer to obtain a fused feature map; the fused feature map is decoded through the upsampling layer of the lower channel network to obtain a denoised feature map, and after several denoising steps, a fitting effect map is obtained; a loss function is constructed, and the lower channel network is trained based on the training data set until the loss function converges to generate a trained virtual fitting model.

[0090] On the one hand, by constructing a dual-channel network to extract features of clothing images, model images, and description texts of clothing images and perform denoising processing to restore the fitting effect images, the clarity of the effect images can be improved.

[0091] On the other hand, by constructing a loss function for training and training the virtual fitting network based on the training data set to generate a virtual fitting model, the fitting effect can be enhanced.

[0092] Before explaining the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0093] (1) Neural network, also referred to as neural network (NNs) or connection model, is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. Neural networks rely on the complexity of the system to adjust the interconnected relationships between a large number of internal nodes to achieve the purpose of processing information. Specifically, a neural network can be composed of neural units, which can be specifically understood as a neural network with an input layer, a hidden layer, and an output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. Among them, a neural network with many hidden layers is called a deep neural network (DNN). The work of each layer in a neural network can be described by the mathematical expression y=a(W·x+b). From a physical level, the work of each layer in a neural network can be understood as completing the transformation from input space to output space (i.e., the row space of a matrix to the column space) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimension increase / reduction; 2. Enlargement / reduction; 3. Rotation; 4. Translation; 5. "Bending". The operations of 1, 2, and 3 are completed by "W·x", the operation of 4 is completed by "+b", and the operation of 5 is implemented by "a()". The word "space" is used here because the classified object is not a single thing, but a class of things. Space refers to the collection of all individuals of this type of thing. W is the weight matrix of each layer of the neural network, and each value in the matrix represents the weight value of a neuron in this layer. The matrix W determines the spatial transformation from the input space to the output space mentioned above, that is, the W of each layer of the neural network controls how to transform the space. The purpose of training a neural network is to eventually obtain the weight matrices of all layers of the trained neural network. Therefore, the training process of a neural network is essentially to learn how to control spatial transformation, or more specifically, to learn the weight matrix.

[0094] It should be noted that in the embodiments of the present application, the models used in the machine learning tasks are essentially neural networks. Common components in neural networks include convolutional layers, activation function layers, and batch normalization layers. By assembling these common components in neural networks, a model is designed. When the model parameters (weight matrices of each layer) are determined so that the model error meets the preset conditions or the number of model parameters is adjusted to reach a preset threshold, the model converges.

[0095] The convolution layer is configured with multiple convolution kernels, each of which is set with a corresponding step size to perform convolution operations on the image. The purpose of the convolution operation is to extract different features of the input image. The first convolution layer may only extract some low-level features such as edges, lines, and corners. Deeper convolution layers can iteratively extract more complex features from low-level features.

[0096] The activation function layer is used to make each neuron in the neural network accept the output value of the previous layer as the input value of the neuron, and pass the processing result to the next layer. Common activation functions of the activation function layer include but are not limited to the Rectified Linear Unit (Relu) function, Swish function, and Parametric Rectified Linear Unit (PRelu) function with parameters.

[0097] Batch Normalization (BN) layer is used to standardize the features of a certain layer in the network. Its purpose is to solve the problem of numerical instability in deep neural networks, that is, to solve the problem that as the number of network layers increases, the update of parameters during training is likely to cause drastic changes in the feature output close to the output layer, which is not conducive to training an effective neural network.

[0098] (2) Loss function refers to a function that maps the value of a random event or its related random variables to a non-negative real number to represent the "risk" or "loss" of the random event. The loss function is a non-negative real function that is used to quantify the difference between the predicted label and the true label predicted by the model. In applications, the loss function is usually associated with the optimization problem as a learning criterion, that is, the model is solved and evaluated by minimizing the loss function. For example, it is used for model parameter estimation in statistics and machine learning. In the process of training a neural network, because we hope that the output of the neural network is as close as possible to the value we really want to predict, we can compare the current network's predicted value with the desired target value, and then update the weight matrix of each layer of the neural network according to the difference between the two (there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the neural network). For example, if the network's predicted value is too high, adjust the weight matrix to make it predict a lower value, and continue to adjust until the neural network can predict the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which is an important equation used to measure the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.

[0099] The technical solution of the present application is described in detail below in conjunction with the accompanying drawings.

[0100] See also Figure 1 , Figure 1 It is a schematic diagram of an application environment of a training method for a virtual fitting model provided in an embodiment of the present application;

[0101] like Figure 1 As shown, the application environment 100 includes: a terminal 101 and a server 102, and the terminal 101 and the server 102 communicate through wired or wireless communication.

[0102] The terminal 101 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal 101 may be provided with a client, which may be a video client, a browser client, an online shopping client, an instant messaging client, etc., and this application does not limit the type of the client.

[0103] The terminal 101 and the server 102 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application. The terminal 101 may receive the image of clothing to be tried on, the image of a model, and the description text of the image of clothing sent by the server 102, and display the image of clothing to be tried on, the image of a model, and the fitting image corresponding to the image of clothing to be tried on obtained by fitting the two on a visual interface. The terminal may respond to a virtual fitting instruction triggered by a user, and obtain the image of clothing to be tried on and the image of a model through an image acquisition device, which may be built into the terminal 101 or may be externally connected to the terminal 101, which is not limited in this application.

[0104] Terminal 101 can send the virtual fitting instruction and the collected images of clothes to be tried on, model images, and description texts of the clothes images to be tried on to server 102, and receive the fitting image corresponding to the image of clothes to be tried on returned by server 102, and then display the fitting image corresponding to the image of clothes to be tried on on a visual interface, so that the user can view the fitting image corresponding to the image of clothes to be tried on.

[0105] It is understandable that the terminal 101 may generally refer to one of a plurality of terminals, and the embodiment of the present application is only illustrated by the terminal 101. Those skilled in the art may know that the number of the above terminals may be more or less. For example, the above terminal may be only one, or the above terminals may be dozens or hundreds, or more. The embodiment of the present application does not limit the number of terminals and device types.

[0106] Among them, server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0107] The server 102 and the terminal 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application. The server 102 may maintain a clothing image database for storing multiple clothing images. The server 102 may receive a virtual fitting instruction and a clothing image to be fitted, a model image, and a description text of the clothing image sent by the terminal 101, and perform a virtual fitting on the clothing image to be fitted and the model image according to the virtual fitting instruction, obtain a fitting image corresponding to the clothing image to be fitted, and then send the fitting image corresponding to the clothing image to be fitted to the terminal 101.

[0108] It is understandable that the number of the above servers 102 may be more or less, and the embodiment of the present application does not limit this. Of course, the server 102 may also include other functional servers to provide more comprehensive and diversified services.

[0109] The technical solution of the present application can be applied to various scenarios of virtual fitting. Specifically, the technical solution of the present application can be used to fuse the image of clothes to be fitted, the image of a model, and the description text of the clothes image in the virtual fitting scenario. In the virtual fitting scenario, in some implementations, the image of clothes to be fitted, the image of a model, and the description text of the clothes image of the scene are fused by a virtual fitting model to determine the fitting image corresponding to the image of clothes to be fitted, wherein the virtual fitting model is pre-trained.

[0110] At present, the virtual fitting algorithm uses the corresponding fitting relationship between the human torso area and the clothing area reflected by the preset matching rule library to split the tried-on clothes into at least two clothing areas, and deforms them according to the corresponding key points in the human image. It only pays attention to the human posture and whether the clothes fit, but ignores the texture details of the clothes, resulting in unsatisfactory fitting effects.

[0111] In view of this, the embodiments of the present application provide a training method for a virtual fitting model, a prediction method for a virtual fitting image, and an electronic device to enhance the clarity of clothing image texture and the virtual fitting effect.

[0112] Embodiment 1

[0113] See also Figure 2 , Figure 2 It is a flowchart of a training method for a virtual fitting model provided in an embodiment of the present application;

[0114] The method can be applied to various electronic devices, such as terminals, servers and other electronic devices. Specifically, the execution subject of the method is one or at least two processors of the electronic device.

[0115] Among them, the virtual fitting model includes a dual-channel attention network, and the dual-channel attention network includes an upper channel network and a lower channel network.

[0116] Among them, the dual-channel attention network is based on U 2 -NET network-based network structure, specifically, U 2The -NET network is a two-layer nested U-shaped structure, which designs a new U-shaped residual block (ReSidual U-blocks, RSU) structure at the bottom layer and a U-Net-like structure at the top layer, where each level is filled with RSU. RSU replaces ordinary single-stream convolution with U-Net and replaces the original features with local features composed of a weight layer. This design change enables the network to directly extract features from multiple scales of each residual block.

[0117] Among them, U 2 -Net network consists of three parts: six-level encoder, five-level decoder, and saliency map fusion module, which is connected to the decoder and the last level encoder. The six-level encoder includes: En_1, En_2, En_3, En_4, En_5 and En_6, and the five-level decoder includes: De_1, De_2, De_3, De_4 and De_5.

[0118] Specifically, in encoders En_1, En_2, En_3, and En_4, residual U blocks RSU-7, RSU-6, RSU-5, and RSU-4 are used, respectively. Where "7", "6", "5", and "4" represent the height (L) of the RSU block. L is usually configured according to the spatial resolution of the input feature map. In En_5 and En_6, RSU-4F is used, where "F" indicates that RSU is an expanded version and the pooling and upsampling operations are replaced by expanded convolutions. All intermediate feature maps of RSU-4F have the same resolution as their input feature maps.

[0119] The structure of the decoder is similar to that of its symmetric encoder, En_6. De_5 also uses a dilated version of the residual U block RSU-4F similar to that used in the encoder stage, En_5 and En_6. Each decoder stage takes as input the concatenation of the upsampled feature map from its previous stage and the upsampled feature map from its symmetric encoder stage.

[0120] Step S201: obtaining a training data set, wherein the training data set includes clothing images, description texts of the clothing images, and model images matching the clothing images;

[0121] It is understandable that before training the virtual fitting model, it is necessary to collect a training data set and perform model training based on the training data set, wherein the training data set includes clothing images, description texts of the clothing images, and model images matching the clothing images.

[0122] Among them, the description text of the clothing image is directly input by the server or the client, or the characteristic information of the clothing image is automatically identified through image recognition technology to generate the description text corresponding to the clothing image. The description text of the clothing image includes clothing style (such as shirt, T-shirt, etc.), clothing color, clothing fabric, clothing pattern, clothing design elements (such as puff sleeves, etc.), clothing size, clothing version (such as slim, loose, etc.), etc.

[0123] Among them, each clothing image corresponds to a model image, that is, the clothing image and the model image are paired one-to-one, for example, the model in the model image wears the clothing corresponding to the clothing image, thereby collecting a paired data set of clothing images and model images.

[0124] Specifically, the training data set includes a large number of model images, wherein each model image is wearing the same or different clothes. The clothing images are tiled images facing up, and the model images are images taken in a frontal posture, wherein the minimum side width of the image size shall not be less than 512.

[0125] It is understandable that, in the actual process of collecting model images, there may be images of different sizes. Therefore, the embodiment of the present application also normalizes each model image to unify the size of each model image, for example, to a size of 1024*768. Alternatively, each model image may be subjected to operations such as illumination transformation and rotation for data enhancement, which helps to improve the robustness of the virtual fitting model.

[0126] In an embodiment of the present application, the method further includes:

[0127] Preprocess the model image, including:

[0128] The human body key points of the model image are extracted to obtain a human body posture diagram. Specifically, the key points of the model image are detected through a human body posture evaluation algorithm to obtain the portrait key points corresponding to the model image.

[0129] The human body posture assessment algorithm includes but is not limited to the human skeleton key point detection (OpenPose) algorithm. Preferably, the OpenPose algorithm is used in the embodiment of the present application.

[0130] The OpenPose algorithm is based on a real-time multi-person key point detection library for the body, face, hands, and feet. By inputting a model image, it detects the key points of the model image's body, face, hands, and feet, thereby obtaining the portrait key points corresponding to the model image, namely, the portrait key point map.

[0131] Specifically, the OpenPose algorithm is an algorithm that relies on convolutional neural networks and supervised learning to achieve human posture assessment. Its main advantage is that it is suitable for open source real-time systems for multi-person two-dimensional posture detection and can accurately and quickly identify key points of the human body.

[0132] Specifically, the OpenPose algorithm takes the entire person image as the input of the network, then predicts the confidence map for body part detection and the part affinity fields (PAF) for part association, then performs a set of two-part matching on the associated body part candidates through the parsing step, and finally builds the human skeleton to connect the human key points and assemble them into a complete pose for all people in the image, obtaining a human key point area map corresponding to each model image.

[0133] Among them, the human body key points are divided into 25 categories in total, each category is marked with 0-24, and the categories of human body key points are: {0, "nose"}, {1, "neck"}, {2, "right upper arm (RShoulder)"}, {3, "right elbow (RElbow)"}, {4, "right wrist (RWrist)"}, {5, "left upper arm (LShoulder)"}, {6, "left elbow (LElbow)"}, {7, "left wrist (LWrist)"}, {8, "middle hip (MidHip)"}, {9, "right hip (RHip)"}, {10, "right knee (RKnee)"}, {11, "right ankle (Rank le)"}, {12, "left hip (Lhip)"}, {13, "left knee (LKnee)"}, {14, "left ankle (LAnkle)"}, {15, "right eye (Reye)"}, {16, "left eye (Leye)"}, {17, "right ear (Rear)"}, {18, "left ear (Lear)"}, {19, "left big toe (LBigToe)"}, {20, "left little toe (LSmallToe)"}, {21, "left heel (LHeel)"}, {22, "right big toe (RBigToe)"}, {23, "right little toe (RSmallToe)"} {24, "right heel (Rheel)"}.

[0134] Specifically, the human body key point area map corresponding to the model image is obtained by extracting the portrait key points of the model image using the OpenPose algorithm, and the key points in the human body key point area map are connected according to the principle of human skeleton construction to form a human body posture map. Among them, the human body posture map is used to help the model accurately identify various parts of the human body during the training of the virtual fitting model, and achieve accurate clothing placement and fitting effects. At the same time, based on the human body posture map, the display mode of the clothing is automatically adjusted according to the user's body shape and movement to ensure that the clothing can fit the body during the fitting process, so that the generated fitting effect can achieve the unchanged human body posture and show a suitable effect.

[0135] In the embodiment of the present application, before training the virtual fitting model, the method further includes:

[0136] The model image is subjected to noise processing to obtain a noisy model image. Specifically, Gaussian noise is gradually added to the model image based on a Conditional Gaussian Process to obtain a noisy model image.

[0137] Among them, the Gaussian process is also called the conditional Gaussian process, which refers to a Gaussian process under certain observation data conditions. The Gaussian process is a random process on a continuous domain (such as time and space), in which each point is associated with a normally distributed random variable. The random variables involved in the Gaussian process obey the multivariate Gaussian distribution, which is defined by the mean vector and the covariance matrix. The covariance matrix describes the correlation between different random variables. The Gaussian process can simulate the cumulative effect of noise, that is, it can describe the distribution and change of noise at different time points or spatial positions.

[0138] Specifically, Gaussian noise is gradually added to the model image based on the Gaussian process, and the total number of steps T is set. In each step, Gaussian noise is added to the model image with a predetermined variance until the model image becomes blurred, wherein the Gaussian noise follows the Gaussian process, and the variance value of the noise is controlled by a variance schedule or a noise schedule. The predetermined variance gradually increases with the increase in the number of steps, so that the amount of noise added in each step is related to the current noise level of the image. The model image after adding noise includes:

[0139]

[0140] Among them, β t is the variance used in each t step, I is the mean, N is the normal distribution, X t is the t-th step based on β t Model image after adding Gaussian noise, Xt-1 is the t-1 step based on β t-1 Model image obtained after adding Gaussian noise, where β t The value of β is between 0 and 1, that is, t The value range is between 0 and 1, that is, β t ∈[0,1],β1<β2<…<β T .

[0141] The variance schedule specifies the variance value of the Gaussian noise in each diffusion step. Variance is a statistic that measures the degree of data dispersion. The larger the variance, the more discrete the data distribution, that is, the higher the noise level. As the number of steps increases, the variance gradually increases, which can ensure that in the process of gradually adding noise, the model image is covered by more noise and eventually becomes random noise.

[0142] The Noise Schedule specifies a specific plan for the amount and type of noise added at each step. It can precisely control the amount of noise added to the data at each step, thereby achieving a smooth transition from clean data to completely random noise.

[0143] For example, setting the total number of steps T to 10,000, for each step t, calculate the noise variance of the current step, generate a Gaussian noise of the same size as the original image, and add the generated Gaussian noise to the model image of the previous step to form a new noisy model image. The model image of the previous step is the model image of the t-1th step. After 10,000 steps of noise addition, the original model image is gradually covered by the noise and eventually becomes a completely random noise image, i.e., the noisy model image is obtained.

[0144] Please refer to Figure 3 , Figure 3 It is a schematic diagram of a process of adding noise to a model image provided by an embodiment of the present application;

[0145] like Figure 3 As shown, set the total number of steps T and randomly select a model image X from the training data set. o , based on the high-speed Katzon process, the model image X is gradually o Add Gaussian noise until the Tth step, model image X o Completely blur the image to get a noisy model image.

[0146] In the embodiment of the present application, the virtual fitting model is mainly based on the diffusion model, which is a generative model, including a forward diffusion process and a reverse denoising process. In the forward diffusion process, the model gradually adds noise to the image until the image becomes blurred. In the reverse denoising process, the model gradually reconstructs the image by learning denoising. The process of training the diffusion model is a two-way process from data to noise and then back to data.

[0147] Among them, adding noise to the model image is the forward diffusion process of the diffusion model, which simulates the transition of the image from clear to completely blurred. The noisy model image generated in each step is generated based on the noisy model image generated in the previous step. The process is regarded as a Markov chain, including:

[0148]

[0149] Among them, q(X 1:T |X o ) is the noisy model image obtained after the total number of steps T, X o is the model image, X t-1 is the model image obtained after adding Gaussian noise in step t-1, X t The model image obtained after adding Gaussian noise in the tth step.

[0150] Among them, Markov chain is used to describe the random process in which the result of the next event in a series of events depends only on the result of the previous event. Taking the diffusion process as a Markov chain, the state of each step (noisy data) depends only on the state of the previous step, that is, when the state of each step is determined, the state of the next step is predicted by probability distribution, without knowing the entire historical state sequence.

[0151] In an embodiment of the present application, the noisy model image is the data used in training the virtual fitting model. By inputting the noisy model image into the model, the model learns how to identify and restore the original model image from the noise, so that the model can better handle the uncertainty factors that may exist in the real world, so as to improve the generalization ability of the model.

[0152] Step S202: performing occlusion processing on the top area of ​​the model image to obtain a model reserved area map;

[0153] Specifically, the upper garment area of ​​the model image is identified and segmented by a parsing graph algorithm, and a mask is generated according to the identified upper garment area of ​​the model image, the mask being a binary image (i.e., only containing pixel values ​​of 0 and 1), and the mask is covered on the upper garment area of ​​the model image to block the upper garment area of ​​the model image, thereby obtaining a model reserved area map. The parsing graph algorithm includes a general human body parsing algorithm, such as a Graphonomy algorithm. By adopting a Graphonomy algorithm to segment the upper garment area of ​​the model image, an area in the model image where clothes need to be changed is obtained, and the upper garment area of ​​the model image is the area in the model image where clothes need to be changed.

[0154] Among them, the Graphonomy algorithm consists of two parts, namely intra-graph reasoning and inter-graph transfer. Specifically, the Graphonomy algorithm first learns and propagates compact high-level semantic graph representations in a dataset through intra-graph reasoning, and then transfers and fuses semantic information in multiple datasets through inter-graph transfer driven by an explicit hierarchical semantic label structure to obtain the human body parsing graph corresponding to each model image. By extracting the general semantic graph representation into each specific task, the Graphonomy algorithm can predict labels of all levels in a system without increasing complexity.

[0155] In the embodiment of the present application, each human body analysis diagram includes multiple area categories. For example, each human body analysis diagram includes 20 area categories, represented by serial numbers 0-19, and the 20 area categories are:

[0156] 0: Background; 1: Hat; 2: Hair; 3: Gloves; 4: Sunglasses; 5: Upper-clothes; 6: Dress; 7: Coat; 8: Socks; 9: Pants; 10: Torso-skin; 11: Scarf; 12: Skirt; 13: Face; 14: Left-arm; 15: Right-arm; 16: Left-leg; 17: Right-leg; 18: Left-shoe; 19: Right-shoe.

[0157] Step S203: extracting features of the clothing image through the upper channel network to obtain a clothing texture feature map;

[0158] Specifically, the clothing image is input into the upper channel network, and features of the clothing image are extracted through the upper channel network to obtain a clothing texture feature map.

[0159] The upper channel network includes at least two sequentially connected downsampling layers, each of which is used to downsample the clothing image to obtain at least two clothing texture feature maps of different scales. The convolution kernel used in the downsampling layer is 3*3, the step size is set to 2, the padding is 1, and the number of convolution kernels in each layer is set to 32, 64, 128, 256, and 512. The clothing image is input to the first downsampling layer of the upper channel network, and the clothing texture feature maps of 32, 64, 128, 256, and 512 channels are output successively through the sequentially connected downsampling layers.

[0160] It is understandable that the network structure of the downsampling layer in the embodiment of the present application can adjust the convolution kernel and / or step size according to actual needs, which is not limited here.

[0161] The upper channel network further includes at least two upsampling layers connected in sequence, each of which is used to perform upsampling operations on the clothing texture feature map to obtain at least two clothing texture feature maps of different scales. The upsampling layer uses the nearest neighbor interpolation method to perform upsampling operations on the clothing texture feature map, gradually doubles the size of the clothing texture feature map generated by downsampling, and splices the clothing texture feature map of the same scale generated by the downsampling layer with the clothing texture feature map enlarged by the current upsampling layer to retain more clothing texture information. The number of downsampling layers is the same as the number of upsampling layers.

[0162] In the embodiment of the present application, a residual structure is introduced between the downsampling layer and the upsampling layer to connect them, which can retain the original feature information and prevent gradient disappearance and gradient explosion. At the same time, residual connections are used in the upsampling process to restore the detail information lost in the downsampling process.

[0163] In the embodiment of the present application, the model parameters of the upper channel network structure adopt the pre-trained stable diffusion u-net parameters, and the u-net parameters include convolution layer parameters of different scales, such as the size of the convolution kernel, step size, padding, number of convolution kernels, activation function, etc. It should be noted that the u-net parameters of stable diffusion are obtained through pre-training, and the u-net parameters are directly used to extract features of clothing images through the upper channel network. Since there is no need to train the upper channel network, it is possible to reduce the demand for GPU resources for model training and improve the training speed of the virtual fitting model.

[0164] Step S204: performing feature encoding on the description text to generate a text feature map corresponding to the description text;

[0165] Among them, the description text of the clothing image includes the color of the clothing, fabric thickness, pattern design, etc. The description text of the clothing image is used to prompt the model to more accurately simulate the appearance and characteristics of the clothing during the virtual fitting process, thereby presenting a more realistic effect.

[0166] Specifically, the descriptive text of the clothing image is feature encoded through a multimodal pre-training model (Contrastive Language-Image Pre-training, CLIP) to generate a text feature map corresponding to the descriptive text.

[0167] The CLIP model includes a text encoder, which is used to extract features from the description text. Specifically, the description text of the clothing image is preprocessed to obtain the preprocessed text, for example, punctuation marks are removed, etc. The preprocessed text is feature encoded by the text encoder to generate a text feature vector, that is, a text feature map is generated, and the text feature map is represented by a text feature vector. The dimension of the text feature vector is configured according to actual needs, for example, the feature dimension of the CLIP model output is set to 1*1768.

[0168] In an embodiment of the present application, by inputting a text feature map corresponding to a description text into a virtual fitting model, the perception ability of the model can be enhanced, a more realistic virtual fitting effect can be generated, and the realism of the virtual fitting image can be improved.

[0169] Step S205: feature fusion is performed on the clothing texture feature map and the model reserved area map through the spatial attention mechanism layer to obtain a spatial attention map;

[0170] In an embodiment of the present application, the down-channel network includes at least two sequentially connected down-sampling layers and at least two sequentially connected up-sampling layers, wherein the down-sampling layer is used to down-sample the feature map input to the down-channel network to obtain at least two feature maps of different sizes.

[0171] Among them, each downsampling layer includes an attention mechanism layer and a convolution layer.

[0172] It is understandable that the attention mechanism layer adopts the attention mechanism for processing. The attention mechanism (Attention Mechanism) is a technology that simulates human vision and cognitive processes, enabling the machine to selectively focus on certain information and ignore unimportant information. By evaluating the importance of different parts of the input information to the current task, the key points are determined and a higher weight is given to the key points, so that the key points can get more attention.

[0173] Among them, the attention mechanism layer includes an attention mechanism module and an embedding layer, and the attention mechanism module includes a spatial attention mechanism layer and a cross attention mechanism layer. The spatial attention mechanism is mainly used to enable the virtual fitting model to adaptively focus on different areas in the image. The spatial attention mechanism assigns different attention weights to different positions of the image, so that the model can focus on areas with higher attention weights, thereby enabling the virtual fitting model to focus on the fitting area, that is, the area where the clothes are worn. The cross attention mechanism layer is used to combine images and description texts and calculate the correlation between images and description texts.

[0174] In an embodiment of the present application, the upper channel network is connected to the lower channel network, and the clothing texture feature map output by the upper channel network serves as the input of the lower channel network.

[0175] Specifically, the model reserved region map and the clothing texture feature map output by the upper channel network are input to the attention mechanism layer of the lower channel network, and the clothing texture feature map and the model reserved region map are feature fused by the spatial attention mechanism layer of the attention mechanism layer to obtain a spatial attention map. It can be understood that the clothing texture feature map input by the upper channel network is spliced ​​with the feature map of the same scale as the model reserved region map of the lower channel network.

[0176] In the embodiment of the present application, the clothing texture feature map and the model reserved area map are feature fused through the spatial attention mechanism layer to obtain a spatial attention map, including:

[0177] The model retention area map and clothing texture feature map of the same scale are spliced ​​to obtain the spatial attention map, which includes:

[0178] The model reserved area map and clothing texture feature map of the same scale are spliced ​​to obtain the spliced ​​feature map, and the spliced ​​feature map is respectively subjected to average pooling operation and maximum pooling operation to generate an average pooling two-dimensional map and a maximum pooling two-dimensional map. According to the average pooling two-dimensional map and the maximum pooling two-dimensional map, the average pooling two-dimensional map and the maximum pooling two-dimensional map are convolved, and then the convolved feature map is processed by an activation function to generate a spatial attention map, including:

[0179] M s (F) = σ(f a*a ([AvgPool(F); MaxPool(F)]))

[0180] Among them, M s (F) is the spatial attention map, F is the concatenated feature map, AvgPool is the average pooling two-dimensional map, MaxPool(F) is the maximum pooling two-dimensional map, and f a*a([AvgPool(F); MaxPool(F)]) is a convolution operation with a filter size of a*a, where a is a positive integer. For example, if a=7, the spatial attention is calculated as follows:

[0181]

[0182] Among them, σ represents the sigmoid function, f 7×7 Represents a convolution operation with a filter size of 7 × 7. F represents the concatenation of the feature maps of the same scale corresponding to the upper and lower channel attention u-net networks.

[0183] Among them, the average pooling operation is used to divide the spliced ​​feature map into several rectangular areas, and average the elements in each area, and use the average value as the output of the area. Specifically, the size of the average pooling window is determined, and the average pooling window is slid from left to right and from top to bottom over the entire spliced ​​feature map, and the average value of all pixel values ​​in the average pooling window is calculated, and the average value is placed in the corresponding position in the new image, that is, the average value of each average pooling window is used as the pixel value of its corresponding window, and an average pooling two-dimensional image is obtained.

[0184] Among them, the maximum pooling operation (Max Pooling) is used to divide the spliced ​​feature map into several rectangular areas, and select the maximum value in each area as the output of the area. Specifically, the size of the maximum pooling window is determined, and the maximum pooling window is slid from left to right and from top to bottom over the entire spliced ​​feature map to determine the maximum value of all pixels in each window, and the maximum value is placed in the corresponding position in the new image, that is, the maximum value of each maximum pooling window is used as the pixel value of its corresponding window to obtain the maximum pooling two-dimensional image.

[0185] For example, assuming that the size of the spliced ​​feature map is 4*4, the spliced ​​feature map is:

[0186]

[0187] Using a 2*2 average pooling window, the average pooling operation is performed on the spliced ​​feature map, and the average pooling two-dimensional map obtained is:

[0188]

[0189] Use a 2*2 maximum pooling window to perform a maximum pooling operation on the spliced ​​feature map, and the maximum pooling two-dimensional map is obtained as follows:

[0190]

[0191] The maximum pooling operation emphasizes the local maximum of the spliced ​​feature map, which can capture the edges and important features in the spliced ​​feature map. The average pooling operation can smooth the spliced ​​feature map to retain the overall texture and color information of the spliced ​​feature map. The average pooled two-dimensional map and the maximum pooled two-dimensional map are then convolved through the convolution operation to obtain a spatial attention map of the same size as the spliced ​​feature map.

[0192] By performing average pooling and maximum pooling operations on the concatenated feature maps, different types of feature information can be retained, and convolution operations are performed on the average pooled two-dimensional map and the maximum pooled two-dimensional map through the convolution layer to keep the size of the feature map unchanged.

[0193] In the embodiment of the present application, the spatial attention map obtained by fusion of the spatial attention mechanism layer is a 2D spatial attention map, which has characteristic information of spatial attributes. It can be understood that generating a 2D spatial attention map enables the virtual fitting model to better allocate computing resources and improve the processing efficiency of the virtual fitting model. In addition, in virtual try-on or image synthesis, spatial attention can help the model accurately align features from different sources (such as human body and clothing) to ensure the naturalness and realism of the generated image.

[0194] In the embodiment of the present application, the convolution operation is performed on the average pooled two-dimensional image and the maximum pooled two-dimensional image to generate the spatial attention map, and then the time step is converted into a continuous vector through the embedding layer, and the continuous vector corresponding to the time step is added to the output of the attention mechanism module. Among them, the time step is input to the lower channel network to indicate the degree of denoising, so that the virtual fitting model considers the current processing stage while considering the spatial attention and cross attention, and adjusts the attention allocation according to the different stages of denoising to achieve more refined control and generation effects.

[0195] In an embodiment of the present application, an embedding layer is added to the network structure of the diffusion model so that the network structure of the diffusion model is combined with the embedding time, so that the model generation process can be more finely controlled to improve the realism and detail expression of the generated effect.

[0196] In an embodiment of the present application, the maximum pooling operation and the average pooling operation are performed on the spliced ​​feature map through the spatial attention mechanism layer, so that the virtual fitting model can automatically learn the most important areas for the final task in the input spliced ​​feature map, such as detecting and segmenting the upper body area, and give higher weights to important areas, thereby highlighting the key information of the upper body area and suppressing irrelevant or noise information. At the same time, the key areas in the spliced ​​feature map are determined by weight, which can more effectively allocate computing resources. That is, when processing complex images, the virtual fitting model can focus on processing key areas without uniformly processing every pixel of the image, thereby improving efficiency, and the important areas of the feature map can be intuitively displayed through the spatial attention map, so that the model can more easily capture the features of important areas to make decisions.

[0197] The spliced ​​feature map includes the features of the model image and the clothing image. The spatial attention mechanism layer can extract the features of the model image and the clothing image, and further accurately align the fitting area of ​​the model image with the clothing image to ensure the naturalness and realism of the generated image.

[0198] Step S206: performing feature fusion on the spatial attention map and the text feature map through a cross attention mechanism layer to obtain a fused feature map;

[0199] Specifically, the spatial attention map and the text feature map are input into the cross attention mechanism layer, and the spatial attention map and the text feature map are fused through the cross attention mechanism layer to obtain a fused feature map.

[0200] Among them, the spatial attention map x1 and the text feature map x2 represent two different modalities. In the cross attention, the spatial attention map x1 is regarded as a query set, and the text feature map x2 is regarded as a key-value set. x1 and x2 are vector matrices, n is the number of rows in the matrix, d1 and d2 are the dimensions of x1 and x2 respectively, let Q be the query vector, K be the key vector, V be the value vector, let Q = x1W Q , K=V=x2W K ,in, W Q , W K is the weight matrix, W Q Used to transform the spatial attention map x1 into a query vector Q, W K It is used to convert the text feature map x2 into a key vector K and a value vector V, so that the query vector, key vector, and value vector can be better matched and compared. The attention between the spatial attention map x1 and the text feature map x2 is calculated through the cross attention mechanism layer, including:

[0201]

[0202] Among them, x1 is the spatial attention map, x2 is the text feature map, Q is the query vector corresponding to x1, K is the key vector corresponding to x2, V is the value vector corresponding to x2, and T is the time step.

[0203] The query set is used as the input of the attention weight to calculate the degree of association between the spatial attention map and the text feature map. The key-value set is used to provide information input and is compared with the features in the query set to determine the degree of association between the spatial attention map and the text feature map.

[0204] By using query sets and key-value sets, we can establish associations between different modalities and calculate attention weights between different modalities. By weighting the information between different modalities through attention weights, we can better understand and utilize the semantic relationships between multimodal data.

[0205] Step S207: decoding the fused feature map through the upsampling layer of the lower channel network to obtain a denoised feature map, and then performing several denoising steps to obtain a fitting effect map;

[0206] Specifically, the fused feature map is input into the upsampling layer of the down-channel network, the fused feature map is decoded by the downsampling layer, and goes through several denoising steps to obtain a fitting effect map.

[0207] Among them, the fused feature map includes a large amount of noise. The fused feature map is decoded through the upsampling layer, that is, the fused feature map is iterated multiple times. Each iteration removes part of the noise in the fused feature map, and the denoised fused feature map obtained in the current iteration is input into the upsampling layer of the lower channel network. The denoising steps are continued for several times. After multiple iterations, the fused feature map is gradually restored to a clear image, that is, a fitting effect picture is obtained.

[0208] Iterating the fused feature map multiple times is essentially the reverse diffusion process of the diffusion model. Through the reverse process, the target fitting image is gradually restored from the noise data, that is, the fused feature map is gradually restored to the fitting effect image. For example, the denoising steps for the model image after adding noise are performed t times, which specifically include:

[0209] p θ (X t-1 |X t )=N(X t-1 |μ θ (X t , P, R, C), 0)

[0210] Among them, X t is the model image after adding Gaussian noise in step t, P is the human body posture map, R is the model reserved area map, C is the clothing image, μθ (X t , P, R, C) are the estimated X in the denoising process t-1 The mean function, N(X t-1 |μ θ (X t , P, R, C), 0) is X t-1 At a given X t Under the normal distribution, For X t-1 Noise, t is the step length, the value of t ranges from 1 to T, and T is the maximum step length;

[0211]

[0212] Among them, X t is the model image after adding Gaussian noise in step t, P is the human body posture map, R is the model reserved area map, C is the clothing image, For X t-1 The noise is, where t is the step length, the value of t ranges from 1 to T, T is the maximum step length, α t A hyperparameter that adjusts the strength of the denoising process.

[0213] In an embodiment of the present application, the fused feature map is denoised several times through the upsampling layer of the lower channel network to gradually restore the fitting effect image, so that the virtual fitting model can be more robust in dealing with changes and noise in the real world, and a clear image is restored in a step-by-step iterative manner, which helps the virtual fitting model to significantly improve the image quality when dealing with complex noise.

[0214] Step S208: constructing a loss function, and training the lower channel network based on the training data set until the loss function converges, so as to generate a trained virtual fitting model;

[0215] Specifically, a loss function is constructed, and the lower channel network is trained based on the training data set until the loss function converges, thereby obtaining the trained lower channel network, that is, obtaining the trained virtual fitting model.

[0216] Please refer to Figure 4 , Figure 4 yes Figure 2 A detailed flow chart of step S208 in FIG.

[0217] like Figure 4 As shown, step S208 includes:

[0218] Step S281: constructing a loss function;

[0219] Specifically, the loss function is used to calculate the loss between the prediction noise and the random noise. The loss function includes:

[0220]

[0221] in, is the mean square error loss of random noise and the noise predicted by the lower channel network, ∈ is random noise, is a hyperparameter, Y o is the sample data, The noise predicted for the lower channel network, is the mean squared error between random noise and the noise predicted by the lower channel network.

[0222] in, It is used to update the parameters of the virtual fitting model to minimize the difference between random noise and the noise predicted by the lower channel network, which can improve the accuracy of the virtual fitting model, improve the realism of the fitting images, and improve the user experience.

[0223] Step S282: Obtain sample data from the training data set;

[0224] Specifically, a loss function is constructed, and sample data is obtained from a training data set, where the sample data includes a clothing image, a description text of the clothing image, and a model image matching the clothing image.

[0225] Step S283: iteratively training the virtual fitting model based on the loss function and sample data;

[0226] Specifically, the virtual fitting model is iteratively trained based on the loss function and sample data, including: preprocessing the sample data to obtain a text feature map, a clothing texture feature map, a human posture map, a model reserved area map, and a noisy model image; inputting the text feature map, the clothing texture feature map, the human posture map, the model reserved area map, and the noisy model image into a lower channel network; learning to predict noise through the lower channel network, and calculating the loss between the predicted noise and the random noise.

[0227] Step S284: whether the loss of the virtual fitting model is less than the loss threshold;

[0228] Specifically, determine whether the loss of the currently calculated virtual fitting model is less than the loss threshold. If the loss of the virtual fitting model is less than the loss threshold, it means that the difference between the predicted result of the virtual fitting model and the real data is small, and the training is terminated. If the loss of the virtual fitting model is greater than or equal to the loss threshold, jump to step S282 and obtain sample data from the training data set again to continue training the virtual fitting model until the loss function converges.

[0229] Preferably, the embodiment of the present application uses the Adam algorithm (Adaptive Moment Estimation Algorithm) to optimize the model parameters. For example, the number of iterations is set to 500, the initialization learning rate is set to 0.001, the weight decay is set to 0.0005, and the learning rate decays to 1 / 10 of the original after every 50 iterations.

[0230] It can be understood that the Adam algorithm (Adaptive Moment Estimation Algorithm) can be seen as a combination of the momentum method and the RMSprop algorithm. It not only uses momentum as the parameter update direction, but also can adaptively adjust the learning rate.

[0231] Specifically, it is determined whether the number of iterations is greater than a first number threshold, which is preset, for example, 500 times. If the number of iterations is greater than the first number threshold, the training is determined to be complete; if the number of iterations is not greater than the first number threshold, sample data is obtained from the training data set again for training. It can be understood that the first number threshold is specifically set according to specific needs and is not limited here.

[0232] When the number of iterations is not greater than the first number threshold, it is further determined whether the loss of the virtual fitting model is less than the loss threshold. If so, the training is determined to be completed. If not, sample data is obtained from the training data set again for training.

[0233] In the embodiment of the present application, the loss threshold can be set to 0.0005 or 0.001. It can be understood that the loss threshold is set according to specific needs and is not limited here.

[0234] Step S285: Training is completed.

[0235] It is understandable that after the training is completed, a trained virtual fitting model is obtained. At this time, the trained virtual fitting model can be called to predict the fitting effect image corresponding to the clothing image to be tried on and the model image.

[0236] In an embodiment of the present application, a loss function is constructed and the difference between the random noise and the noise predicted by the lower channel network is calculated to adjust the difference to improve the effect of the virtual fitting model prediction, thereby further improving the quality of the fitting effect picture, so that the fitting effect picture can show the fit between the skin and clothes, the fit between the skin and clothes is more natural, and the texture of the clothes is clearer.

[0237] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the overall structure of generating a fitting effect diagram provided by an embodiment of the present application;

[0238] like Figure 5 As shown, the overall structure of generating the fitting effect diagram, namely the virtual fitting model, includes an encoder, a text editor, an upper channel network, a lower channel network, and a decoder;

[0239] Among them, the encoder is used to encode the clothing image and the model reserved area map, for example: converting the clothing image and the model reserved area map into a digital matrix composed of pixels, so that the upper channel network and the lower channel network can efficiently process and understand the image information, wherein each pixel has a numerical value associated with it, representing the brightness, chromaticity or other image attributes of the pixel.

[0240] Specifically, the model retention area map input to the encoder is obtained by occluding the upper body area of ​​the model image through the analytical graph algorithm. Specifically, the upper body area of ​​the model image is identified and segmented, and a mask with the same size as the upper body area of ​​the model image is generated according to the identified upper body area of ​​the model image. The mask is covered on the upper body area of ​​the model image to occlude the upper body area of ​​the model image, thereby obtaining the model retention area map.

[0241] The upper channel network includes at least two layers of downsampling layers connected in sequence and at least two layers of upsampling layers connected in sequence. A clothing image is input into the upper channel network, and features of the clothing image are extracted through the downsampling layer of the upper channel network to output clothing texture feature maps of different sizes. The clothing texture feature map output by the downsampling layer of the upper channel network is upsampled through the upsampling layer of the upper channel network to obtain at least two clothing texture feature maps of different scales. At the same time, the clothing texture feature map output by each upsampling layer is spliced ​​with the feature map output by the downsampling layer of the same level to retain more clothing texture features.

[0242] In an embodiment of the present application, the upper channel network is connected to the lower channel network, and the clothing texture feature map output by the upper channel network serves as the input of the lower channel network.

[0243] In an embodiment of the present application, the feature map input to the lower channel network also includes a human posture map. Specifically, the human body key points of the model image are extracted through a human posture evaluation algorithm to obtain the human body posture map.

[0244] Among them, the text encoder is used to perform feature encoding on the description text of the clothing image to generate a text feature map.

[0245] In an embodiment of the present application, the feature map input to the lower channel network also includes a noisy model image. Specifically, Gaussian noise is gradually added to the model image based on a high-speed Casson process until the model image is completely blurred to obtain a noisy model image.

[0246] The clothing texture feature map, the model reserved area map, the human body posture map, the text feature map, and the noisy model image are input into the down-channel network to obtain a fitting effect map, wherein the down-channel network includes at least two downsampling layers connected in sequence and at least two upsampling layers connected in sequence, each downsampling layer includes an attention mechanism layer and a convolution layer, the attention mechanism layer includes an attention mechanism module and an embedding layer, and the attention mechanism module includes a spatial attention mechanism layer and a cross attention mechanism layer.

[0247] Specifically, the model reserved area map and the clothing texture feature map are spliced ​​to obtain a spliced ​​feature map, and the spliced ​​feature map is subjected to average pooling and maximum pooling operations respectively through the spatial attention mechanism layer to obtain an average pooled two-dimensional map and a maximum pooled two-dimensional map, and the average pooled two-dimensional map and the maximum pooled two-dimensional map are subjected to convolution operations to obtain a spatial attention map. Among them, the clothing texture feature map input by the upper channel network is spliced ​​with the feature map of the same scale as the model reserved area map of the lower channel network. Figure 5 As shown, multiple clothing texture feature maps of different scales generated by the upper channel network are spliced ​​with the model reserved area maps of the same scale corresponding to the lower channel network to fuse the clothing texture feature maps of different scales extracted by the upper channel network.

[0248] The noisy model image, human posture map, and time step are embedded into the spatial attention mechanism layer through the embedding layer, so that the spatial attention mechanism layer can combine the feature information of multiple input data, so that the spatial attention mechanism layer can give higher weights to areas with higher attention points, and make the upper part of the clothes in the output spatial attention map fit the human body better.

[0249] Specifically, the spatial attention map and the text feature map are fused through the cross-attention mechanism layer, so that the lower channel network can learn more information about the clothing image through the text feature map, adjust the spatial attention map, and obtain the fused feature map.

[0250] Furthermore, the fused feature map is decoded through the upsampling layer of the lower channel network to perform multiple denoising on the fused feature map to obtain a denoised image, and the denoised image is input into the decoder for decoding to obtain a fitting effect image.

[0251] It can be understood that the decoder is used to convert the encoded digital image signal into an analog signal of a visible image so as to be presented on a display device.

[0252] In the embodiment of the present application, the feature map output by each upsampling layer of the lower channel network is spliced ​​with the feature map output by the downsampling layer at the same level, and the detail information restored by the upsampling layer and the information extracted by the downsampling layer can be fused to enhance the expressiveness of the feature map, so that the model can make more accurate judgments when processing real image data. At the same time, as the number of network layers increases, feature information will gradually be lost. By splicing feature maps between different levels, the underlying features can be effectively propagated to deeper layers to enhance the perception ability of the model.

[0253] Please refer to Figure 6 , Figure 6 It is a structural diagram of an attention mechanism layer provided in an embodiment of the present application;

[0254] like Figure 6 As shown, the attention mechanism layer includes an attention mechanism module and an embedding layer, and the attention mechanism module includes a spatial attention mechanism layer, a cross attention mechanism layer, and a convolutional layer.

[0255] Among them, the embedding layer is used to convert the feature map input to the lower channel network into a high-dimensional vector. For example, the input model-reserved area map is converted into a high-dimensional vector representation suitable for the model to facilitate the subsequent lower channel network to perform feature fusion on images corresponding to different high-dimensional vectors.

[0256] The embedding layer is also used to inject the noisy model image into the lower channel network, so that the model can predict the noise from the noisy model image to gradually remove the noise and restore a clear image.

[0257] It can be understood that the embedding layer is the Embedding layer. In the Diffusion Models, the main function of the Embedding layer is to convert the input data (such as images, text, etc.) into a high-dimensional vector representation suitable for the model. This representation can capture the complex characteristics and structure of the input data, which is convenient for subsequent processing and modeling. In the diffusion model, the Embedding layer is sometimes also used to inject noise into the feature representation. This is part of the diffusion process, which learns the generative distribution of the data by gradually adding noise. The main role of the Embedding layer in the diffusion model is to convert the original input data into a high-dimensional feature representation so that the model can effectively process and generate data.

[0258] Among them, the attention mechanism layer is used to splice the input human posture map, model retention area map, and clothing texture feature map to obtain the spliced ​​feature map, and perform average pooling operations and maximum pooling operations on them respectively through the spatial attention mechanism layer of the attention mechanism layer to obtain the average pooling two-dimensional map and the maximum pooling two-dimensional map, and perform convolution operations on the average pooling two-dimensional map and the maximum pooling two-dimensional map to obtain the spatial attention map, and perform feature fusion on the input text feature map and the spatial attention map through the cross attention mechanism layer to obtain the fused feature map.

[0259] In an embodiment of the present application, the feature map and text data are input into the spatial attention mechanism layer and the cross attention mechanism layer, which can highlight the features of the key areas, and can learn the features of the original image from the text data, providing the model with the key features of the image so as to restore the image after adding noise.

[0260] In an embodiment of the present application, features of clothing images are extracted through an upper channel network, and noise is gradually reduced through a lower channel network to obtain a fitting effect image, which can improve the resolution of images generated by a virtual fitting model, and make the clothing image more closely aligned with the human body, thereby enhancing the fitting effect.

[0261] After training the lower channel network, the present application also includes a testing phase. In the testing phase, the clothing image and the model image are input into the virtual fitting model to obtain a fitting image corresponding to the clothing image to be fitted.

[0262] Among them, the model image is an image of the model's front posture. By selecting different clothes for virtual fitting, under different posture and light environments, the reverse process reasoning of the lower channel network is used to gradually remove the noise of the model image, so that the network can adaptively complete diffusion and inverse diffusion to generate fitting effect pictures, so as to improve the quality and clarity of the image, and achieve a more realistic and natural fitting effect.

[0263] The embodiment of the present application provides a training method for a virtual fitting model, wherein the virtual fitting model includes a dual-channel attention network, wherein the dual-channel attention network includes an upper channel network and a lower channel network, wherein the lower channel network includes a spatial attention mechanism layer, a cross attention mechanism layer, and an upsampling layer, and the method includes: obtaining a training data set, wherein the training data set includes a clothing image, a description text of the clothing image, and a model image matching the clothing image; performing occlusion processing on the upper clothing area of ​​the model image to obtain a model reserved area map; performing feature extraction on the clothing image through the upper channel network to obtain a clothing texture feature map; and performing occlusion processing on the description text. The feature encoding is performed to generate a text feature map corresponding to the description text; the clothing texture feature map and the model reserved area map are fused through the spatial attention mechanism layer to obtain the spatial attention map; the spatial attention map and the text feature map are fused through the cross attention mechanism layer to obtain the fused feature map; the fused feature map is decoded through the upsampling layer of the lower channel network to obtain a denoised feature map, and after several denoising steps, a fitting effect map is obtained; a loss function is constructed, and the lower channel network is trained based on the training data set until the loss function converges to generate a trained virtual fitting model.

[0264] On the one hand, by constructing a dual-channel network to extract features of clothing images, model images, and description texts of clothing images and perform denoising processing to restore the fitting effect images, the clarity of the effect images can be improved.

[0265] On the other hand, by constructing a loss function for training and training the virtual fitting network based on the training data set to generate a virtual fitting model, the fitting effect can be enhanced.

[0266] Embodiment 2

[0267] After the training method of the virtual fitting model provided in the above embodiment is used, a trained virtual fitting model is obtained, and the virtual fitting model can be used to perform virtual fitting.

[0268] It can be understood that the above-mentioned embodiment 1 is a training stage of the virtual fitting model, and the embodiment 2 of the present application is a use stage of the virtual fitting model to obtain a virtual fitting image.

[0269] For details, please refer to Figure 7 , Figure 7 It is a flowchart of a virtual fitting method provided in an embodiment of the present application;

[0270] like Figure 7 As shown, the virtual fitting method comprises:

[0271] Step S701: Acquire an image of clothing to be tried on, an image of a user, and a description text of the image of clothing to be tried on;

[0272] Specifically, after obtaining the virtual fitting model, an image of clothes to be fitted, an image of a user, and a description text of the image of clothes to be fitted are obtained.

[0273] The virtual fitting model is obtained by training the dual-channel attention network in the above embodiment. The virtual fitting model has the same structure and function as the dual-channel attention network, and processes the training data set in the same way as the dual-channel attention network in the above embodiment. The specific process will not be repeated here.

[0274] Step S702: inputting the image of the clothes to be tried on, the user image, and the description text of the image of the clothes to be tried on into the virtual fitting model, and obtaining a fitting effect diagram corresponding to the image of the clothes to be tried on and the model image;

[0275] Specifically, the upper clothing area of ​​the user image is occluded to obtain a user reserved area map, which is also input into the virtual fitting model. Figure 5 As shown, after the clothing image to be tried on is encoded by the encoder, it is input into the upper channel network of the virtual fitting model; after the user reserved area map is encoded by the encoder, it is input into the lower channel network of the virtual fitting model; after the description text of the clothing image to be tried on is passed through the text editor, a text feature map is obtained, and the text feature map is input into the lower channel network of the virtual fitting model; after the user image is denoised, a noisy user image is obtained, and the noisy user image is input into the lower channel network of the virtual fitting model.

[0276] It should be noted that the specific processing process of the feature maps input into the upper channel network and the lower channel network, as well as the related processing processes of the encoder, the upper channel network, the lower channel network, the text editor, and the decoder can refer to the processing process mentioned in the above embodiment 1, and will not be repeated here.

[0277] It should be noted that the virtual fitting model in the embodiment of the present application is a model obtained by training in the above-mentioned embodiment 1. The relevant content of the virtual fitting model can refer to the content mentioned in the above-mentioned embodiment and will not be repeated here.

[0278] In an embodiment of the present application, a virtual fitting model is trained, and after the trained virtual fitting model is obtained, an image of clothes to be tried on, a user image, and a description text of the image of clothes to be tried on are obtained, and the image of clothes to be tried on, the user image, and the description text of the image of clothes to be tried on are input into the virtual fitting model to obtain a fitting image corresponding to the image of clothes to be tried on. The present application can improve the virtual fitting effect.

[0279] Embodiment 3

[0280] See also Figure 8 , Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0281] like Figure 8 As shown, the electronic device 800 includes one or more processors 801 and a memory 802. Figure 8 A processor 801 is taken as an example.

[0282] The processor 801 and the memory 802 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0283] The processor 801 is used to provide computing and control capabilities to control the electronic device 800 to perform corresponding tasks, for example, to control the electronic device 800 to perform the training method of the virtual fitting model in any one of the above method embodiments, the virtual fitting model includes a dual-channel attention network, the dual-channel attention network includes an upper channel network and a lower channel network, the lower channel network includes a spatial attention mechanism layer, a cross attention mechanism layer, and an upsampling layer, including:

[0284] A training data set is obtained, wherein the training data set includes a clothing image, a description text of the clothing image, and a model image matching the clothing image; an upper clothing area of ​​the model image is occluded to obtain a model reserved area map; a feature extraction is performed on the clothing image through an upper channel network to obtain a clothing texture feature map; feature encoding is performed on the description text to generate a text feature map corresponding to the description text; feature fusion is performed on the clothing texture feature map and the model reserved area map through a spatial attention mechanism layer to obtain a spatial attention map; feature fusion is performed on the spatial attention map and the text feature map through a cross attention mechanism layer to obtain a fused feature map; the fused feature map is decoded through an upsampling layer of a lower channel network to obtain a denoised feature map, and after several denoising steps, a fitting effect map is obtained; a loss function is constructed, and the lower channel network is trained based on the training data set until the loss function converges to generate a trained virtual fitting model.

[0285] On the one hand, by constructing a dual-channel network to extract features of clothing images, model images, and description texts of clothing images and perform denoising processing to restore the fitting effect images, the clarity of the effect images can be improved.

[0286] On the other hand, by constructing a loss function for training and training the virtual fitting network based on the training data set to generate a virtual fitting model, the fitting effect can be enhanced.

[0287] Alternatively, controlling the electronic device 800 to execute the virtual fitting method in any one of the above method embodiments includes:

[0288] Obtaining an image of clothes to be tried on, an image of a user, and a description text of the image of clothes to be tried on;

[0289] The image of clothes to be tried on, the user image, and the description text of the image of clothes to be tried on are input into a virtual fitting model to obtain a fitting effect diagram corresponding to the image of clothes to be tried on and the image of the model, wherein the virtual fitting model is trained based on the method of embodiment one.

[0290] By training a virtual fitting model, after obtaining the trained virtual fitting model, obtaining the image of clothes to be tried on, the user image, and the description text of the image of clothes to be tried on, inputting the image of clothes to be tried on, the user image, and the description text of the image of clothes to be tried on into the virtual fitting model, and obtaining a fitting image corresponding to the image of clothes to be tried on. The present application can improve the virtual fitting effect.

[0291] The processor 801 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a hardware chip or any combination thereof; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above-mentioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0292] The memory 802, as a non-transient computer-readable storage medium, can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the training method of the virtual fitting model or the program instructions / modules corresponding to the virtual fitting method in the embodiment of the present application. The processor 801 can implement the training method of the virtual fitting model or the virtual fitting method in any of the above method embodiments by running the non-transient software programs, instructions and modules stored in the memory 802. Specifically, the memory 802 may include a volatile memory (volatile memory, VM), such as a random access memory (random access memory, RAM); the memory 802 may also include a non-volatile memory (non-volatile memory, NVM), such as a read-only memory (read-only memory, ROM), a flash memory (flash memory), a hard disk drive (hard disk drive, HDD) or a solid-state drive (solid-state drive, SSD) or other non-transient solid-state storage devices; the memory 802 may also include a combination of the above types of memories.

[0293] The memory 802 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 802 may optionally include a memory remotely arranged relative to the processor 801, and these remote memories may be connected to the processor 801 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0294] One or more modules are stored in the memory 802, and when executed by one or more processors 801, the training method of the virtual fitting model or the virtual fitting method in any of the above method embodiments is executed, for example, the training method of the virtual fitting model or the virtual fitting method described above is executed. Figure 2 or Figure 7 The steps shown.

[0295] In the embodiment of the present application, the electronic device 800 may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output. The electronic device 800 may also include other components for realizing device functions, which will not be described in detail here.

[0296] The electronic device of the embodiment of the present application exists in various forms. Figure 2 or Figure 7 The various steps shown include but are not limited to: servers, mobile terminals, electronic watches, fixed terminals, wearable devices and other devices.

[0297] The embodiment of the present application also provides a non-volatile computer-readable storage medium, such as a memory including a program code, and the program code can be executed by a processor to complete the training method of the virtual fitting model or the virtual fitting method in the above embodiment. For example, the non-volatile computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CDROM), a magnetic tape, a floppy disk, and an optical data storage device.

[0298] The embodiment of the present application also provides a computer program product, which includes one or more program codes, and the program codes are stored in a non-volatile computer-readable storage medium. The processor of the electronic device reads the program code from the non-volatile computer-readable storage medium, and the processor executes the program code to complete the method steps of the training method of the virtual fitting model or the virtual fitting method provided in the above embodiment.

[0299] A person skilled in the art will appreciate that all or part of the steps for implementing the above embodiments may be accomplished by hardware or by hardware associated with a program code, and the program may be stored in a non-volatile computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0300] Through the description of the above implementation methods, a person of ordinary skill in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, can also be implemented by hardware. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0301] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Under the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present application as mentioned above, which are not provided in detail for the sake of simplicity. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A training method for a virtual fitting model, characterized in that: The virtual fitting model includes a dual-channel attention network, the dual-channel attention network includes an upper channel network and a lower channel network, the lower channel network includes a spatial attention mechanism layer, a cross attention mechanism layer, and an upsampling layer, and the method includes: Acquire a training data set, wherein the training data set includes clothing images, description texts of the clothing images, and model images matching the clothing images; Performing masking processing on the top area of ​​the model image to obtain a model reserved area map; Performing feature extraction on the clothing image through the upper channel network to obtain a clothing texture feature map; Performing feature encoding on the description text to generate a text feature map corresponding to the description text; Performing feature fusion on the clothing texture feature map and the model reserved area map through the spatial attention mechanism layer to obtain a spatial attention map; Performing feature fusion on the spatial attention map and the text feature map through the cross attention mechanism layer to obtain a fused feature map; The fused feature map is decoded through the upsampling layer of the lower channel network to obtain a denoised feature map, and after several denoising steps, a fitting effect map is obtained; A loss function is constructed, and the lower channel network is trained based on the training data set until the loss function converges to generate a trained virtual fitting model.

2. The method according to claim 1, characterized in that The upper channel network includes at least two sequentially connected downsampling layers, each of which is used to perform a downsampling operation on the clothing image to obtain at least two clothing texture feature maps of different scales; The upper channel network also includes at least two upsampling layers connected in sequence, each of which is used to perform an upsampling operation on the clothing texture feature map to obtain at least two clothing texture feature maps of different scales.

3. The method according to claim 2, characterized in that The lower channel network further comprises at least two sequentially connected downsampling layers, each of which is used to downsample the model-reserved region map to obtain at least two model-reserved region maps of different scales; The step of fusing the clothing texture feature map and the model reserved area map through the spatial attention mechanism layer to obtain a spatial attention map includes: The model retention area map and clothing texture feature map of the same scale are spliced ​​to obtain the spatial attention map.

4. The method according to claim 3, characterized in that: The model reserved area map and clothing texture feature map of the same scale are spliced ​​to obtain a spatial attention map, including: The model reserved area map and clothing texture feature map of the same scale are spliced ​​to obtain a spliced ​​feature map; Performing an average pooling operation and a maximum pooling operation on the spliced ​​feature maps respectively to generate an average pooling two-dimensional map and a maximum pooling two-dimensional map; Generate a spatial attention map according to the average pooled two-dimensional map and the maximum pooled two-dimensional map, specifically including: M s (F)=σ(f a*a ([AvgPool(F);MaxPool(F)])) Among them, M s (F) is the spatial attention map, F is the concatenated feature map, AvgPool is the average pooling two-dimensional map, MaxPool(F) is the maximum pooling two-dimensional map, and f a*a ([AvgPool(F); MaxPool(F)]) is a convolution operation with a filter size of a*a, where a is a positive integer.

5. The method according to claim 3, characterized in that: The downsampling layer includes an attention mechanism layer, the attention mechanism layer includes an attention mechanism module and an embedding layer, the attention mechanism module includes a spatial attention mechanism layer, a cross attention mechanism layer and a convolution layer; wherein, The embedding layer is used to convert the feature map input to the lower channel network into a high-dimensional vector.

6. The method according to claim 1, characterized in that The step of performing noise processing on the model image comprises: The model image is subjected to t noise adding steps, specifically comprising: Among them, β t is the variance used in each t step, I is the mean, N is the normal distribution, X t is the t-th step based on β t Model image after adding Gaussian noise, X t-1 is the t-1 step based on β t-1 The model image obtained after adding Gaussian noise, where t is the step size, the value of t ranges from 1 to T, T is the maximum step size, β t The value of is between 0 and 1, β1<β2<…<β T .

7. The method according to claim 1, characterized in that The method further comprises: Extracting key points of the human body of the model image to obtain a human body posture graph; The human body posture graph is input into the lower channel network.

8. The method according to claim 7, characterized in that The step of performing denoising on the noisy model image through the lower channel network to generate a fitting effect image includes: The denoising steps are performed on the model image after adding noise for t times, specifically including: p θ (X t-1 |X t )=N(X t-1 |μ θ (X t ,P,R,C),0) Among them, X t is the model image after adding Gaussian noise in step t, P is the human body posture map, R is the model reserved area map, C is the clothing image, μ θ (X t , P, R, C) are the estimated X in the denoising process t-1 The mean function, N(X t-1 |μ θ (X t , P, R, C), 0) is X t-1 At a given X t Under the normal distribution, For X t-1 Noise, t is the step length, the value of t ranges from 1 to T, and T is the maximum step length; Among them, X t is the model image after adding Gaussian noise in step t, P is the human body posture map, R is the model reserved area map, C is the clothing image, For X t-1 The noise is, where t is the step length, the value of t ranges from 1 to T, T is the maximum step length, α t A hyperparameter that adjusts the strength of the denoising process.

9. The method according to claim 7, characterized in that: The constructing of the loss function, based on the image data set, training the lower channel network, comprises: Acquire sample data from the training data set, the sample data including a clothing image, a description text of the clothing image, and a model image matching the clothing image; adding random noise to the model image to obtain a noisy model image; Inputting the noisy model image, the model reserved area map, the clothing texture feature map, the text feature map, and the human body posture map into the lower channel network to generate prediction noise through the lower channel network; Calculating the loss between the prediction noise and the random noise; Repeatedly obtain sample data for training until the calculated loss is less than the loss threshold to obtain the trained lower channel network.

10. The method according to claim 9, characterized in that The loss function includes: in, is the mean square error loss of random noise and the noise predicted by the lower channel network, ∈ is random noise, is a hyperparameter, Y o is the sample data, The noise predicted for the lower channel network, is the mean squared error between random noise and the noise predicted by the lower channel network.

11. The method according to claim 1, characterized in that: The step of fusing the spatial attention map and the text feature map through the cross attention mechanism layer to obtain a fused feature map includes: The feature fusion of the spatial attention map and the text feature map is performed through a cross attention mechanism, specifically including: Among them, x1 is the spatial attention map, x2 is the text feature map, Q is the query vector corresponding to x1, K is the key vector corresponding to x2, V is the value vector corresponding to x2, and T is the time step.

12. The method according to claim 1, characterized in that The virtual fitting model also includes: An encoder, used for encoding the clothing image to obtain a feature map corresponding to the clothing image; and, used for encoding the model reserved area map to obtain a feature map corresponding to the model reserved area map; A decoder, used to decode the denoised model image to obtain a fitting effect image; The text encoder is used to perform feature encoding on the description text to generate a text feature map corresponding to the description text.

13. A virtual fitting method, characterized in that: The method comprises: Obtaining an image of clothes to be tried on, an image of a user, and a description text of the image of clothes to be tried on; The image of the clothes to be tried on, the user image, and the description text of the image of the clothes to be tried on are input into a virtual fitting model to obtain a fitting effect diagram corresponding to the image of the clothes to be tried on and the model image, wherein the virtual fitting model is trained based on the method described in any one of claims 1 to 12.

14. An electronic device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor, wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 13.

Images