Training method for virtual fitting model, virtual fitting method and electronic equipment

Abstract

The embodiments of the present application relate to the technical field of face image recognition, and disclose a training method for a virtual fitting model, a virtual fitting method, and an electronic device. The training method for the virtual fitting model generates text description content corresponding to a model image, and extracts features of portrait key points, human body analysis diagrams, and clothing images through an image and text dual attention module to obtain guiding features. In addition, a data local noisy module is used to perform local noise on a human body reserved area map, and the human body reserved area map is further encoded to obtain encoding features. The guiding features and encoding features are fused using a style feature fusion module, and finally a model loss function is constructed to perform model training to obtain a trained virtual fitting model. The present application can enhance the fitting effect.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a training method for a virtual fitting model, a virtual fitting method, and an electronic device. Background Art

[0002] Virtual fitting refers to the use of virtual technology to automatically display a three-dimensional image of what a new garment will look like after trying it on. This technology allows customers to put on new clothes and see how they will look without having to remove their clothes.

[0003] Currently, offline smart fitting mostly uses interactive fitting mirrors. Shoppers stand in front of the fitting mirror and choose to try on clothes. The fitting mirror will display the overall outfit based on the user's basic characteristics and clothing style, and the user can freely switch clothes. Online fitting generally involves taking a user's image, selecting the target clothing provided by the system, and automatically replacing it. However, whether it is a smart fitting mirror or online virtual fitting, most existing technologies use the method of collecting human body data and reshaping the user's image through 3D modeling. The combination of clothing and user image is often not appropriate and lacks a sense of naturalness, resulting in a poor user experience. At the same time, collecting 3D information of the body and clothing is usually costly and cumbersome. Therefore, virtual fitting based on 2D images has become a major research direction in recent years.

[0004] Existing virtual fitting algorithms require the clothes to be deformed first and then image generated through a generative network. However, this method relies heavily on the effect of clothing deformation. Poorly deformed clothes will directly affect the generation effect, resulting in poor fitting results. Summary of the Invention

[0005] The embodiments of the present application provide a training method for a virtual fitting model, a virtual fitting method, and an electronic device to solve the problem of poor fitting effect caused by first deforming the clothes and then generating images through a generation network, thereby enhancing the virtual fitting effect.

[0006] In a first aspect, an embodiment of the present application provides a training method for a virtual fitting model, wherein the virtual fitting model includes a data local noise module, an image and text dual attention module, and a style feature fusion module. The method includes:

[0007] Acquire an image dataset, wherein the image dataset includes clothing images and model images;

[0008] The model image is processed by an image-to-text algorithm to generate text description content corresponding to the model image;

[0009] Extract features from the model image to obtain key points of the portrait;

[0010] Extract features from the model image to obtain a human body analysis map, and obtain a human body retention area map based on the human body analysis map;

[0011] The text description content, portrait key points, and clothing images are input into the image and text dual attention module, and feature extraction is performed through the image and text dual attention module to obtain guiding features;

[0012] The human body retention area map is input into the data local noise addition module, and local noise is added by the data local noise addition module to obtain the human body retention area map after local noise addition;

[0013] Encode the human body retention area map after local noise addition to obtain the encoding feature;

[0014] The guidance feature and the coding feature are input into the style feature fusion module, and the guidance feature and the coding feature are fused by the style feature fusion module to obtain the fused feature;

[0015] Decode the fused features to obtain the fitting effect image;

[0016] A model loss function is constructed, and the virtual fitting model is trained based on the image dataset until the model loss function converges to generate a trained virtual fitting model, wherein the model loss function includes text bias loss and image bias loss.

[0017] In some embodiments, the clothing images and the model images correspond one-to-one, and the method further includes:

[0018] Preprocess clothing images and model images, including:

[0019] Through U 2 -Net network extracts the outer contours of the clothing image and the model image, and removes the background information of the clothing image and the model image respectively, to obtain the processed clothing image and the processed model image respectively.

[0020] In some embodiments, feature extraction is performed on the model image to obtain key points of the portrait, including:

[0021] The key points of the model image are detected using the human posture evaluation algorithm to obtain the portrait key points corresponding to the model image.

[0022] In some embodiments, feature extraction is performed on the model image to obtain a human body analysis map, and a human body reserved area map is obtained based on the human body analysis map, including:

[0023] The features of the model image are extracted by the analytical graph algorithm to obtain the human body analytical graph;

[0024] Remove the clothing area in the human body analysis image to obtain the human body retention area map.

[0025] In some embodiments, the human body reserved area map is input into a data local noise adding module, and the data local noise adding module performs local noise adding to obtain a human body reserved area map after local noise adding, including:

[0026]

[0027] Among them, q(x t |P parse-cut ) is the unit time t, given the noise-free P parse-cut , which obeys the mean The variance is Normal distribution, x t is the human body retained area map after local noise addition, P parse-cut Keeping area maps for the human body, a i is the coefficient corresponding to the i-th step, I is the unit matrix, M is P parse-cut Binarized edge map of .

[0028] In some embodiments, the image and text dual attention module includes:

[0029] The encoding layer is used to extract features from the text description content to obtain text information;

[0030] The first fully connected layer is connected to the encoding layer and is used to process the text information to obtain the text feature vector;

[0031] The downsampling layer is used to extract features of portrait key points and clothing images to obtain image information;

[0032] The second fully connected layer is connected to the downsampling layer and is used to process the image information to obtain the image feature vector.

[0033] In some embodiments, feature extraction is performed through the image and text dual attention module to obtain guidance features, including:

[0034] Extract features from the text description content to obtain text information;

[0035] Process the text information to obtain the text feature vector;

[0036] Extract features of portrait key points and clothing images to obtain image information;

[0037] Process the image information to obtain the image feature vector;

[0038] The text feature vector and the image feature vector are fused to obtain the guiding features.

[0039] In some embodiments, the style feature fusion module includes several residual connection modules, wherein each residual connection module includes a convolution layer, an activation function layer, and a batch normalization layer.

[0040] In some embodiments, the model loss function includes:

[0041] Loss = L txt +λ1L img

[0042] Among them, Loss is the model loss, L txt is the text bias loss, λ1 is a hyperparameter, L img is the image bias loss.

[0043] In some embodiments, text bias loss includes:

[0044]

[0045] Among them, L txt is the text bias loss, P text For text, Clip text For text encoding network, Clip text (P text ) is the text P text Encoding Web Clip via Text text The output feature vector, x gt For real samples, To predict the sample, Clip img is the image encoding network, For the prediction sample Clip through image coding network img Output feature vector, Clip img (x gt ) is the real sample x gt Clip through image coding network img The output feature vector, L1 is the calculated absolute difference.

[0046] In some embodiments, image bias loss includes:

[0047]

[0048] Among them, L img is the image bias loss, M is the number of layers of the VGG network, N i is the number of pixels in the i-th layer of the VGG network, x gt For real samples, VGG i (x gt) is the real sample input into the VGG network to obtain the eigenvalue of the i-th feature scale, For the prediction sample, To predict the sample input into the VGG network to obtain the eigenvalue of the i-th feature scale, L2 is used to calculate the average standard deviation.

[0049] In a second aspect, an embodiment of the present application provides a virtual fitting method, comprising:

[0050] Obtain images of clothes to be tried on and images of models;

[0051] The clothes image to be tried on and the model image are input into a virtual fitting model to obtain a fitting image corresponding to the clothes image to be tried on, wherein the virtual fitting model is trained based on the above method.

[0052] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0053] at least one processor, and

[0054] a memory communicatively coupled to at least one processor, wherein:

[0055] The memory stores instructions that can be executed by at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method of the first aspect or the second aspect.

[0056] In a fourth aspect, an embodiment of the present application provides an electronic device, including:

[0057] at least one processor, and

[0058] a memory communicatively coupled to at least one processor, wherein:

[0059] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method provided in the first aspect or the second aspect as described above.

[0060] In a fifth aspect, an embodiment of the present application provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are used to enable an electronic device to execute the virtual fitting model training method provided in the first aspect or the virtual fitting method provided in the second aspect.

[0061] The beneficial effects of the embodiments of the present application are as follows: Different from the prior art, the embodiments of the present application provide a training method for a virtual fitting model, the virtual fitting model includes a data local noise module, an image and text dual attention module, and a style feature fusion module, and the method includes: obtaining an image data set, wherein the image data set includes clothing images and model images; processing the model images through an image-to-text algorithm to generate text description content corresponding to the model images; extracting features from the model images to obtain portrait key points; extracting features from the model images to obtain a human body analysis map, and obtaining a human body retention area map based on the human body analysis map; inputting the text description content, portrait key points, and clothing images into the image and text dual attention module, and This dual attention module performs feature extraction to obtain guiding features; the human body retention area map is input into the data local denoising module, and local denoising is performed through the data local denoising module to obtain the human body retention area map after local denoising; the human body retention area map after local denoising is encoded to obtain encoding features; the guiding features and the encoding features are input into the style feature fusion module, and the guiding features and the encoding features are fused through the style feature fusion module to obtain fusion features; the fusion features are decoded to obtain a fitting effect map; a model loss function is constructed, and the virtual fitting model is trained based on the image dataset until the model loss function converges to generate the trained virtual fitting model, wherein the model loss function includes text bias loss and image bias loss.

[0062] On the one hand, by generating text description content corresponding to the model image, and extracting features of the portrait key points, human body analysis map, and clothing image through the image and text dual attention module, we obtain the guiding features. The data local denoising module performs local denoising on the human body reserved area map, and further encodes the human body reserved area map to obtain the encoding features. The style feature fusion module is used to fuse the guiding features and encoding features to obtain the fitting effect image, which can improve the generation quality of the fitting effect image.

[0063] On the other hand, model training is performed by constructing a model loss function, which includes text bias loss and image bias loss, to obtain a virtual fitting model after training. By training with text bias loss and image bias loss, this application can enhance the fitting effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0065] Figure 1Schematic diagram of an application environment of a training method for a virtual fitting model provided in an embodiment of the present application;

[0066] Figure 2 1 is a flow chart of a training method for a virtual fitting model provided in an embodiment of the present application;

[0067] Figure 3 is a schematic diagram of an image and text dual attention module provided in an embodiment of the present application;

[0068] Figure 4 yes Figure 2 A detailed flowchart of step S205 in FIG.

[0069] Figure 5 This is a schematic diagram of the overall structure of a virtual fitting model provided in an embodiment of the present application;

[0070] Figure 6 This is a flow chart of a virtual fitting method provided in an embodiment of the present application;

[0071] Figure 7 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] The present application is described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but are not intended to limit the present application in any form. It should be noted that those skilled in the art may make several variations and improvements without departing from the scope of the present application. These all fall within the scope of protection of the present application.

[0073] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0074] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other and are all within the scope of protection of the present application. In addition, although the functional modules are divided in the device schematic and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order than the module division in the device or the order in the flow chart. In addition, the words "first", "second", "third", etc. used herein do not limit the data and execution order, but only distinguish between the same items or similar items with basically the same functions and effects.

[0075] Unless otherwise defined, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which this application belongs. The terms used in this specification and in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the relevant listed items.

[0076] In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0077] Before explaining the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0078] (1) Neural network, also referred to as neural network (NNs) or connection model, is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. Neural networks rely on the complexity of the system to adjust the interconnected relationships between a large number of internal nodes to achieve the purpose of processing information. Specifically, a neural network can be composed of neural units, which can be understood as a neural network with an input layer, a hidden layer, and an output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. Among them, a neural network with many hidden layers is called a deep neural network (DNN). The work of each layer in the neural network can be described by the mathematical expression y=a(W·x+b). From a physical perspective, the work of each layer in the neural network can be understood as completing the transformation from input space to output space (i.e., the row space to the column space of the matrix) through five operations on the input space (the set of input vectors). These five operations include: 1. Dimension increase / reduction; 2. Zoom in / out; 3. Rotation; 4. Translation; 5. "Bending". Among them, operations 1, 2, and 3 are completed by "W·x", operation 4 is completed by "+b", and operation 5 is implemented by "a()". The word "space" is used here because the object being classified is not a single thing, but a class of things. Space refers to the collection of all individuals of this type of thing. Among them, W is the weight matrix of each layer of the neural network. Each value in the matrix represents the weight value of a neuron in the layer. The matrix W determines the spatial transformation from the input space to the output space mentioned above, that is, the W of each layer of the neural network controls how to transform the space. The purpose of training a neural network is to eventually obtain the weight matrices of all layers of the trained neural network. Therefore, the training process of a neural network is essentially learning how to control spatial transformation, and more specifically, learning the weight matrix.

[0079] It should be noted that in the embodiments of the present application, the models used for machine learning tasks are essentially neural networks. Common components in neural networks include convolutional layers, activation function layers, and batch normalization layers. By assembling these common components in neural networks, a model is designed. When the model parameters (weight matrices of each layer) are determined so that the model error meets the preset conditions or the number of model parameters is adjusted to reach a preset threshold, the model converges.

[0080] The convolutional layer is configured with multiple convolution kernels, each with a corresponding step size, to perform convolution operations on the image. The purpose of the convolution operation is to extract different features of the input image. The first convolution layer may only extract low-level features such as edges, lines, and corners. Deeper convolution layers can iteratively extract more complex features from these low-level features.

[0081] The activation function layer is used to enable each neuron in a neural network to accept the output value of the previous layer as its input value and pass the processed result to the next layer. Common activation functions used in the activation function layer include but are not limited to the Rectified Linear Unit (ReLU) function, the Swish function, and the Parametric Rectified Linear Unit (PReLU) function.

[0082] Batch Normalization (BN) layer is used to standardize the features of a certain layer in the network. Its purpose is to solve the problem of numerical instability in deep neural networks. That is, as the number of network layers deepens, the update of parameters during training easily causes drastic changes in the feature output near the output layer, which is not conducive to training an effective neural network.

[0083] (2) Loss function refers to a function that maps the value of a random event or its related random variables to a non-negative real number to represent the "risk" or "loss" of the random event. The loss function is a non-negative real function that is used to quantify the difference between the predicted label and the true label predicted by the model. In applications, the loss function is usually associated with the optimization problem as a learning criterion, that is, the model is solved and evaluated by minimizing the loss function. For example, it is used for model parameter estimation (parametric estimation) in statistics and machine learning. In the process of training a neural network, because we hope that the output of the neural network is as close as possible to the value we really want to predict, we can compare the current network's predicted value with the target value we really want, and then update the weight matrix of each layer of the neural network based on the difference between the two (however, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the neural network). For example, if the network's predicted value is too high, adjust the weight matrix to make it predict lower, and continue to adjust until the neural network can predict the target value we really want. Therefore, it's necessary to predefine how to compare the predicted value to the target value. This is known as the loss function or objective function, a crucial equation used to measure the difference between the predicted and target values. For example, a higher loss function output (loss) indicates a greater difference, and neural network training becomes a process of minimizing this loss.

[0084] The technical solution of this application is described in detail below with reference to the accompanying drawings.

[0085] See also Figure 1 , Figure 1 Schematic diagram of an application environment of a training method for a virtual fitting model provided in an embodiment of the present application;

[0086] like Figure 1 As shown, the application environment 100 includes: a terminal 101 and a server 102, and the terminal 101 and the server 102 communicate through wired or wireless communication.

[0087] The terminal 101 may be, but is not limited to, a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 101 may be provided with a client, which may be a video client, a browser client, an online shopping client, an instant messaging client, etc. This application does not limit the type of the client.

[0088] The terminal 101 and the server 102 can be connected directly or indirectly via wired or wireless communication, which is not limited in this application. The terminal 101 can receive the image of the clothes to be tried on and the image of the model sent by the server 102, and display the image of the clothes to be tried on, the image of the model, and the fitting image corresponding to the image of the clothes to be tried on obtained by fitting the two on a visual interface. The terminal can respond to a virtual fitting instruction triggered by the user and obtain the image of the clothes to be tried on and the image of the model through an image acquisition device. The image acquisition device can be built into the terminal 101 or externally connected to the terminal 101, which is not limited in this application.

[0089] Terminal 101 can send the virtual fitting instruction and the collected images of clothes to be tried on and model images to server 102, and receive the fitting image corresponding to the image of clothes to be tried on returned by server 102, and then display the fitting image corresponding to the image of clothes to be tried on on a visual interface, so that the user can view the fitting image corresponding to the image of clothes to be tried on.

[0090] It is understood that terminal 101 may generally refer to one of multiple terminals. The embodiments of this application only use terminal 101 as an example. Those skilled in the art will appreciate that the number of terminals may be greater or lesser. For example, there may be only one terminal, or there may be dozens, hundreds, or even more terminals. The embodiments of this application do not limit the number of terminals or device types.

[0091] Among them, server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0092] The server 102 and the terminal 101 may be connected directly or indirectly via wired or wireless communication, which is not limited in this application. The server 102 may maintain a clothing image database for storing multiple clothing images. The server 102 may receive a virtual fitting instruction and an image of clothing to be fitted and an image of a model sent by the terminal 101, and perform a virtual fitting on the image of clothing to be fitted and the image of a model according to the virtual fitting instruction, obtain a fitting image corresponding to the image of clothing to be fitted, and then send the fitting image corresponding to the image of clothing to be fitted to the terminal 101.

[0093] It is understandable that the number of the above servers 102 can be more or less, and the embodiment of the present application does not limit this. Of course, the server 102 can also include other functional servers to provide more comprehensive and diversified services.

[0094] The technical solution of this application is applicable to various virtual fitting scenarios. Specifically, the technical solution of this application can be used to fuse an image of a garment to be fitted and an image of a model in a virtual fitting scenario. In some implementations of the virtual fitting scenario, the image of the garment to be fitted and the image of the model in the scene are fused using a virtual fitting model to determine the fitting image corresponding to the garment image to be fitted, wherein the virtual fitting model is pre-trained.

[0095] Currently, virtual fitting algorithms need to deform the clothes first and then generate images through a generative network. However, this method relies heavily on the effect of clothing deformation. Poorly deformed clothes will directly affect the generation effect, resulting in poor fitting results.

[0096] In view of this, embodiments of the present application provide a virtual fitting model training method, a virtual fitting method, and an electronic device to enhance the virtual fitting effect.

[0097] Example 1

[0098] See also Figure 2 , Figure 2 1 is a flow chart of a training method for a virtual fitting model provided in an embodiment of the present application;

[0099] The method can be applied to various electronic devices, such as terminals, servers and other electronic devices. Specifically, the execution subject of the method is one or at least two processors of the electronic device.

[0100] Among them, the virtual fitting model includes a data local noise module, an image and text dual attention module, and a style feature fusion module.

[0101] like Figure 2 As shown, the training method of the virtual fitting model includes:

[0102] Step S201: Acquire an image dataset, wherein the image dataset includes clothing images and model images;

[0103] It is understandable that before training the virtual fitting model, it is necessary to collect an image dataset and perform model training based on the image dataset.

[0104] Among them, each clothing image corresponds to a model image, that is, the clothing image and the model image are paired one-to-one, for example, the model in the model image wears the clothing corresponding to the clothing image, thereby collecting a paired data set of clothing images and model images.

[0105] Specifically, the image dataset includes a large number of model images, each wearing the same or different clothing. The clothing images are tiled images facing up, and the model images are taken from the front. The minimum image size must be no less than 512 pixels.

[0106] It is understood that, in the actual process of collecting model images, there may be images of varying sizes. Therefore, in this embodiment of the application, each model image is normalized to a uniform size, for example, to 1024*768. Alternatively, each model image may be subjected to operations such as illumination transformation and rotation for data augmentation, which helps improve the robustness of the virtual fitting model.

[0107] In the embodiment of the present application, the clothing image and the model image correspond one to one, and the method further includes:

[0108] Preprocess clothing images and model images, including:

[0109] Through U 2 -NET network extracts the outer contours of the clothing image and the model image, and removes the background information of the clothing image and the model image respectively, to obtain the processed clothing image and the processed model image respectively.

[0110] Specifically, U 2 The -NET network is a two-layer nested U-shaped structure, which designs a new U-shaped residual block (ReSidual U-blocks, RSU) structure at the bottom layer and a U-Net-like structure at the top layer, where each level is filled with RSU. RSU replaces ordinary single-stream convolution with U-Net and replaces the original features with local features composed of a weight layer. This design change enables the network to directly extract features from multiple scales of each residual block.

[0111] Among them, U 2 The -Net network consists of three parts: a six-level encoder, a five-level decoder, and a saliency map fusion module. The saliency map fusion module is connected to the decoder and the last encoder level. The six-level encoder includes: En_1, En_2, En_3, En_4, En_5, and En_6, and the five-level decoder includes: De_1, De_2, De_3, De_4, and De_5.

[0112] Specifically, in encoders En_1, En_2, En_3, and En_4, residual U blocks RSU-7, RSU-6, RSU-5, and RSU-4 are used, respectively. "7," "6," "5," and "4" represent the height (L) of the RSU block. L is typically configured based on the spatial resolution of the input feature map. In En_5 and En_6, RSU-4F is used, where "F" indicates that RSU is an expanded version and the pooling and upsampling operations are replaced by expanded convolutions. All intermediate feature maps of RSU-4F have the same resolution as their input feature maps.

[0113] The decoder has a similar architecture to its symmetric encoder, En_6. De_5 also uses a dilated version of the residual U-block RSU-4F, similar to that used in the encoder stages En_5 and En_6. Each decoder stage takes as input the concatenation of the upsampled feature maps from its previous stage and the upsampled feature maps from its symmetric encoder stage.

[0114] The saliency map fusion module is used to generate saliency probability maps. 2 -Net first outputs saliency probability maps S(6), S(5), S(4), S(3), S(2), S(1) generated by stages En_6, De_5, De_4, De_3, De_2 and De_1, which are composed of a 3×3 convolution layer and a sigmoid function. Then, it entropy samples these output saliency logits (the convolution output before the sigmoid function) and maps them to the input image size, fuses them through a cascade operation (concat operation), and finally a 1×1 convolution layer and a sigmoid function to generate the final saliency probability map S_fuse, which is the processed clothing image or the processed model image.

[0115] In the embodiment of the present application, U 2 -NET network is used as an image segmentation network to perform high-precision image clipping processing, which can better remove background information and facilitate better model training.

[0116] Step S202: Processing the model image using an image-to-text algorithm to generate text description content corresponding to the model image;

[0117] Specifically, the image-to-text algorithm includes a clip-interrogator algorithm, which processes the model image through the clip-interrogator algorithm to generate text description content corresponding to the model image.

[0118] It is understandable that the clip-interrogator algorithm is based on a large number of image and text datasets, uses text as image labels for training, and by converting the image and text classification task into an image and text matching task, it can better generate text description content corresponding to the model image.

[0119] Step S203: extracting features from the model image to obtain key points of the portrait;

[0120] Specifically, feature extraction is performed on the model image to obtain key points of the portrait, including:

[0121] The key points of the model image are detected using the human posture evaluation algorithm to obtain the portrait key points corresponding to the model image.

[0122] The human body posture assessment algorithm includes but is not limited to the human skeleton key point detection (OpenPose) algorithm. Preferably, the OpenPose algorithm is used in the embodiment of the present application.

[0123] The OpenPose algorithm is based on a real-time multi-person key point detection library for the body, face, hands, and feet. By inputting a model image, it detects the key points of the model image's body, face, hands, and feet, thereby obtaining the portrait key points corresponding to the model image, namely the portrait key point map.

[0124] Specifically, the OpenPose algorithm relies on convolutional neural networks and supervised learning to achieve human posture assessment. Its main advantage is that it is suitable for open source real-time systems for multi-person two-dimensional posture detection and can accurately and quickly identify key points of the human body.

[0125] Specifically, the OpenPose algorithm takes the entire person image as input to the network, then predicts confidence maps for body part detection and part affinity fields (PAF) for part association. It then performs a set of two-part matching on the associated body part candidates through a parsing step, and finally performs a human skeleton construction to connect the human body key points and assemble them into a complete pose of all people in the image, obtaining a human body key point area map corresponding to each model image.

[0126] Among them, the human body key points are divided into 25 categories in total, and each category is marked with 0-24. The categories of human body key points are: {0, "nose"}, {1, "neck"}, {2, "right upper arm (RShoulder)"}, {3, "right elbow (RElbow)"}, {4, "right wrist (RWrist)"}, {5, "left upper arm (LShoulder)"}, {6, "left elbow (LElbow)"}, {7,

[0127] "Left wrist (LWrist)"}, {8, "Mid hip (MidHip)"}, {9, "Right hip (RHip)"}, {10, "Right knee (RKnee)"}, {11, "Right ankle (Rankle)"}, {12, "Left hip (Lhip)"}, {13, "Left knee (LKnee)"}, {14, "Left ankle (LAnkle)"}, {15, "Right eye (Reye)"}, {16, "Left eye (Le ye)"}, {17,"Right ear(Rear)"}, {18,"Left ear(Lear)"}, {19,"Left big toe(LBigToe)"}, {20,"Left little toe(LSmallToe)"}, {21,"Left heel(LHeel)"}, {22,"Right big toe(RBigToe)"}, {23,"Right little toe(RSmallToe)"} {24,"Right heel(Rheel)"}.

[0128] Step S204: extracting features from the model image to obtain a human body analysis map, and obtaining a human body reserved area map based on the human body analysis map;

[0129] Specifically, feature extraction is performed on the model image to obtain a human body analysis map, and based on the human body analysis map, a human body retention area map is obtained, including:

[0130] The features of the model image are extracted by the analytical graph algorithm to obtain the human body analytical graph;

[0131] Remove the clothing area in the human body analysis image to obtain the human body retention area map.

[0132] Specifically, the human body parsing map includes multiple area categories, and the parsing map algorithm is used to extract the corresponding area according to the detected category. The parsing map algorithm includes a general human body parsing algorithm, for example: Graphonomy algorithm, which uses the Graphonomy algorithm to extract features from the model image to obtain a human body parsing map, and obtain the area in the human body parsing map where clothes need to be changed, deduct it, and obtain a human body retention area map.

[0133] The Graphonomy algorithm consists of two parts: intra-graph reasoning and inter-graph transfer. Specifically, the Graphonomy algorithm first learns and propagates a compact, high-level semantic graph representation within a dataset through intra-graph reasoning. Then, through inter-graph transfer driven by an explicit hierarchical semantic label structure, it transfers and fuses semantic information across multiple datasets to obtain a human body parsing graph corresponding to each model image. By extracting a universal semantic graph representation for each specific task, the Graphonomy algorithm is able to predict labels at all levels of a system without increasing complexity.

[0134] In the embodiment of the present application, each human body analysis diagram includes multiple area categories. For example, each human body analysis diagram includes 20 area categories, represented by serial numbers 0-19. The 20 area categories are:

[0135] 0: Background; 1: Hat; 2: Hair; 3: Gloves; 4: Sunglasses; 5: Upper--clothes; 6: Dress; 7: Coat; 8: Socks; 9: Pants; 10: Torso-skin; 11: Scarf; 12: Skirt; 13: Face; 14: Left-arm; 15: Right-arm; 16: Left-leg; 17: Right-leg; 18: Left-shoe; 19: Right-shoe.

[0136] Step S205: Input the text description content, portrait key points, and clothing image into the image and text dual attention module, perform feature extraction through the image and text dual attention module, and obtain guiding features;

[0137] See also Figure 3 , Figure 3 is a schematic diagram of an image and text dual attention module provided in an embodiment of the present application;

[0138] like Figure 3 As shown, the image and text dual attention module includes an encoding layer, a first fully connected layer, a downsampling layer, and a second fully connected layer.

[0139] Specifically, the encoding layer is used to extract features from the text description content to obtain text information, for example, extract features from the text description content based on the Word2Vec algorithm to obtain text information (Embedding information).

[0140] Specifically, the first fully connected layer is connected to the encoding layer, and is used to obtain text information output by the encoding layer, process the text information, and obtain a text feature vector.

[0141] Among them, the first fully connected layer includes a Transformer network, which includes an encoder and a decoder. Both the encoder and the decoder contain 6 blocks. It should be noted that the softmax layer in the Transformer network in the first fully connected layer is replaced by a fully connected layer to output a 1024-channel text feature vector.

[0142] Specifically, the downsampling layer is used to extract features of portrait key points and clothing images to obtain image information.

[0143] Among them, the portrait key points, that is, the portrait key point map and the clothing image are connected and fused based on the channel dimension and input into the downsampling layer. In this embodiment of the present application, the downsampling layer includes 6 downsampling modules, wherein each downsampling module includes a convolution layer, an activation function layer and a batch normalization layer, and the convolution kernel of the convolution layer of each downsampling module is set to 32, 64, 128, 256, 512, and 1024. The size of the convolution kernel is 3*3, the step size is set to 2, and the activation function used is the PRelu activation function.

[0144] Specifically, the second fully connected layer is connected to the downsampling layer to process the image information and obtain an image feature vector.

[0145] Among them, the second fully connected layer is used to output the image feature vector of 1024 channels. The network structure of the second fully connected layer is the same as that of the first fully connected layer, which will not be repeated here.

[0146] The text feature vector is obtained through the first fully connected layer, and the image feature vector is obtained through the second fully connected layer. The text feature vector and the image feature vector are further weightedly fused to obtain the guidance feature, i.e., the image-text guidance feature map. The weight ratio of the text feature vector to the image feature vector is set according to specific needs. For example, the ratio of the text feature vector to the image feature vector is 2:8.

[0147] Please refer to Figure 4 , Figure 4 yes Figure 2 A detailed flowchart of step S205 in FIG.

[0148] like Figure 4 As shown, step S205, feature extraction is performed through the image and text dual attention module to obtain the guiding features, including:

[0149] Step S2051: extracting features from the text description content to obtain text information;

[0150] Specifically, the text description content is subjected to feature extraction through the encoding layer to obtain text information.

[0151] Step S2052: Process the text information to obtain a text feature vector;

[0152] Specifically, the text information is processed through the first fully connected layer to obtain a text feature vector.

[0153] Step S2053: extracting features from the portrait key points and clothing image to obtain image information;

[0154] Specifically, the features of the portrait key points and clothing images are extracted through the downsampling layer to obtain image information.

[0155] Step S2054: Process the image information to obtain an image feature vector;

[0156] Specifically, the image information is processed through the second fully connected layer to obtain an image feature vector.

[0157] Step S2055: Fuse the text feature vector and the image feature vector to obtain the guidance feature.

[0158] Specifically, the text feature vector and the image feature vector are weightedly fused to obtain the image-text guidance feature map, and the fusion method can be completed by the fusion module.

[0159] Step S206: inputting the human body reserved area map into the data local noise adding module, performing local noise adding through the data local noise adding module, and obtaining the human body reserved area map after local noise adding;

[0160] Specifically, the human body retention area map is input into the data local noise adding module, and the local noise adding module is used to perform local noise adding to obtain the human body retention area map after local noise adding, including:

[0161]

[0162] Among them, q(x t |P parse-cut ) is the unit time t, given the noise-free P parse-cut , which obeys the mean The variance is Normal distribution, x t is the human body retained area map after local noise addition, P parse-cut Keeping area maps for the human body, a i is the coefficient corresponding to the i-th step, I is the unit matrix, M is P parse-cut Binarized edge map of .

[0163] Among them, P parse-cut is the human body retention area map without noise, t is the time, a t =1-β t , β t is a preset constant. In the binary edge map, the value of the human body retention area is 1, and the value of other areas is 0. Local noise addition includes adding local Gaussian noise. By performing local noise addition through the data local noise addition module, the human body retention area map x after local noise addition at any time t can be obtained. t .

[0164] Step S207: Encode the human body retained area map after local noise addition to obtain encoding features;

[0165] Specifically, the human body retention area map after local noise addition is encoded by the encoding module to obtain encoding features.

[0166] In an embodiment of the present application, the encoding module includes a plurality of sequentially connected feature extraction convolution layers, fully connected layers and softmax layers. Each feature extraction convolution layer is configured with an activation function layer and a batch normalization layer. For example, the convolution kernel of the feature extraction convolution layer is 3*3 in size, the step size is set to 2, the activation function is set to the Relu activation function, and the convolution kernel of each layer is set to 32, 64, 128, 256, 512, 1024. After multiple layers of convolution operations, a feature map of 1024 channels is finally obtained, that is, the encoding feature.

[0167] It is understandable that the network structure of the feature extraction convolution layer in the embodiment of the present application can adjust the convolution kernel and / or step size according to actual needs, which is not limited here.

[0168] Step S208: inputting the guidance feature and the coding feature into the style feature fusion module, and fusing the guidance feature and the coding feature through the style feature fusion module to obtain a fusion feature;

[0169] Specifically, a style feature fusion module is constructed, and the guiding feature and the coding feature are input into the style feature fusion module, and the style feature fusion module fuses the guiding feature and the coding feature to obtain a fusion feature.

[0170] In an embodiment of the present application, the style feature fusion module includes several residual connection modules, wherein each residual connection module includes a convolution layer, an activation function layer and a batch normalization layer. For example, the convolution layer of the residual connection module is set to 2048, 2048, 2048, the size of the convolution kernel is 3*3, and the step size is set to 1, so that the guiding feature can better and effectively guide the encoding feature to be decoded in a specific direction to obtain a fusion feature.

[0171] Step S209: Decode the fused features to obtain a fitting effect image;

[0172] Specifically, the fusion features are decoded by the decoding module to obtain a fitting effect image.

[0173] In an embodiment of the present application, the decoding module includes a Skip connection layer and a transposed convolution layer, wherein the Skip connection layer is connected and fused with each downsampling output in the encoding module to obtain the same channel feature map. Among them, the convolution kernel size of the transposed convolution layer is 3*3, and the step size is 2. The activation function used is PRelu and batch normalization layer. After the same number of decoding modules, the convolution kernel of each layer is set to 512, 256, 128, 64, 32, 3 to output a 3-channel fitting effect map, which includes a fitting effect map in RGB format. The low-resolution feature layer is upsampled by the transposed convolution layer. Compared with the simple upsampling layer, the transposed convolution can better restore image details, which is conducive to improving the decoding effect.

[0174] In an embodiment of the present application, the U-Net algorithm uses concat (splicing layer) to fuse feature maps at corresponding positions in the encoding and decoding processes, so that the decoder can obtain more high-resolution information when upsampling, thereby more perfectly restoring the detail information in the original image and improving the segmentation accuracy.

[0175] Furthermore, the addition of a skip connection layer to the U-Net network allows the network to fuse the feature maps corresponding to the encoder position across channels during each level of upsampling. By fusing low-level features with high-level features, the network can retain more high-resolution details contained in high-level feature maps, thereby improving image segmentation accuracy.

[0176] Step S210: Construct a model loss function and train the virtual fitting model based on the image dataset until the model loss function converges to generate a trained virtual fitting model, wherein the model loss function includes text bias loss and image bias loss.

[0177] Specifically, the model loss function includes:

[0178] Loss = Ltxt +λ1L im

[0179] Among them, Loss is the model loss, L txt is the text bias loss, λ1 is a hyperparameter, L img is the image bias loss.

[0180] In the embodiment of the present application, the hyperparameter λ1 can be set according to specific needs, for example, set to 10.

[0181] In the embodiment of the present application, text bias loss includes:

[0182]

[0183] Among them, L txt is the text bias loss, P text For text, Clip text For text encoding network, Clip text (P text ) is the text P text Encoding Web Clip via Text text The output feature vector, x gt For real samples, To predict the sample, Clip img is the image encoding network, For the prediction sample Clip through image coding network img Output feature vector, Clip img (x gt ) is the real sample x gt Clip through image coding network img The output feature vector, L1 is the calculated absolute difference.

[0184] In the embodiment of the present application, the text bias loss is used to measure the similarity between the fitting effect picture and the text description content. The measurement is performed through the CLIP model, which includes a text encoding network Clip text and image coding network Clip img , the text encoding network and image encoding network can respectively encode text and pictures into vectors of size 1*512, and then measure the similarity between the two by cosine distance.

[0185] In the embodiment of the present application, the image bias loss includes:

[0186]

[0187] Among them, Lim gis the image bias loss, M is the number of layers of the VGG network, N i is the number of pixels in the i-th layer of the VGG network, x gt For real samples, VGG i (x gt ) is the real sample input into the VGG network to obtain the eigenvalue of the i-th feature scale, For the prediction sample, To predict the sample input into the VGG network to obtain the eigenvalue of the i-th feature scale, L2 is the calculated average standard deviation. It can be understood that N i Represents the number of pixels in the feature map in the i-th layer of the VGG network. The number of pixels = the product of the length, width, and number of channels of the feature map.

[0188] It is understandable that the VGG network is used to calculate the style loss. The style loss is the difference between the parameters of the convolution layer in the VGG network between the generated image and the style image. The number of feature scales can be set to 5 or other values.

[0189] Among them, the VGG network includes the VGG19 network. The VGG19 network is a convolutional neural network with 19 hidden layers (16 convolutional layers and 3 fully connected layers). The VGG19 network is used to extract image features, which can reflect the difference between predicted samples and real samples under the same VGG19 feature extraction, that is, the perceptual loss. It can also improve the depth of the network while ensuring the same perceptual field, thereby improving the effect of the neural network to a certain extent.

[0190] The virtual fitting model is trained through text bias loss and image bias loss, so that more editable virtual fitting functions can be achieved by subsequently changing the text description content and clothing images. In addition, through text bias loss and image bias loss, the diffusion model's ability to refer to text and image guidance information can be improved, and specific fitting effects can be generated in a more controllable manner, thereby reducing the randomness of the diffusion model.

[0191] In an embodiment of the present application, a model loss function is used for back propagation so that each network in the virtual fitting network converges, for example: one or at least two networks in the data local noise module, the image and text dual attention module, the encoding module, the style feature fusion module, and the decoding module converge, wherein the network convergence includes that the loss reaches a minimum and / or the loss fluctuates within a range and / or the number of training times reaches a certain number. For example: the Adam algorithm (Adaptive Moment Estimation Algorithm) is used to optimize the network parameters of each network in the virtual fitting model, the number of iterations is set to 500 times, the initialization learning rate is set to 0.001, and the weight decay is set to 0.0005. Every 50 iterations, the learning rate decays to the original 1 / 10, and the network is trained until convergence to save the corresponding network parameters during convergence. Wherein, model training can be trained using a graphics card, for example: dual-card parallel training, and the number of samples selected for one training (batchsize) can be set according to specific needs, for example: set to 64.

[0192] It can be understood that the Adam algorithm can be seen as a combination of the momentum method and the RMSprop algorithm. It not only uses momentum as the parameter update direction, but also can adaptively adjust the learning rate.

[0193] By constructing a model loss function, which includes text bias loss and image bias loss, and using the model loss function to train the virtual fitting model, the diffusion model's ability to refer to text and image guidance information can be improved, specific fitting effects can be generated more controllably, and the original randomness of the diffusion model can be reduced, so that the virtual fitting model can better generate realistic and natural fitting effect images.

[0194] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the overall structure of a virtual fitting model provided in an embodiment of the present application;

[0195] like Figure 5 As shown in the figure, the virtual fitting model includes a data local noise module, an image and text dual attention module, an encoding module, a style feature fusion module, and a decoding module.

[0196] Among them, the data local noise module is connected to the encoding module, the encoding module is connected to the style feature fusion module, the image and text dual attention module is connected to the style feature fusion module, and the style feature fusion module is connected to the decoding module.

[0197] Specifically, the data local noise module is used to locally add noise to the input human body retention region map to obtain a locally noisy human body retention region map. The human body retention region map is obtained by extracting features from the model image using a parsing graph algorithm, obtaining a human body parsing graph, and removing the clothing area from the human body parsing graph.

[0198] In an embodiment of the present application, by gradually adding local Gaussian noise to the human body retained area map, a local noisy image at any time can be obtained, thereby better preserving the original feature information of the image, making the skin area and clothes generated by the diffusion model more naturally connected with the original retained area.

[0199] Specifically, the image and text dual attention module is used to extract features from the input portrait key points and clothing images to obtain guiding features. The portrait key points are images obtained by performing key point detection on the model image using a human pose estimation algorithm.

[0200] Specifically, the encoding module is used to encode the human body retention area map after local noise addition to obtain encoding features.

[0201] Specifically, the style feature fusion module is used to fuse the input guidance features and encoding features to obtain fused features.

[0202] In the embodiment of the present application, the style feature fusion module performs fusion in a specific manner to obtain diffusion guidance feature information, which is conducive to providing a strong generation basis for the subsequent diffusion network decoding module.

[0203] Specifically, the decoding module is used to decode the input fusion features to obtain a fitting effect image.

[0204] In an embodiment of the present application, the human body retention area map is locally denoised by a data local denoising module, and the human body retention area map is further encoded to obtain encoding features. The guidance features and encoding features are fused using a style feature fusion module to obtain a fitting effect map, which can improve the generation quality of the fitting effect map.

[0205] After the training phase, the present application further includes a testing phase. In the testing phase, clothing images and model images are input into a virtual fitting model to obtain fitting images corresponding to the clothing images to be fitted.

[0206] The model image is a frontal pose image of the model. By selecting different clothes for virtual fitting, and in different posture and lighting environments, the reverse process reasoning of the diffusion network is used to gradually remove noise. Based on the characteristics of the specified clothing and human key points and retained areas, the fitting effect image is generated by directionally diffusing. Compared with the virtual fitting algorithm based on the Generative Adversarial Network (GAN), this application generates fitting effect images through directionally diffusing, which can achieve a more realistic and natural fitting effect.

[0207] In an embodiment of the present application, a training method for a virtual fitting model is provided, wherein the virtual fitting model includes a data local noise adding module, an image and text dual attention module, and a style feature fusion module. The method includes: obtaining an image data set, wherein the image data set includes clothing images and model images; processing the model images through an image-to-text algorithm to generate text description content corresponding to the model images; extracting features from the model images to obtain portrait key points; extracting features from the model images to obtain a human body analysis map, and obtaining a human body reserved area map based on the human body analysis map; inputting the text description content, the portrait key points, and the clothing images into the image and text dual attention module, and performing the image and text dual attention module to perform the text description. Feature extraction to obtain guiding features; input the human body retention area map into the data local denoising module, perform local denoising through the data local denoising module, and obtain the human body retention area map after local denoising; encode the human body retention area map after local denoising to obtain encoding features; input the guiding features and the encoding features into the style feature fusion module, fuse the guiding features and the encoding features through the style feature fusion module to obtain fusion features; decode the fusion features to obtain a fitting effect map; construct a model loss function, and train the virtual fitting model based on the image data set until the model loss function converges to generate the trained virtual fitting model, wherein the model loss function includes text bias loss and image bias loss.

[0208] On the one hand, by generating text description content corresponding to the model image, and extracting features of the portrait key points, human body analysis map, and clothing image through the image and text dual attention module, we obtain the guiding features. The data local denoising module performs local denoising on the human body reserved area map, and further encodes the human body reserved area map to obtain the encoding features. The style feature fusion module is used to fuse the guiding features and encoding features to obtain the fitting effect image, which can improve the generation quality of the fitting effect image.

[0209] On the other hand, model training is performed by constructing a model loss function, which includes text bias loss and image bias loss, to obtain a virtual fitting model after training. By training with text bias loss and image bias loss, this application can enhance the fitting effect.

[0210] Example 2

[0211] After the training method of the virtual fitting model provided in the above embodiment is used, a trained virtual fitting model is obtained, which can be used to perform virtual fitting.

[0212] It can be understood that the above-mentioned embodiment 1 is a training stage of the virtual fitting model, and the embodiment 2 of the present application is a use stage of the virtual fitting model to obtain a virtual fitting image.

[0213] For details, please refer to Figure 6 , Figure 6 This is a flow chart of a virtual fitting method provided in an embodiment of the present application;

[0214] like Figure 6 As shown, the virtual fitting method includes:

[0215] Step S601: Acquire an image of clothing to be tried on and an image of a model;

[0216] Specifically, after obtaining the trained virtual fitting model, images of clothes to be fitted and images of models are acquired.

[0217] The virtual fitting model is trained by the composite neural network in the above embodiment (including a data local noise module, an image and text dual attention module, a style feature fusion module, an encoding network, and a decoding network), and has the same structure and function as the above composite neural network. The specific way in which the virtual fitting model processes the clothing images and model images of the fitting is the same as the way in which the composite neural network in the above embodiment processes the clothing images and model images, and will not be repeated here.

[0218] Step S602: inputting the image of the clothes to be tried on and the image of the model into the virtual fitting model to obtain a fitting image corresponding to the image of the clothes to be tried on.

[0219] It should be noted that the virtual fitting model in the embodiment of the present application is a model obtained by training through the above-mentioned embodiment 1. The relevant content of the virtual fitting model can refer to the content mentioned in the above-mentioned embodiment and will not be repeated here.

[0220] In an embodiment of the present application, by training a virtual fitting model, after obtaining the trained virtual fitting model, an image of clothes to be tried on and an image of a model are obtained, and the image of clothes to be tried on and the image of a model are input into the virtual fitting model to obtain a fitting image corresponding to the image of clothes to be tried on. The present application can improve the virtual fitting effect.

[0221] Example 3

[0222] See also Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0223] like Figure 7 As shown, the electronic device 70 includes one or more processors 701 and a memory 702. Figure 7 A processor 701 is taken as an example.

[0224] The processor 701 and the memory 702 may be connected via a bus or other means. Figure 7 The bus connection is taken as an example.

[0225] The processor 701 is configured to provide computing and control capabilities to control the electronic device 70 to perform corresponding tasks. For example, the processor 70 controls the electronic device 70 to perform the virtual fitting model training method in any of the above method embodiments, including:

[0226] Acquire an image dataset, wherein the image dataset includes clothing images and model images; process the model images through an image-to-text algorithm to generate text description content corresponding to the model images; extract features from the model images to obtain portrait key points; extract features from the model images to obtain a human body analysis map, and obtain a human body retention area map based on the human body analysis map; input the text description content, portrait key points, and clothing images into an image and text dual attention module, perform feature extraction through the image and text dual attention module to obtain guiding features; input the human body retention area map into a data local noise module, and perform feature extraction through the data local noise module. The local noise addition module performs local noise addition to obtain a human body retention area map after local noise addition; the human body retention area map after local noise addition is encoded to obtain encoding features; the guiding features and encoding features are input into the style feature fusion module, and the guiding features and encoding features are fused through the style feature fusion module to obtain fusion features; the fusion features are decoded to obtain a fitting effect map; a model loss function is constructed, and the virtual fitting model is trained based on the image dataset until the model loss function converges to generate a trained virtual fitting model, wherein the model loss function includes text bias loss and image bias loss.

[0227] On the one hand, by generating text description content corresponding to the model image, and extracting features of the portrait key points, human body analysis map, and clothing image through the image and text dual attention module, we obtain the guiding features. The data local denoising module performs local denoising on the human body reserved area map, and further encodes the human body reserved area map to obtain the encoding features. The style feature fusion module is used to fuse the guiding features and encoding features to obtain the fitting effect image, which can improve the generation quality of the fitting effect image.

[0228] On the other hand, by constructing a model loss function for model training, the model loss function includes text bias loss and image bias loss to obtain a trained virtual fitting model. This application can enhance the fitting effect.

[0229] Alternatively, controlling the electronic device 70 to execute the virtual fitting method in any of the above method embodiments includes:

[0230] Obtain images of clothes to be tried on and images of models;

[0231] The image of the clothes to be tried on and the image of the model are input into a virtual fitting model to obtain a fitting image corresponding to the image of the clothes to be tried on, wherein the virtual fitting model is trained based on the method of embodiment 1.

[0232] By training a virtual fitting model, after obtaining the trained virtual fitting model, obtaining images of clothes to be tried on and images of models, inputting the images of clothes to be tried on and images of models into the virtual fitting model, and obtaining fitting images corresponding to the images of clothes to be tried on, this application can improve the virtual fitting effect.

[0233] Processor 701 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or any combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0234] The memory 702 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions / modules corresponding to the training method of the virtual fitting model or the virtual fitting method in the embodiment of the present application. The processor 701 can implement the training method of the virtual fitting model or the virtual fitting method in any of the following method embodiments by running the non-transitory software programs, instructions and modules stored in the memory 702. Specifically, the memory 702 may include a volatile memory (VM), such as a random access memory (RAM); the memory 702 may also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory (flash memory), a hard disk drive (HDD) or a solid-state drive (SSD) or other non-transitory solid-state storage device; the memory 702 may also include a combination of the above types of memories.

[0235] The memory 702 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 702 may optionally include a memory remotely located relative to the processor 701, and such remote memory may be connected to the processor 701 via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0236] One or more modules are stored in the memory 702, and when executed by one or more processors 701, the training method of the virtual fitting model or the virtual fitting method in any of the above method embodiments is executed, for example, the training method described above is executed. Figure 2 、 4 , and the steps shown in 6.

[0237] In the embodiment of the present application, the electronic device 70 may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output. The electronic device 70 may also include other components for realizing the functions of the device, which will not be described in detail here.

[0238] The electronic devices of the present application are available in various forms. Figure 2 、 4 , 6, including but not limited to: servers, mobile terminals, electronic watches, fixed terminals, wearable devices and other devices.

[0239] The present application also provides a computer-readable storage medium, such as a memory including program code, which can be executed by a processor to implement the virtual fitting model training method or virtual fitting method in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CDROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0240] The present application also provides a computer program product comprising one or more program codes stored in a computer-readable storage medium. A processor of an electronic device reads the program code from the computer-readable storage medium and executes the program code to perform the steps of the virtual fitting model training method or virtual fitting method provided in the above embodiments.

[0241] Those skilled in the art will understand that all or part of the steps of implementing the above embodiments can be completed by hardware, or by hardware related to program code, and the program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0242] Through the description of the above embodiments, it is clear to those skilled in the art that each embodiment can be implemented by means of software plus a general hardware platform, or of course by hardware. It is understood by those skilled in the art that all or part of the processes in the above embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0243] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Based on the idea of ​​the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the present application as mentioned above. For the sake of simplicity, they are not provided in detail. Although the present application has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A training method for a virtual fitting model, characterized in that: The virtual fitting model includes a data local noise module, an image and text dual attention module, and a style feature fusion module. The method includes: Acquire an image dataset, wherein the image dataset includes clothing images and model images; Processing the model image using an image-to-text algorithm to generate text description content corresponding to the model image; Performing feature extraction on the model image to obtain portrait key points; Performing feature extraction on the model image to obtain a human body analysis map, and obtaining a human body reserved area map based on the human body analysis map; Inputting the text description content, portrait key points, and clothing image into the image and text dual attention module, performing feature extraction through the image and text dual attention module to obtain guiding features; Inputting the human body reserved area map into the data local noisy module, performing local noise addition through the data local noisy module to obtain a human body reserved area map after local noise addition; Encoding the human body retained area map after local noise addition to obtain encoding features; Inputting the guiding feature and the coding feature into a style feature fusion module, and fusing the guiding feature and the coding feature through the style feature fusion module to obtain a fused feature; Decoding the fused features to obtain a fitting effect image; Constructing a model loss function, and training the virtual fitting model based on the image dataset until the model loss function converges to generate a trained virtual fitting model, wherein the model loss function includes text bias loss and image bias loss.

2. The method according to claim 1, characterized in that The clothing image and the model image correspond one to one, and the method further includes: Preprocessing the clothing image and the model image specifically includes: Through U 2 -Net network extracts the outer contours of the clothing image and the model image, and removes the background information of the clothing image and the model image respectively, to obtain the processed clothing image and the processed model image respectively.

3. The method according to claim 1 or 2, characterized in that The step of extracting features from the model image to obtain key points of the portrait includes: Key points of the model image are detected using a human posture assessment algorithm to obtain portrait key points corresponding to the model image.

4. The method according to claim 1 or 2, characterized in that The step of extracting features from the model image to obtain a human body analysis map, and obtaining a human body reserved area map based on the human body analysis map, comprises: Extract features from the model image using a parsing graph algorithm to obtain a human body parsing graph; The clothing area in the human body analysis image is removed to obtain a human body retained area image.

5. The method according to claim 1 or 2, characterized in that The step of inputting the human body reserved area map into the data local noise adding module, performing local noise adding by the data local noise adding module, and obtaining the human body reserved area map after local noise adding comprises: Among them, q(x t |P parse-cut ) is the unit time t, given the noise-free P parse-cut , which obeys the mean The variance is Normal distribution, x t is the human body retained area map after local noise addition, P parse-cut Keeping area maps for the human body, a i is the coefficient corresponding to the i-th step, I is the unit matrix, M is P parse-cut Binarized edge map of .

6. The method according to claim 1 or 2, characterized in that The image and text dual attention module includes: The encoding layer is used to extract features from the text description content to obtain text information; A first fully connected layer, connected to the encoding layer, is used to process the text information to obtain a text feature vector; A downsampling layer, configured to extract features of the portrait key points and the clothing image to obtain image information; The second fully connected layer is connected to the downsampling layer and is used to process the image information to obtain an image feature vector.

7. The method according to claim 6, characterized in that The feature extraction is performed by the image and text dual attention module to obtain the guiding features, including: Performing feature extraction on the text description content to obtain text information; Processing the text information to obtain a text feature vector; Extracting features of the portrait key points and the clothing image to obtain image information; Processing the image information to obtain an image feature vector; The text feature vector and the image feature vector are fused to obtain a guiding feature.

8. The method according to claim 1 or 2, characterized in that The style feature fusion module includes several residual connection modules, wherein each residual connection module includes a convolution layer, an activation function layer and a batch normalization layer.

9. The method according to claim 1 or 2, characterized in that The model loss function includes: Loss=L txt +λ1L img Among them, Loss is the model loss, L txt is the text bias loss, λ1 is a hyperparameter, L img is the image bias loss.

10. The method according to claim 9, characterized in that The text bias loss includes: Among them, L txt is the text bias loss, P text For text, Clip text For text encoding network, Clip text (P text ) is the text P text Encoding Web Clip via Text text The output feature vector, x gt For real samples, To predict the sample, Clip img is the image encoding network, For the prediction sample Clip through image coding network img Output feature vector, Clip img (x gt ) is the real sample x gt Clip through image coding network img The output feature vector, L1 is the calculated absolute difference.

11. The method according to claim 9, characterized in that The image bias loss includes: Among them, L img is the image bias loss, M is the number of layers of the VGG network, N i is the number of pixels in the i-th layer of the VGG network, x gt For real samples, VGG i (x gt ) is the real sample input into the VGG network to obtain the eigenvalue of the i-th feature scale, For the prediction sample, To predict the sample input into the VGG network to obtain the eigenvalue of the i-th feature scale, L2 is used to calculate the average standard deviation.

12. A virtual fitting method, characterized in that: include: Obtain images of clothes to be tried on and images of models; The clothing image to be tried on and the model image are input into a virtual fitting model to obtain a fitting image corresponding to the clothing image to be tried on, wherein the virtual fitting model is trained based on the method according to any one of claims 1 to 11.

13. An electronic device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor, wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

Images