Virtual clothes changing method and apparatus, computer device, and storage medium
By using multiple sets of samples and dressing models in virtual dressing technology, we learn the positional relationship between objects and clothing, and optimize the model to achieve paired data training, solving the problem of low data resource utilization in the existing technology and achieving high efficiency virtual dressing effect.
Patent Information
- Application Number
- PCT/CN2024/075235
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-04-04
- Filing Date
- 2024-02-01
- Publication Date
- 2024-11-28
AI Technical Summary
The existing virtual dressing technology relies on paired data, resulting in low utilization of data resources and the inability to effectively utilize other data resources other than paired data.
By determining multiple sets of samples, including target clothing data and dressing object data, using the dressing model to be trained to learn the positional relationship between the key points of the object and the original clothing from the dressing object data, virtual dressing processing is carried out based on the positional relationship. And optimize the model based on the difference between the dressing image and the original dressing image to achieve paired data training.
It improves the utilization rate of data resources, can achieve accurate virtual dress change without the need for pairing data, and improves the utilization rate of data resources in clothing categories.
Smart Images

Figure CN2024075235_28112024_PF_FP_ABST
Abstract
Description
Virtual clothing changing method, device, computer equipment and storage medium
[0001] Related applications
[0002] This application claims priority to Chinese patent application number 2023103511493, filed on April 4, 2023, entitled “Virtual Clothing Changing Method, Apparatus, Computer Equipment and Storage Medium,” the entire text of which is incorporated herein by reference. Technical Field
[0003] The present application relates to the field of artificial intelligence technology, and in particular to a virtual clothing changing method, apparatus, computer equipment, and storage medium. Background Art
[0004] E-commerce has grown rapidly in recent years, with more and more consumers choosing to shop on e-commerce platforms. However, when purchasing non-standard items such as clothing online, consumers can only rely on the data provided by the merchant and their personal experience. Therefore, virtual clothing-changing technology has emerged.
[0005] Traditional techniques require at least paired data—a set of matching clothing items and images of models wearing them—to achieve virtual clothing changes through deep learning. This approach relies too heavily on paired data, making inefficient use of data beyond the paired data and resulting in a waste of data resources.
[0006] Summary of the Invention
[0007] According to various embodiments of the present application, a virtual clothing changing method, apparatus, computer device, computer-readable storage medium, and computer program product are provided that can improve data resource utilization.
[0008] In a first aspect, the present application provides a virtual clothing changing method. The method comprises:
[0009] Determine multiple groups of samples; each group of samples includes target clothing data and clothing-changing subject data; the clothing-changing subject data is extracted from an original clothing image showing the subject wearing the original clothing; the category of the original clothing is consistent with the category of the target clothing represented by the target clothing data;
[0010] The clothing-changing model to be trained learns the positional relationship between the key points of the object and the key points of the original clothing from the clothing-changing object data; the key points of clothing in the same category have consistent semantics;
[0011] Performing virtual clothing-changing processing on the clothing-changing subject data and the target clothing data based on the positional relationship to obtain a clothing-changing image output by the clothing-changing model to be trained; the clothing-changing image presents the effect of the subject wearing the target clothing;
[0012] The target loss is determined according to the difference between the clothing-changing image and the original clothing-wearing image, and the clothing-changing model to be trained is optimized according to the target loss to obtain a trained clothing-changing model.
[0013] In some embodiments, the clothing-changing object data includes object skeleton data and original clothing key point data; the original clothing key point data is used to represent key points of the original clothing; and learning the positional relationship between the key points of the object and the key points of the original clothing from the clothing-changing object data by the clothing-changing model to be trained includes:
[0014] Extracting skeleton key point data from the object skeleton data based on the gating mechanism in the clothing changing model to be trained;
[0015] By associating the skeleton key point data with the original clothing key point data, the positional relationship between the skeleton key points of the object and the key points of the original clothing is learned.
[0016] In some embodiments, the clothing-changing object data includes object texture data and region mask data; the region mask data refers to the mask of the area remaining after removing the area that affects the wearing of the target clothing; skeleton key point data is extracted from the object skeleton data based on the gating mechanism in the clothing-changing model to be trained, including:
[0017] fusing the object texture data and the region mask data to obtain object fusion data;
[0018] Extracting wearing area data from the object fusion data according to the gated data in the clothing changing model to be trained; the wearing area data is used to indicate the wearing area of the object when wearing the target clothing;
[0019] The skeleton key point data is extracted from the object skeleton data according to the wearing area data.
[0020] In some embodiments, performing virtual clothing-changing processing on the clothing-changing object data and the target clothing data based on the positional relationship to obtain a clothing-changing image output by the clothing-changing model to be trained includes:
[0021] Mapping the key points of the target garment according to the positional relationship to obtain target key point data of the target garment; the target key point data is used to characterize the positions of the key points of the target garment when the subject wears the target garment;
[0022] The target clothing data is adjusted based on the target key point data to obtain deformable clothing data; the deformable clothing represented by the deformable clothing data matches the posture of the object;
[0023] The deformable clothing data and the clothing-changing object data are subjected to virtual clothing-changing processing to obtain clothing-changing images output by the clothing-changing model to be trained.
[0024] In some embodiments, adjusting the target clothing data based on the target key point data to obtain deformable clothing data includes:
[0025] Extract features from target key point data and target clothing data to obtain deformation feature data;
[0026] The target clothing image in the target clothing data is deformed according to the deformation feature data to obtain deformed clothing data.
[0027] In some embodiments, the target loss includes a first loss; the first loss is determined based on the difference between the clothing-changing image and the original clothing-wearing image; the target loss also includes at least one of a second loss or a third loss; the second loss is determined based on the difference between the target clothing key point data and the original key point data; the third loss is determined based on the difference between the deformed clothing data and the original clothing data in the clothing-changing object data.
[0028] In some embodiments, performing virtual clothing-changing processing on the deformable clothing data and the clothing-changing object data to obtain a clothing-changing image output by the clothing-changing model to be trained includes:
[0029] Performing a fusion process on the deformable clothing data and the clothing-changing object data at a first resolution to obtain a clothing-changing result corresponding to the first resolution;
[0030] Starting from the next resolution of the first resolution, the current resolution is determined in sequence, and the deformable clothing data, the clothing changing object data, and the clothing changing result corresponding to the previous resolution of the current resolution are fused to obtain the clothing changing result corresponding to the current resolution;
[0031] The clothing-changing result corresponding to the last resolution is subjected to nonlinear mapping processing to obtain the clothing-changing image.
[0032] In a second aspect, the present application also provides a virtual fitting device. The device includes:
[0033] a determination module for determining a plurality of groups of samples; each group of samples includes target clothing data and clothing-changing subject data; the clothing-changing subject data is extracted from an original clothing image showing the subject wearing the original clothing; the category of the original clothing is consistent with the category of the target clothing represented by the target clothing data;
[0034] A learning module is used to learn the positional relationship between the key points of the object and the key points of the original clothing from the clothing changing object data through the clothing changing model to be trained; the key points of clothing in the same category have the same semantics;
[0035] An output module is used to perform virtual clothing-changing processing on the clothing-changing subject data and the target clothing data based on the positional relationship, thereby obtaining a clothing-changing image output by the clothing-changing model to be trained; the clothing-changing image presents the effect of the subject wearing the target clothing;
[0036] The optimization module is used to determine the target loss according to the difference between the clothing-changing image and the original clothing-wearing image, and optimize the clothing-changing model to be trained according to the target loss to obtain a trained clothing-changing model.
[0037] In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.
[0038] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned method when executed by a processor.
[0039] In a fifth aspect, the present application further provides a computer program product, which includes a computer program that implements the steps of the above method when executed by a processor.
[0040] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.
[0042] FIG1 is a schematic flow chart of a virtual clothing changing method in some embodiments;
[0043] FIG2 is a schematic diagram of the structure of a key point prediction unit in a clothing changing model in some embodiments;
[0044] FIG3 is a schematic diagram of the structure of a clothing deformation unit in a clothing changing model in some embodiments;
[0045] FIG4 is a schematic structural diagram of a try-on unit in a clothing changing model in some embodiments;
[0046] FIG5 is a schematic diagram of the structure of a residual block in some embodiments;
[0047] FIG6 is a schematic structural diagram of a clothing-changing model in some embodiments;
[0048] FIG7 is a structural block diagram of a virtual clothing changing device in some embodiments;
[0049] FIG8 is a diagram illustrating the internal structure of a computer device in some embodiments;
[0050] FIG9 is a diagram showing the internal structure of a computer device in some other embodiments. DETAILED DESCRIPTION
[0051] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0052] In some embodiments, as shown in FIG1 , a virtual clothing changing method is provided. The method is described by taking the application of the method to a computer device as an example. It can be understood that the computer device includes at least one of a terminal or a server. The method includes steps S102 - S108 .
[0053] S102, determining multiple groups of samples.
[0054] Among them, each group of samples includes target clothing data and clothing-changing object data; the clothing-changing object data is extracted from the original dressing image of the presenting object wearing the original clothing; the category of the original clothing is consistent with the target clothing represented by the target clothing data.
[0055] For example, the computer device can search for original clothing images of the subject wearing original clothing of the target clothing category. The computer device can use the target clothing and the original clothing images as a set of training data. It is understood that since the virtual clothing-changing method provided in this embodiment uses unpaired data training, that is, there are no product clothing and corresponding clothing images of the subject wearing the product clothing, it is necessary to rely on the category of the product clothing to search for original clothing images of the subject wearing clothing of the same category and randomly select one of the original clothing images to form a set of training data.
[0056] The computer device can obtain target clothing data of the target clothing and extract clothing object data from the original clothing image for each set of training data to obtain each set of samples.
[0057] In some embodiments, the computer device may obtain a classification tag (cloth-class) marked for the target clothing to determine the category of the target clothing. The classification tag may include at least one of clothing types such as long sleeves, short sleeves, dresses, skirts, trousers, or shorts.
[0058] In some embodiments, the target garment data may include target garment key point data (cloth-points), a target garment image, a target garment mask (cloth-mask), and a classification label of the target garment. The target garment key point data is used to characterize the positions of each key point on the initial target garment. The target garment key point data may be the key point coordinates of the target garment. The target garment mask is a mask corresponding to the target garment image. The classification label of the target garment is used to indicate the category of the target garment.
[0059] In some embodiments, the clothing-changing object data further includes an original clothing image and a corresponding original image mask. It is understood that the original clothing image presents the deformed original clothing worn on the object.
[0060] In some embodiments, the clothing-changing object data may include the original clothing image, the classification label of the original clothing (person-cloth-class), object texture data (densepose-uv), object skeleton data (pose), original key point data (person-cloth-points), object region image (agno-person), region mask data (agno-parse), and object component segmentation mask (human parse). The classification label of the original clothing is used to indicate the category of the original clothing. The object texture data is used to represent the posture of the object in the texture space. The object texture data can be the texture mapping coordinates of the object. The object skeleton data refers to the medial axis of the object, which maintains the shape and structure information of the object and can reflect the shape and structure of the object. The original key point data is used to represent the key point positions of the original clothing when the object is wearing the original clothing. The original key point data can be the key point coordinates of the original clothing. The region mask data is used to represent the mask of the area remaining after the area that affects the wearing of the target clothing is removed. The region mask data corresponds to the object region image. The object region image is the image of the area remaining after the area that affects the wearing of the target clothing is removed. It is understood that if the target garment and the original garment are both short-sleeved, the object region image may be the image of the area remaining after removing the original garment, arms, and neck regions. A component segmentation mask is a mask that represents multiple semantically consistent regions on an object. A component segmentation mask may include at least one of the following: hair, face, legs, arms, or tops.
[0061] In some embodiments, terminals may include, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. The server may be implemented as a standalone server or a server cluster consisting of multiple servers.
[0062] S104 , learning the positional relationship between the key points of the object and the key points of the original clothing from the clothing-changing object data through the clothing-changing model to be trained.
[0063] Among them, the key points of clothing in the same category have the same semantics. The semantic consistency here means that the key points at corresponding positions of clothing in the same category represent the same semantics, for example, they all represent collars or sleeves, etc. The number of key points of clothing in the same category is consistent. It can be understood that the number and position of key points of clothing in different categories are different. The key points of clothing can be used to locate the position of each area of the clothing. For example, for clothing in the short-sleeved category, 13 key points can be included. The left and right sleeves can each include 4 key points, the hem of the sleeve can have 2 key points, and the connection between the sleeve and the torso area can have 2 key points. The collar can include 3 key points, respectively at the center and both sides of the collar. The hem of short-sleeved clothing can have 2 key points.
[0064] For example, the computer device may use each set of samples as input to a clothing-changing model to be trained. The computer device may then extract skeleton keypoint data from the object's skeleton data using the clothing-changing model to be trained. The skeleton keypoint data represents keypoints on the object associated with the original clothing being worn. The computer device may learn the positional relationship between the keypoints of the object and the keypoints of the original clothing by correlating the skeleton keypoint data with the original clothing keypoint data.
[0065] In some embodiments, the object may be a human model. The pose of the object is not changed during the inference process of the clothing changing model.
[0066] In some embodiments, the object skeleton data may be the object skeleton coordinates. The original key point data may be the key point coordinates of the original clothing. The computer device may associate the object skeleton coordinates with the key point coordinates of the original clothing through the attention mechanism in the clothing changing model to be trained to obtain the positional relationship between the object's skeleton key points and the key points of the original clothing when the object wears the original clothing. It can be understood that the key points of the object include the skeleton key points. When wearing clothing of the same category, the positional relationship between the object's key points and the key points of the worn clothing is consistent, that is, the positional relationship when wearing corresponding clothing of the same category is consistent to a certain extent.
[0067] In some embodiments, the object skeleton data may be data in the form of a heat map.
[0068] S106 , performing virtual clothing changing processing on the clothing changing object data and the target clothing data based on the positional relationship, and obtaining a clothing changing image output by the clothing changing model to be trained.
[0069] Among them, the clothing-changing image presents the effect of the subject wearing the target clothing.
[0070] For example, the computer device may adjust the target garment data based on the positional relationship to obtain deformable garment data. The relationship between the key points of the deformable garment represented by the deformable garment data and the skeleton key points of the object conforms to the aforementioned positional relationship. The computer device may perform a virtual clothing change process on the deformable garment data and the clothing change object data to obtain a clothing change image.
[0071] S108, determining a target loss according to the difference between the clothes-changing image and the original clothes-wearing image, and optimizing the clothes-changing model to be trained according to the target loss to obtain a trained clothes-changing model.
[0072] For example, the target loss may include a clothing-changing image discriminator loss. The computer device may use the clothing-changing image and the original clothing-wearing image as inputs to the discriminator, and obtain the clothing-changing image discriminator loss as output by the discriminator. It will be understood that the discriminator is trained based on real clothing-wearing images. The discriminator can determine whether the input image is a real clothing-wearing image. The computer device may optimize the clothing-changing model to be trained in a direction that reduces the target loss, thereby obtaining a trained clothing-changing model.
[0073] In some embodiments, the target loss may include a perceptual loss for the clothing-changing image. The computer device may calculate the perceptual loss for the clothing-changing image and the original image to obtain the perceptual loss for the clothing-changing image. It is understood that the perceptual loss calculation may be implemented using a perceptual loss function based on a Visual Geometry Group Network (VGG).
[0074] In some embodiments, a computer device may determine garment data to be processed and object data to be processed. The object data to be processed is extracted from an image showing the subject to be changed wearing clothing of the same type as the garment to be processed. The garment data to be processed is used to characterize the garment to be processed. The computer device may use the garment data to be processed and the object data to be processed as input to a trained clothing-changing model, thereby obtaining a target clothing-changing image output by the clothing-changing model. It will be appreciated that in a real-world scenario, when a user selects a product for virtual try-on, they only need to provide an image of the user wearing the same type of clothing to achieve a virtual clothing change.
[0075] In the above-mentioned virtual clothing-changing method, the target clothing data and clothing-changing object data in each set of samples are used as training data for the clothing-changing model to be trained. The clothing-changing object data is extracted from the original clothing-dressing image. The target clothing represented by the target clothing data is not the original clothing in the original clothing-dressing image. The target clothing and the original clothing do not need to be completely identical, as long as they are of the same category. The clothing-changing model to be trained learns the positional relationship between the key points of the object and the key points of the original clothing from the clothing-changing object data. The key points of clothing of the same category correspond to the same semantics. The positional relationship can, to a certain extent, reflect the relationship between the key points of the target clothing and the key points of the object. Based on the positional relationship, virtual clothing-changing processing can be achieved for the clothing-changing object data and the target clothing data, and an accurate clothing-changing image can be obtained. The target loss is determined based on the difference between the clothing-changing image and the original image. The clothing-changing model to be trained is optimized according to the target loss to obtain a trained clothing-changing model. The target clothing and the original clothing only need to be of the same category, not completely identical, which greatly improves the utilization of data resources.
[0076] In some embodiments, the clothing-changing object data includes object skeleton data and original key point data; the original key point data is used to characterize the key points of the original clothing; the positional relationship between the key points of the object and the key points of the original clothing is learned from the clothing-changing object data through the clothing-changing model to be trained, including: extracting skeleton key point data from the object skeleton data based on the gating mechanism in the clothing-changing model to be trained; and learning the positional relationship between the skeleton key points of the object and the key points of the original clothing by associating the skeleton key point data with the original key point data.
[0077] For example, the computer device can extract skeleton keypoint data from the subject's skeleton data based on a gating mechanism in a clothing-changing model to be trained. It is understood that when a subject wears clothing, the clothing often does not cover the entire subject, resulting in areas of the subject that are relevant to the clothing and areas that are irrelevant to the clothing. The gating mechanism in the clothing-changing model is used to restrict areas that are irrelevant to the clothing and retain areas that are friendly to the clothing. The computer device can fuse the skeleton keypoint data with the original keypoint data to obtain positional association data. The positional association data is used to characterize the positional relationship between the skeleton keypoints and the keypoints of the original clothing.
[0078] In some embodiments, the computer device may perform a preliminary fusion process on the skeleton key point data and the original key point data to obtain preliminary fusion data. The computer device may perform an advanced fusion process on the preliminary fusion data and the original key point data to obtain position association data.
[0079] In some embodiments, the computer device may encode the skeleton key point data to obtain encoded skeleton key point data. The computer device may extract the skeleton key point data from the encoded skeleton key point data.
[0080] In some embodiments, the computer device may perform multi-stage encoding processing on the original key point data to obtain encoded original key point data. For example, the multi-stage encoding processing may be a two-stage encoding processing.
[0081] In some embodiments, the computer device may encode the skeleton key point data to obtain encoded skeleton key point data. The computer device may perform a matrix multiplication operation on the encoded skeleton key point data and the encoded original key point data to obtain preliminary fused data. The computer device may perform a matrix multiplication operation on the preliminary fused data and the encoded original key point data to obtain position association data.
[0082] In some embodiments, the encoding process may be a convolution process followed by instance normalization and finally a nonlinear activation (conv-IN-prelu).
[0083] In this embodiment, skeleton key point data is extracted from the object skeleton data based on the gating mechanism in the clothing changing model to be trained; by associating the skeleton key point data with the original key point data, the positional relationship between the skeleton key points of the object and the key points of the original clothing is learned. The positional relationship can reflect the relationship between the key points of the target clothing and the key points of the object to a certain extent, and then based on the positional relationship, virtual clothing changing of the object and the target clothing can be realized to obtain an accurate clothing changing image.
[0084] In some embodiments, the clothing-changing object data includes object texture data and region mask data; the region mask data refers to the mask of the area remaining after removing the area that affects the wearing of the target clothing; skeleton key point data is extracted from the object skeleton data based on the gating mechanism in the clothing-changing model to be trained, including: fusing the object texture data and the region mask data to obtain object fusion data; extracting wearing area data from the object fusion data according to the gating data in the clothing-changing model to be trained; the wearing area data is used to indicate the wearing area when the object wears the target clothing; and skeleton key point data is extracted from the object skeleton data according to the wearing area data.
[0085] Exemplarily, the computer device can splice the object texture data (buv) and the region mask data (bnc) to obtain object fusion data (bnc|buv). It can be understood that the region mask data lacks the area related to the target clothing, while the object texture data is complete, and the two can be complemented by splicing them. The computer device can encode the object fusion data to obtain encoded object fusion data. The computer device can extract the wearing area data from the encoded object fusion data through the first gate data in the clothing changing model to be trained. The computer device can perform weighted mapping on the encoded object skeleton data through the wearing area data to obtain skeleton key point data. It can be understood that the wearing area data is actually the attention weight of the encoded object skeleton data.
[0086] In some embodiments, the computer device may perform an element-wise multiplication operation on the wearing area data and the encoded object skeleton data to obtain skeleton key point data.
[0087] In this embodiment, the object texture data and the region mask data are fused to obtain the object fusion data; the wearing region data is extracted from the object fusion data according to the gating data in the clothing changing model to be trained; the skeleton key point data is extracted from the object skeleton data according to the wearing region data, and the skeleton key point data is subsequently associated with the original key point data to extract the positional relationship. The positional relationship can, to a certain extent, reflect the relationship between the key points of the target clothing and the key points of the object when the object wears the target clothing, and thus an accurate clothing changing image can be obtained based on the positional relationship.
[0088] In some embodiments, a virtual clothing changing process is performed on the clothing changing object data and the target clothing data based on the positional relationship to obtain a clothing changing image output by the clothing changing model to be trained, including: mapping the key points of the target clothing according to the positional relationship to obtain target key point data of the target clothing; the target key point data is used to characterize the positions of the key points of the target clothing when the object wears the target clothing; the target clothing data is adjusted based on the target key point data to obtain deformable clothing data; the deformable clothing represented by the deformable clothing data matches the posture of the object; the deformable clothing data and the clothing changing object data are virtually processed to obtain a clothing changing image output by the clothing changing model to be trained.
[0089] For example, the computer device may perform weighted mapping on the target clothing key point data using the position association data to obtain weighted key point data. The computer device may perform superposition processing on the weighted key point data and the position association data to obtain target key point data of the target clothing.
[0090] The computer device can fuse the target key point data and the target clothing data to obtain deformable clothing data. It will be appreciated that during the above fusion process, the target key point data will guide the adjustment direction of the target clothing data. The computer device can fuse the deformable clothing data and the clothing change object data to obtain a clothing change image.
[0091] In some embodiments, the computer device may perform an element-wise multiplication operation on the position association data and the target clothing key point data to obtain weighted key point data.
[0092] In some embodiments, the computer device may perform an element-by-element addition operation on the weighted key point data and position association data to obtain target key point data of the target garment.
[0093] In some embodiments, the computer device may encode the position-associated data to obtain encoded position-associated data. The computer device may perform weighted mapping on the target garment key point data using the encoded position-associated data to obtain weighted key point data. The computer device may superimpose the weighted key point data and the encoded position-associated data and then perform encoding processing to obtain target key point data of the target garment.
[0094] In some embodiments, the computer device may perform multi-resolution fusion processing on the deformable clothing data and the clothing-changing object data to obtain a clothing-changing image.
[0095] In this embodiment, the key points of the target clothing are mapped according to the positional relationship to obtain the target key point data of the target clothing; the target clothing data is adjusted based on the target key point data to obtain the deformed clothing data; the deformed clothing data and the clothing-changing object data are subjected to virtual clothing-changing processing to obtain the clothing-changing image output by the clothing-changing model to be trained, the deformation of the target clothing is guided by the positional relationship, and the deformed clothing is then worn on the object to obtain an accurate clothing-changing image.
[0096] In some embodiments, as shown in FIG2 , there is a schematic diagram of the structure of the key point prediction unit in the clothing changing model. The output of the key point prediction unit includes target key point data and predicted component segmentation mask. The computer device can extract advanced region data from the wearing region data through the second gated data in the clothing changing model to be trained. The advanced region data is more detailed than the wearing region data, and the region represented by the wearing region data is larger than the region represented by the advanced region data. In order to avoid incomplete extraction of positional relationships, the wearing region data can be used. The computer device can encode the advanced region data to obtain encoded advanced region data. It should be noted that the coding of the eyes of the task object in the image in the embodiment of the present application is for privacy protection considerations and is not the processing of this application.
[0097] The computer device may perform weighted mapping processing on the target clothing mask using the advanced region data to obtain a weighted target clothing mask. It is understood that the computer device may specifically encode the target clothing mask to obtain an encoded target clothing mask. The computer device may perform an element-wise multiplication operation on the advanced region data and the encoded target clothing mask to obtain a weighted target clothing mask.
[0098] The computer device may fuse the target key point data and the weighted target clothing mask to obtain preliminary clothing fusion data. It is understood that the computer device may perform advanced encoding on the weighted target clothing mask to obtain an advanced-encoded target clothing mask. The computer device may perform a dimension transformation 1 on the advanced-encoded target clothing mask and the target key point data so that their dimensions match, thereby obtaining a dimensionally transformed target clothing mask and target key point data. The computer device may perform a matrix multiplication operation on the dimensionally transformed target clothing mask and the target key point data to obtain preliminary clothing fusion data.
[0099] The computer device may perform advanced fusion processing on the preliminary clothing fusion data and the weighted target clothing mask to obtain clothing fusion data. It is understood that the computer device may specifically perform a dimensionality transformation 2 on the preliminary clothing fusion data so that the dimension of the transformed preliminary clothing fusion data matches the dimension of the advanced-encoded target clothing mask. The computer device may perform a matrix multiplication operation on the dimensionally transformed preliminary clothing fusion data and the advanced-encoded target clothing mask to obtain clothing fusion data.
[0100] The computer device may perform weighted mapping on the advanced region data using the clothing fusion data to obtain weighted advanced region data. It is understood that the computer device may specifically encode the clothing fusion data to obtain encoded clothing fusion data. The computer device may perform an element-wise multiplication operation on the encoded clothing fusion data and the encoded advanced region data to obtain weighted advanced region data.
[0101] The computer device may perform an element-wise addition operation on the encoded clothing fusion data and the weighted advanced region data to obtain a predicted component segmentation mask. It is understood that the computer device may specifically perform an element-wise addition operation on the encoded clothing fusion data and the weighted advanced region data to obtain a predicted component segmentation mask. The predicted component segmentation mask is the component segmentation mask of the subject wearing the target clothing.
[0102] The computer device may perform a dimension transformation on the encoded skeleton key point data and the encoded original key point data, and then perform a matrix multiplication operation to obtain preliminary fused data. The computer device may perform a dimension transformation on the preliminary fused data to obtain dimension-transformed preliminary fused data. The computer device may perform a matrix multiplication operation on the dimension-transformed preliminary fused data and the encoded original key point data to obtain position association data.
[0103] In some embodiments, the clothing changing model may include multiple cascaded key point prediction units, and the output of each key point prediction unit serves as the input of the next key point prediction unit.
[0104] In some embodiments, the target clothing data is adjusted based on the target key point data to obtain deformed clothing data, including: extracting features from the target key point data and the target clothing data to obtain deformed feature data; and deforming the target clothing image in the target clothing data according to the deformation feature data to obtain deformed clothing data.
[0105] For example, the computer device may encode the target key point data, the predicted component segmentation mask, and the object skeleton data to obtain object encoded data. The computer device may encode the target clothing mask and the target clothing image to obtain target clothing encoded data. The computer device may fuse the object encoded data and the target clothing encoded data to obtain deformation feature data. The computer device may interpolate the target clothing image based on the deformation feature data to obtain deformed clothing data. The deformed clothing data may include a deformed clothing image and a deformed clothing mask.
[0106] In some embodiments, the computer device may perform preliminary fusion processing on the object coding data and the target clothing coding data to obtain initial deformation feature data. The computer device may perform advanced fusion processing on the object coding data and the initial deformation feature data to obtain advanced deformation feature data. The computer device may perform weighted mapping processing on the target clothing coding data using the advanced deformation feature data to obtain weighted target clothing coding data. The computer device may perform superposition processing on the advanced deformation feature data and the weighted target clothing coding data to obtain deformation feature data. The computer device may perform bilinear interpolation processing on the target clothing image based on the deformation feature data to obtain deformed clothing data. The bilinear interpolation processing may be implemented by grid sampling.
[0107] In some embodiments, as shown in FIG3 , a schematic diagram of the structure of the clothing deformation unit in the clothing changing model is shown. The computer device can use the target key point data, the predicted component segmentation mask, and the object skeleton data as inputs to the multi-layer encoder, and encode the output of the multi-layer encoder to obtain object encoding data. The computer device can use the target clothing mask and the target clothing image as inputs to the multi-layer encoder, and encode the output of the multi-layer encoder to obtain target clothing encoding data. The multi-layer encoder is used to extract features from high to low resolution. As the encoding level increases, the lower the resolution, the higher the dimensionality of the corresponding features.
[0108] The computer device may perform a dimensionality transformation 1 on the target clothing encoding data and the object encoding data, and perform a matrix multiplication operation on the transformed target clothing encoding data and the object encoding data to obtain initial deformation feature data. The computer device may perform a dimensionality transformation 2 on the initial deformation feature data, and perform a matrix multiplication operation on the transformed initial deformation feature data and the object encoding data to obtain advanced deformation feature data. The computer device may encode the advanced deformation feature data to obtain encoded advanced deformation feature data. The computer device may perform an element-wise multiplication operation on the encoded advanced deformation feature data and the target clothing encoding data to obtain weighted target clothing encoding data. The computer device may perform an element-wise addition operation on the weighted target clothing encoding data and the encoded advanced deformation feature data to obtain deformation feature data. The computer device may perform a bilinear interpolation operation on the target clothing image based on the deformation feature data to obtain deformed clothing data.
[0109] In some embodiments, the computer device may perform bilinear interpolation processing on the target clothing image and the target clothing mask according to the deformation feature data to obtain deformable clothing data.
[0110] In some embodiments, the deformation feature data is a flow feature (flow).
[0111] In this embodiment, feature extraction is performed on target key point data and target clothing data to obtain deformation feature data; the target clothing image in the target clothing data is deformed according to the deformation feature data to obtain deformed clothing data, and the target clothing image is deformed by guiding the deformation of the deformation feature data so that the deformed clothing data matches the shape of the target clothing when the object wears the target clothing, thereby achieving accurate virtual clothing changing.
[0112] In some embodiments, the target loss includes a first loss; the first loss is determined based on the difference between the clothing-changing image and the original clothing-wearing image; the target loss also includes at least one of a second loss or a third loss; the second loss is determined based on the difference between the target clothing key point data and the original key point data; the third loss is determined based on the difference between the deformed clothing data and the original clothing data in the clothing-changing object data.
[0113] For example, the first loss may include a clothing-changing image perception loss and a clothing-changing image discriminator loss. The computer device may input the clothing-changing image and the original clothing-wearing image into the first discriminator, and obtain the clothing-changing image discriminator loss as output by the first discriminator. The first loss is used to evaluate the performance of the try-on unit.
[0114] The computer device can input the predicted component segmentation mask, the component segmentation mask of the object, and the object skeleton data into the second discriminator to obtain the segmentation mask discriminator loss output by the second discriminator. The second discriminator is trained based on the real component segmentation mask and can determine whether the input component segmentation mask is a real component segmentation mask. The computer device can calculate the mean absolute error of the target clothing key point data and the original key point data to obtain the key point loss. The computer device can calculate the perceptual loss of the predicted component segmentation mask and the component segmentation mask of the object to obtain the segmentation mask perceptual loss. The second loss includes the segmentation mask discriminator loss, the key point loss, and the segmentation mask perceptual loss. The second loss is used to evaluate the performance of the key point prediction unit.
[0115] The computer device can input the deformable clothing image and the deformable clothing mask into a third discriminator, obtaining a deformable clothing discriminator loss as an output of the third discriminator. The third discriminator is trained based on real clothing worn by a person and can determine whether the input deformable clothing is a real deformable clothing. The computer device can calculate a perceptual loss on the deformable clothing mask and the original clothing mask to obtain a deformable clothing perceptual loss. The computer device can perform classification prediction on the deformable clothing image and the original clothing image to obtain a deformable clothing category loss. It is understood that the deformable clothing and the original clothing should belong to the same category. The computer device can calculate an adjacent gradient smoothing loss on the deformable feature data obtained in each round of training to obtain a deformable feature loss. The third loss includes the deformable clothing discriminator loss, the deformable clothing perceptual loss, the deformable clothing category loss, and the deformable feature loss.
[0116] The computer device can perform a weighted summation of the first loss, the second loss, and the third loss to obtain a target loss, and optimize the clothing changing model to be trained in the direction of reducing the target loss to obtain a trained clothing changing model.
[0117] In this embodiment, the optimization of the clothing changing model to be trained is supervised by multiple losses, which can ensure the accuracy of the clothing changing model training process and the performance of the trained clothing changing model.
[0118] In some embodiments, a virtual clothing changing process is performed on the deformable clothing data and the clothing changing object data to obtain a clothing changing image output by the clothing changing model to be trained, including: performing a fusion process on the deformable clothing data and the clothing changing object data at a first resolution to obtain a clothing changing result corresponding to the first resolution; starting from the next resolution of the first resolution, the current resolution is determined in sequence, and the deformable clothing data, the clothing changing object data and the clothing changing result corresponding to the previous resolution of the current resolution are fused to obtain a clothing changing result corresponding to the current resolution; and a nonlinear mapping process is performed on the clothing changing result corresponding to the last resolution to obtain a clothing changing image.
[0119] Exemplarily, the computer device may encode the object region image, region mask data, object skeleton data, object texture data, and deformable clothing data to obtain first encoded data. The computer device may perform a fusion process at a first resolution on the target key point data, the predicted component segmentation mask, the first encoded data, the object region image, the region mask data, the object skeleton data, the object texture data, and the deformable clothing data to obtain a clothing-changing result corresponding to the first resolution. The computer device may sequentially determine the current resolution starting from the next resolution below the first resolution, and perform a fusion process on the target key point data, the predicted component segmentation mask, the object region image, the region mask data, the object skeleton data, the object texture data, the deformable clothing data, and the clothing-changing result corresponding to the resolution above the current resolution to obtain a clothing-changing result corresponding to the current resolution. The computer device may encode the clothing-changing result corresponding to the last resolution and then perform nonlinear activation to obtain a clothing-changing image. Each resolution is smaller than the next resolution.
[0120] In some embodiments, as shown in FIG4 , there is shown a schematic diagram of the structure of the try-on unit in the clothing changing model. The computer device may use the first encoded data, the deformed clothing data, and the clothing changing object data as the input of the residual block at the first resolution, and obtain the clothing changing result corresponding to the first resolution output by the residual block at the first resolution. Starting from the next resolution of the first resolution, the current resolution is determined in sequence, and the deformed clothing data, the clothing changing object data, and the clothing changing result corresponding to the previous resolution of the current resolution are used as the input of the residual block at the current resolution, and obtain the clothing changing result corresponding to the current resolution output by the residual block at the current resolution. The computer device may encode the clothing changing result corresponding to the last resolution output by the residual block at the last resolution and perform nonlinear activation to obtain a clothing changing image. The nonlinear activation function may be a hyperbolic tangent function.
[0121] In one embodiment, as shown in FIG5 , there is a schematic diagram of the structure of the residual block. Since each residual block has an output at a different resolution, the residual block at the current resolution can be upsampled to the clothing-changing result corresponding to the previous resolution, and the upsampled clothing-changing result conforms to the current resolution. The computer device can perform an initial convolution process and then an advanced convolution process on the target key point data and the predicted component segmentation mask to obtain second encoded data. The residual block at the current resolution can merge the object region image, region mask data, object skeleton data, object texture data, deformable clothing data, and the upsampled clothing-changing result to obtain merged data. The computer device can perform an element-wise multiplication operation on the merged data and the second encoded data to obtain an initial clothing-changing result. The computer device can perform an element-wise addition operation on the second encoded data and the initial clothing-changing result to obtain a clothing-changing result.
[0122] In some embodiments, as shown in Figure 6, it is a structural diagram of the clothing changing model. The categories of the target clothing and the original clothing can be short sleeves. The parts pointed by the arrows are the outputs of each unit. The clothing changing model includes a key point prediction unit, a clothing deformation unit and a try-on unit. The input of the key point prediction unit includes the target clothing key point data, the target clothing mask, the region mask data, the object skeleton data, the object texture data and the original key point data. The output of the key point prediction unit includes the predicted component segmentation mask and the target key point data. The input of the clothing deformation unit includes the predicted component segmentation mask, the target key point data, the object skeleton data, the target clothing mask and the target clothing image. The output of the clothing deformation unit includes the deformed clothing mask and the deformed clothing image. The input of the try-on unit includes the deformed clothing mask, the deformed clothing image, the object region image, the region mask data, the object skeleton data and the object texture data. The output of the try-on unit includes the clothing changing image.
[0123] In this embodiment, the deformable clothing data and the clothing-changing object data are fused at a first resolution to obtain a clothing-changing result corresponding to the first resolution; starting from the next resolution of the first resolution, the current resolution is determined in sequence, and the deformable clothing data, the clothing-changing object data and the clothing-changing result corresponding to the previous resolution of the current resolution are fused to obtain a clothing-changing result corresponding to the current resolution; the clothing-changing result corresponding to the last resolution is nonlinearly mapped to obtain a clothing-changing image, and an accurate clothing-changing image is obtained through multi-resolution fusion processing combined with multi-resolution features.
[0124] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.
[0125] Based on the same inventive concept, embodiments of the present application also provide a virtual clothing-changing device for implementing the aforementioned virtual clothing-changing method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the virtual clothing-changing device provided below can be found in the above-described limitations of the virtual clothing-changing method and will not be further elaborated here.
[0126] In some embodiments, as shown in FIG7 , a virtual clothing changing device 700 is provided, comprising: a determination module 702 , a learning module 704 , an output module 706 , and an optimization module 708 , wherein:
[0127] Determination module 702 is used to determine multiple groups of samples; each group of samples includes target clothing data and clothing-changing object data; the clothing-changing object data is extracted from the original dressing image of the subject wearing the original clothing; the category of the original clothing is consistent with the category of the target clothing represented by the target clothing data.
[0128] The learning module 704 is used to learn the positional relationship between the key points of the object and the key points of the original clothing from the clothing changing object data through the clothing changing model to be trained; the key points of clothing in the same category have the same semantics.
[0129] The output module 706 is used to perform virtual clothing-changing processing on the clothing-changing object data and the target clothing data based on the position relationship, and obtain a clothing-changing image output by the clothing-changing model to be trained; the clothing-changing image presents the effect of the object wearing the target clothing.
[0130] The optimization module 708 is used to determine the target loss according to the difference between the clothing-changing image and the original clothing-wearing image, and optimize the clothing-changing model to be trained according to the target loss to obtain a trained clothing-changing model.
[0131] In some embodiments, the clothing-changing object data includes object skeleton data and original key point data; the original key point data is used to characterize the key points of the original clothing; the learning module 704 is used to extract the skeleton key point data from the object skeleton data based on the gating mechanism in the clothing-changing model to be trained; by associating the skeleton key point data with the original key point data, the positional relationship between the skeleton key points of the learning object and the key points of the original clothing is learned.
[0132] In some embodiments, the clothing-changing object data includes object texture data and region mask data; the region mask data refers to the mask of the area remaining after removing the area that affects the wearing of the target clothing; the learning module 704 is used to fuse the object texture data and the region mask data to obtain object fusion data; the wearing region data is extracted from the object fusion data according to the gating data in the clothing-changing model to be trained; the wearing region data is used to indicate the wearing region when the object wears the target clothing; and the skeleton key point data is extracted from the object skeleton data according to the wearing region data.
[0133] In some embodiments, the output module 706 is used to map the key points of the target clothing according to the position relationship to obtain target key point data of the target clothing; the target key point data is used to characterize the positions of the key points of the target clothing when the object wears the target clothing; the target clothing data is adjusted based on the target key point data to obtain deformable clothing data; the deformable clothing represented by the deformable clothing data matches the posture of the object; the deformable clothing data and the clothing changing object data are subjected to virtual clothing changing processing to obtain a clothing changing image output by the clothing changing model to be trained.
[0134] In some embodiments, the output module 706 is used to extract features from the target key point data and the target clothing data to obtain deformation feature data; and deform the target clothing image in the target clothing data according to the deformation feature data to obtain deformed clothing data.
[0135] In some embodiments, the output module 706 is configured to perform a fusion process on the deformable clothing data and the clothing-changing object data at a first resolution to obtain a clothing-changing result corresponding to the first resolution;
[0136] Starting from the next resolution of the first resolution, the current resolution is determined in sequence, and the deformable clothing data, the clothing changing object data, and the clothing changing result corresponding to the previous resolution of the current resolution are fused to obtain the clothing changing result corresponding to the current resolution;
[0137] The clothing-changing result corresponding to the last resolution is subjected to nonlinear mapping processing to obtain the clothing-changing image.
[0138] In some embodiments, the target loss includes a first loss; the first loss is determined based on the difference between the clothing-changing image and the original clothing-wearing image; the target loss also includes at least one of a second loss or a third loss; the second loss is determined based on the difference between the target clothing key point data and the original key point data; the third loss is determined based on the difference between the deformed clothing data and the original clothing data in the clothing-changing object data.
[0139] Each module in the virtual clothing-changing device 700 may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.
[0140] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be shown in Figure 8. The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store multiple sets of samples. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a virtual clothing changing method is implemented.
[0141] In some embodiments, a computer device is provided, which may be a terminal. Its internal structure diagram may be as shown in FIG9 . The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, where the wireless means may be implemented via Wi-Fi, a mobile cellular network, NFC (near-field communication), or other technologies. When executed by the processor, the computer program implements a virtual clothing-changing method. The display unit of the computer device is used to produce a visually visible image and may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.
[0142] Those skilled in the art will understand that the structure shown in Figure 8 or Figure 9 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.
[0143] In some embodiments, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.
[0144] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0145] In some embodiments, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0146] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.
[0147] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.
[0148] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0149] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.
Claims
1. A virtual clothing changing method, the method comprising: Identify multiple groups of samples; Each group of samples includes target clothing data and clothing-changing object data; The clothing-changing object data is extracted from an original dressing image showing the object wearing original clothing; the category of the original clothing is consistent with the category of the target clothing represented by the target clothing data; Learning the positional relationship between the key points of the object and the key points of the original clothing from the clothing-changing object data through the clothing-changing model to be trained; The key points of clothing in the same category have the same semantics; Based on the positional relationship, a virtual clothing-changing process is performed on the clothing-changing object data and the target clothing data to obtain a clothing-changing image output by the clothing-changing model to be trained; the clothing-changing image presents an effect of the object wearing the target clothing; as well as A target loss is determined according to a difference between the clothing-changing image and the original clothing-wearing image, and the clothing-changing model to be trained is optimized according to the target loss to obtain a trained clothing-changing model.
2. The method according to claim 1, wherein: The clothing-changing object data includes object skeleton data and original key point data; the original key point data is used to characterize the key points of the original clothing; The learning, from the clothing changing object data, the positional relationship between the key points of the object and the key points of the original clothing by the clothing changing model to be trained comprises: Extracting skeleton key point data from the object skeleton data based on a gating mechanism in a clothing-changing model to be trained; as well as By associating the skeleton key point data with the original clothing key point data, the positional relationship between the skeleton key points of the object and the key points of the original clothing is learned.
3. The method according to claim 2, wherein: The clothing-changing object data includes object texture data and region mask data; the region mask data refers to the mask of the region remaining after removing the region affecting the wearing of the target clothing; The extracting skeleton key point data from the object skeleton data based on the gating mechanism in the clothing changing model to be trained includes: Fusing the object texture data and the region mask data to obtain object fusion data; extracting wearing area data from the object fusion data according to the gated data in the clothing changing model to be trained; the wearing area data is used to indicate the wearing area of the object when wearing the target clothing; and Skeleton key point data is extracted from the object skeleton data according to the wearing area data.
4. The method according to claim 2, wherein: The performing virtual clothing-changing processing on the clothing-changing object data and the target clothing data based on the positional relationship to obtain the clothing-changing image output by the clothing-changing model to be trained includes: Mapping the key points of the target garment according to the positional relationship to obtain target key point data of the target garment; the target key point data is used to characterize the positions of the key points of the target garment when the subject wears the target garment; The target clothing data is adjusted based on the target key point data to obtain deformable clothing data; the deformable clothing represented by the deformable clothing data matches the posture of the object; and The deformable clothing data and the clothing-changing object data are subjected to virtual clothing-changing processing to obtain a clothing-changing image output by the clothing-changing model to be trained.
5. The method according to claim 4, wherein: The step of adjusting the target clothing data based on the target key point data to obtain the deformed clothing data includes: Extracting features from the target key point data and the target clothing data to obtain deformation feature data; and The target clothing image in the target clothing data is deformed according to the deformation feature data to obtain deformed clothing data.
6. The method according to claim 5, wherein: The target loss includes a first loss; the first loss is determined based on the difference between the clothing-changing image and the original clothing-wearing image; the target loss also includes at least one of a second loss or a third loss; the second loss is determined based on the difference between the target clothing key point data and the original key point data; the third loss is determined based on the difference between the deformed clothing data and the original clothing data in the clothing-changing object data.
7. The method according to claim 4, wherein: The performing virtual clothing-changing processing on the deformable clothing data and the clothing-changing object data to obtain the clothing-changing image output by the clothing-changing model to be trained comprises: Performing a fusion process of a first resolution on the deformable clothing data and the clothing-changing object data to obtain a clothing-changing result corresponding to the first resolution; Starting from the next resolution of the first resolution, sequentially determining the current resolution, fusing the deformable clothing data, the clothing-changing object data, and the clothing-changing result corresponding to the previous resolution of the current resolution, to obtain the clothing-changing result corresponding to the current resolution; and The clothing-changing result corresponding to the last resolution is subjected to nonlinear mapping processing to obtain a clothing-changing image.
8. A virtual fitting device, wherein: The device comprises: A determination module is used to determine multiple groups of samples; each group of samples includes target clothing data and clothing-changing object data; the clothing-changing object data is extracted from an original dressing image showing an object wearing original clothing; the original clothing is consistent with the category of the target clothing represented by the target clothing data; A learning module, used for learning the positional relationship between the key points of the object and the key points of the original clothing from the clothing-changing object data through the clothing-changing model to be trained; the key points of clothing in the same category have the same semantics; an output module, configured to perform virtual clothing-changing processing on the clothing-changing object data and the target clothing data based on the positional relationship, to obtain a clothing-changing image output by the clothing-changing model to be trained; the clothing-changing image presents an effect of the object wearing the target clothing; and The optimization module is used to determine the target loss according to the difference between the clothing-changing image and the original clothing-wearing image, and optimize the clothing-changing model to be trained according to the target loss to obtain a trained clothing-changing model.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
