Virtual trial assembly system, method, device and computer readable medium

Abstract

The present application relates to a virtual fitting system, method, device, and computer-readable medium. The system comprises: a deformation segmentation unit, configured to extract the human features of a target object using a preset key point connection topology as a limb connection constraint, and extract the clothing features of the target clothing. After fusing the human features and clothing features, a clothing segmentation image of the target clothing matching the appearance of the target object is generated based on the fused features; a clothing transformation unit, configured to transform the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing; and a fitting unit, configured to segment the target object according to the clothing segmentation image, use the pattern detail features of the target clothing extracted by a detail feature extraction model as information supplement to the clothing transformation image, fit the clothing transformation image to the segmented region, and obtain a fitting image of the target object wearing the target clothing. The present application solves the problem of inaccurate virtual fitting results due to cumulative errors.

Description

Technical Field

[0001] The present application relates to the field of multimedia intelligent processing technology, and in particular to a virtual trial installation system, method, device and computer-readable medium. Background Art

[0002] Virtual try-on involves generating a photo of a model wearing the target garment, given an image of a target garment and a model, to virtually try the garment on the model. The resulting try-on must meet four requirements: 1. Maintain the pose and shape of the person in the image; 2. Ensure the target garment fits the person's pose and shape; 3. Preserve all other information about the garment, such as undergarments; and 4. Ensure that creases, patterns, and logos remain clear and natural after the garment is worn. In e-commerce, effective virtual try-on technology not only provides consumers with a novel interactive experience but also guides and stimulates them to make faster purchasing decisions. In video platforms, virtual try-on technology can offer viewers a completely new viewing experience. For example, while watching a movie, viewers can upload their own photos and virtually wear the costumes of the characters in the show, creating an immersive viewing experience. Alternatively, viewers can switch between characters' outfits at will, increasing interactivity and fun, and potentially increasing user retention on the video platform.

[0003] At present, in related technologies, virtual fitting is usually performed through the ACGPN algorithm. The ACGPN algorithm consists of a total of four modules: module A, module B, module C, and module D. Module A is used to generate a human body instance segmentation map, parse the input human body image, and mark the limbs of the person in the image with different numbers, such as filling the left arm area with 0, the right arm area with 1, the body torso with 2, and so on; module B is used to generate a deformed clothing segmentation map, which is a binary image, with the deformed clothing area in the image filled with 1 and the rest of the positions filled with 0; module C consists of two parts: the spatial transformer network STN (Spatial Transformer Network) and the correction network, where STN outputs a rough transformed clothing image, and the correction network corrects the output of STN and outputs the final transformed clothing image; module D combines the outputs of the above modules to generate the final virtual fitting image. However, in deep learning tasks, the segmentation of human instances involves the prediction of many detailed areas such as hands, feet, hair, face, and torso. It is very difficult to obtain very precise prediction results. Therefore, the error in module A in the ACGPN algorithm further leads to the error in module B. As a result, the errors of modules A and B accumulate in modules C and D, resulting in inaccurate virtual try-on results in the final output.

[0004] Currently, no effective solution has been proposed to the problem of inaccurate virtual try-on results due to cumulative errors. Summary of the Invention

[0005] The present application provides a virtual fitting system, method, device and computer-readable medium to solve the technical problem of inaccurate virtual fitting results due to cumulative errors.

[0006] According to one aspect of an embodiment of the present application, the present application provides a virtual fitting system, comprising:

[0007] a deformation segmentation unit, configured to extract human features of a target object using a preset key point connection topology map as a limb connection constraint, extract clothing features of a target garment, fuse the human features with the clothing features, and generate a clothing segmentation image of the target garment that matches the appearance of the target object based on the fused features, wherein the preset key point connection topology map is used to constrain the object feature extraction model to recognize the target object's limbs when the target object's limbs are obscured;

[0008] a clothing transformation unit, configured to transform the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing;

[0009] The fitting unit is configured to segment the target object according to the clothing segmentation image, extract pattern detail features from the target clothing using a detail feature extraction model as information supplement to the clothing transformation image, fit the clothing transformation image to the segmented area, and obtain a fitting image of the target object in the target clothing.

[0010] Optionally, the deformation and segmentation unit includes:

[0011] a first feature extraction network, for extracting clothing features of the target clothing using an image of the target clothing;

[0012] a second feature extraction network for extracting body features of the target object using a preset segmentation map, a preset key point map, and a preset key point connection topology map of the target object, wherein the preset segmentation map is an image obtained by pre-identifying and segmenting each part of the target object one by one, and the preset key point map is used to mark key points of the body. The object feature extraction model includes the second feature extraction network;

[0013] The first feature decoding network is used to fuse the clothing features output by each convolutional layer of the first feature extraction network and the human body features output by each convolutional layer of the second feature extraction network, and upsample and decode the fused features throughout the fusion process to obtain a clothing segmentation image.

[0014] Optionally, the first feature extraction network includes multiple convolutional layers and multiple downsampling layers, which are connected in series and alternately arranged. The convolutional layers are used to extract the clothing features of the target clothing, and the downsampling layers are used to downsample and encode the clothing features output by the convolutional layers.

[0015] Optionally, the second feature extraction network includes multiple convolutional layers and multiple downsampling layers, which are connected in series and alternately arranged. The convolutional layers are used to extract the human body features of the target object, and the downsampling layers are used to downsample and encode the human body features output by the convolutional layers.

[0016] Optionally, the first feature decoding network includes multiple convolutional layers and multiple upsampling layers, the convolutional layers and the upsampling layers are connected in series and alternately arranged, and the upsampling layers are used to upsample and decode the fused features;

[0017] The input of the first target convolutional layer of the first feature decoding network includes: the clothing features output by the last downsampling layer in the first feature extraction network and the human body features output by the last downsampling layer in the second feature extraction network;

[0018] The input of the second target convolution layer of the first feature decoding network includes: the fused features output by the previous upsampling layer adjacent to the second target convolution layer, the clothing features output by the first convolution layer in the first feature extraction network, and the human body features output by the second convolution layer in the second feature extraction network;

[0019] The input of the third target convolution layer of the first feature decoding network includes: the fused features output by the previous upsampling layer adjacent to the third target convolution layer, the clothing features output by the third convolution layer in the first feature extraction network, and the human body features output by the fourth convolution layer in the second feature extraction network;

[0020] The input of the fourth target convolutional layer of the first feature decoding network includes: the fused features output by the previous upsampling layer adjacent to the fourth target convolutional layer, the clothing features output by the fifth convolutional layer in the first feature extraction network, and the human body features output by the sixth convolutional layer in the second feature extraction network;

[0021] Among them, the position order of the first target convolution layer to the fourth target convolution layer in the first feature decoding network is consistent with the ascending order, the position order of the first convolution layer, the third convolution layer and the fifth convolution layer in the first feature extraction network is consistent with the ascending order, and the position order of the second convolution layer, the fourth convolution layer and the sixth convolution layer in the second feature extraction network is consistent with the ascending order.

[0022] Optionally, the clothing changing unit includes:

[0023] a spatial transformation network, configured to take the target clothing image and the clothing segmentation image as input, transform the target clothing according to the clothing segmentation image, and obtain a first clothing transformation image;

[0024] The correction network is used to perform transformation correction on the clothing transformation image output by the spatial transformation network to obtain a final second clothing transformation image.

[0025] Optionally, the correction network includes:

[0026] The first feature encoding network includes multiple convolutional layers and multiple downsampling layers, where the convolutional layers and the downsampling layers are connected in series and alternately arranged;

[0027] The second feature decoding network is connected in series with the first feature encoding network, and includes multiple convolutional layers and multiple upsampling layers, where the convolutional layers and the upsampling layers are connected in series and alternately arranged;

[0028] The input of the first feature encoding network includes: the clothing transformation image and clothing segmentation image output by the spatial transformation network;

[0029] The input of the fifth target convolutional layer of the second feature decoding network includes: the intermediate transformed image output by the last downsampling layer of the first feature encoding network;

[0030] The input of the sixth target convolutional layer of the second feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the sixth target convolutional layer and the intermediate transformed image output by the seventh convolutional layer in the first feature encoding network;

[0031] The input of the seventh target convolutional layer of the second feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the seventh target convolutional layer and the intermediate transformed image output by the eighth convolutional layer in the first feature encoding network;

[0032] The input of the eighth target convolutional layer of the second feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the eighth target convolutional layer and the intermediate transformed image output by the ninth convolutional layer in the first feature encoding network;

[0033] The input of the ninth target convolutional layer of the second feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the ninth target convolutional layer and the intermediate transformed image output by the tenth convolutional layer in the first feature encoding network;

[0034] Among them, the position order of the fifth target convolution layer to the ninth target convolution layer in the second feature decoding network is consistent with the ascending order, and the position order of the seventh convolution layer to the tenth convolution layer in the first feature encoding network is opposite to the ascending order.

[0035] Optionally, the trial installation unit includes:

[0036] A branch network is used to extract pattern detail features in the target clothing through the first clothing transformation image, and the detail feature extraction model includes a branch network;

[0037] The backbone network is used to fit the second clothing transformation image to the segmented region and connect the pattern detail features during the fitting process to obtain a fitting image of the target object wearing the target clothing.

[0038] Optionally, the branch network includes:

[0039] The second feature encoding network includes multiple convolutional layers and multiple downsampling layers, where the convolutional layers and the downsampling layers are connected in series and alternately arranged;

[0040] The third feature decoding network is connected in series with the second feature encoding network, and includes multiple convolutional layers and multiple upsampling layers, where the convolutional layers and the upsampling layers are connected in series and alternately arranged;

[0041] The input of the second feature encoding network includes the first clothing transformation image.

[0042] Optionally, the backbone network includes:

[0043] The third feature encoding network includes multiple convolutional layers and multiple downsampling layers, where the convolutional layers and the downsampling layers are connected in series and alternately arranged;

[0044] a fourth feature decoding network, connected in series with the second feature encoding network, comprising a plurality of convolutional layers and a plurality of upsampling layers, wherein the convolutional layers and the upsampling layers are connected in series and alternately arranged;

[0045] The input of the third feature encoding network includes: the second clothing transformation image, the clothing segmentation image, and the target object's to-be-retained information, wherein the target object's to-be-retained information includes the target object's to-be-retained part obtained by segmenting the target object's image;

[0046] The input of the tenth target convolutional layer of the fourth feature decoding network includes: the intermediate encoded image output by the last downsampling layer of the third feature encoding network;

[0047] The input of the eleventh target convolutional layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the eleventh target convolutional layer, the intermediate encoded image output by the eleventh convolutional layer in the third feature encoding network, and the pattern detail features output by the twelfth convolutional layer in the third feature decoding network. The decoding scale of the eleventh target convolutional layer is the same as that of the twelfth convolutional layer.

[0048] The input of the twelfth target convolutional layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the twelfth target convolutional layer, the intermediate encoded image output by the thirteenth convolutional layer in the third feature encoding network, and the pattern detail features output by the fourteenth convolutional layer in the third feature decoding network. The decoding scale of the twelfth target convolutional layer is the same as that of the fourteenth convolutional layer.

[0049] The input of the thirteenth target convolutional layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the thirteenth target convolutional layer, the intermediate encoded image output by the fifteenth convolutional layer in the third feature encoding network, and the pattern detail features output by the sixteenth convolutional layer in the third feature decoding network. The decoding scale of the thirteenth target convolutional layer is the same as that of the sixteenth convolutional layer.

[0050] The input of the fourteenth target convolutional layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the fourteenth target convolutional layer, the intermediate encoded image output by the seventeenth convolutional layer in the third feature encoding network, and the pattern detail features output by the eighteenth convolutional layer in the third feature decoding network. The decoding scale of the fourteenth target convolutional layer is the same as that of the eighteenth convolutional layer.

[0051] Among them, the position order of the tenth target convolution layer to the fourteenth target convolution layer in the fourth feature decoding network is consistent with the ascending order, the position order of the eleventh convolution layer, the thirteenth convolution layer, the fifteenth convolution layer and the seventeenth convolution layer in the third feature encoding network is opposite to the ascending order, and the position order of the twelfth convolution layer, the fourteenth convolution layer, the sixteenth convolution layer and the eighteenth convolution layer in the third feature decoding network is consistent with the ascending order.

[0052] According to another aspect of the embodiments of the present application, the present application provides a virtual fitting method, comprising:

[0053] Extracting the human features of the target object using a preset key point connection topology as a limb connection constraint, extracting the clothing features of the target clothing, and fusing the human features and clothing features to generate a clothing segmentation image of the target clothing that matches the appearance of the target object based on the fused features, wherein the preset key point connection topology is used to constrain the feature extraction model to recognize the target object's limbs when the target object's limbs are occluded;

[0054] Transform the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing;

[0055] After the target object is segmented according to the clothing segmentation image, the pattern detail features of the target clothing obtained by the detail feature extraction model are used as information supplement of the clothing transformation image. The clothing transformation image is fitted to the segmented area to obtain the fitting image of the target object in the target clothing.

[0056] Optionally, extracting the human features of the target object using a preset key point connection topology as a limb connection constraint, extracting clothing features of the target clothing, and fusing the human features with the clothing features to generate a clothing segmentation image of the target clothing that matches the appearance of the target object based on the fused features includes:

[0057] Extracting clothing features of the target clothing using an image of the target clothing;

[0058] The human body features of the target object are extracted using a preset segmentation map, a preset key point map, and a preset key point connection topology map of the target object, wherein the preset segmentation map is an image obtained by pre-identifying and segmenting each part of the target object one by one, and the preset key point map is used to mark the key points of the human body;

[0059] The clothing features and human body features are fused, and the fused features are upsampled and decoded throughout the fusion process to obtain the clothing segmentation image.

[0060] Optionally, transforming the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing includes:

[0061] Obtain an image of the target's clothing;

[0062] transforming the target clothing image according to the clothing segmentation image to obtain a first clothing transformation image;

[0063] Perform transformation correction on the first clothing transformation image to obtain a final second clothing transformation image.

[0064] Optionally, using pattern detail features of the target clothing extracted by the detail feature extraction model as information supplement of the clothing transformation image, fitting the clothing transformation image to the segmented region, and obtaining a fitting image of the target object wearing the target clothing includes:

[0065] Inputting the first clothing transformation image into a detail feature extraction model to obtain pattern detail features of the target clothing;

[0066] The second clothing transformation image is decoded based on the clothing segmentation image and the target object's information to be retained. During the decoding process, pattern detail features are connected to fit the second clothing transformation image, which incorporates the pattern detail features, to the segmented region, thereby obtaining a fitting image of the target object wearing the target clothing. The information to be retained of the target object includes the target object's part to be retained obtained by segmenting the target object's image.

[0067] Optionally, before inputting the first clothing transformation image into the detail feature extraction model, the method further includes training the detail feature extraction model in the following manner:

[0068] Obtaining a generated image and a ground-truth image corresponding to the generated image obtained by performing feature extraction on the training image using a detail feature extraction model, wherein both the generated image and the ground-truth image include images at multiple decoding scales, and the generated image and the corresponding ground-truth image at the same decoding scale are grouped together;

[0069] The center of the generated image in the same group is used as the center of the cropping frame, and the generated image and the true value image are cropped to obtain a first generated image and a first true value image;

[0070] Randomly determining the centers of multiple cropping frames on the first generated image and the first ground-truth image in the same group, and cropping the first generated image and the first ground-truth image according to the centers of the multiple cropping frames to obtain multiple second generated images and multiple second ground-truth images, wherein the centers of the multiple cropping frames on the first generated image and the first ground-truth image are the same, and the second generated images with the same cropping frames correspond to the second ground-truth image, and a group of images with a larger decoding scale has more cropping frames;

[0071] Determine the L1 loss value between each second generated image and the corresponding second true image using the target loss function, and take the sum of all L1 loss values ​​as the final loss value of the current decoding scale;

[0072] The final loss value is used to adjust the parameters of the detail feature extraction model to optimize the decoding accuracy of the detail feature extraction model for the current decoding scale.

[0073] According to another aspect of an embodiment of the present application, the present application provides an electronic device, including a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, the memory and the processor communicate through the communication bus and the communication interface, and the steps of the above method are implemented when the processor executes the computer program.

[0074] According to another aspect of an embodiment of the present application, the present application further provides a computer-readable medium having a non-volatile program code executable by a processor, where the program code enables the processor to execute the above method.

[0075] The above technical solution provided by the embodiment of the present application has the following advantages compared with the related art:

[0076] The technical solution of the present application provides a virtual fitting system, comprising: a deformation segmentation unit, for extracting the human body features of a target object using a preset key point connection topology map as a limb connection constraint, and extracting the dressing features of the target clothing, and after fusing the human body features and the clothing features, generating a clothing segmentation image of the target clothing that matches the appearance of the target object based on the fused features, wherein the preset key point connection topology map is used to constrain the object feature extraction model to identify the limbs of the target object when the limbs of the target object are occluded; a clothing transformation unit, for transforming the target clothing according to the clothing segmentation image, to obtain a clothing transformation image of the target clothing; and a fitting unit, for segmenting the target object according to the clothing segmentation image, using the pattern detail features of the target clothing extracted by the detail feature extraction model as information supplement of the clothing transformation image, fitting the clothing transformation image to the segmented area, and obtaining a fitting image of the target object in the target clothing. The present application integrates the human features of the target object and the clothing features of the target clothing, and directly predicts and outputs a deformed clothing segmentation image that matches the appearance of the target object. This reduces the accumulated prediction error in the related art of first predicting the human instance segmentation image of the target object and then predicting the deformed clothing image segmentation map based on the human instance segmentation image. This further improves the accuracy of the fitting image, making the fitting effect more fitting and natural, and solving the technical problem of inaccurate virtual fitting results due to accumulated errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0079] Figure 1a Schematic diagram of a human body instance segmentation map predicted and output by module A of the ACGPN algorithm in the related art;

[0080] Figure 1b Schematic diagram of the segmentation map of the deformed clothing image predicted and output by the B module of the ACGPN algorithm in the related art;

[0081] Figure 1c Schematic diagram of the deformed clothing image predicted and output by the C module of the ACGPN algorithm in the related art;

[0082] Figure 1d Schematic diagram of the test results of the predicted output of the D module of the ACGPN algorithm in the related art;

[0083] Figure 2 Schematic diagram of an optional virtual trial assembly system structure provided according to an embodiment of the present application;

[0084] Figure 3 Schematic diagram of a feature extraction network structure of an optional deformation segmentation unit provided according to an embodiment of the present application;

[0085] Figure 4 Schematic diagram of a feature decoding network structure of an optional deformation segmentation unit provided according to an embodiment of the present application;

[0086] Figure 5 This is a schematic diagram of a correction network structure of an optional clothing transformation unit provided according to an embodiment of the present application;

[0087] Figure 6 A schematic diagram of a network structure of an optional trial installation unit provided according to an embodiment of the present application;

[0088] Figure 7 A schematic flow chart of an optional virtual trial assembly method provided according to an embodiment of the present application;

[0089] Figure 8 This is a schematic diagram of an optional image center cropping method provided according to an embodiment of the present application;

[0090] Figure 9 A schematic diagram of an optional multi-scale random cropping of an image provided according to an embodiment of the present application;

[0091] Figure 10 A schematic diagram of an optional electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0092] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0093] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of this application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0094] In related technologies, virtual fitting is usually performed using the ACGPN algorithm. The ACGPN algorithm consists of four modules: module A, module B, module C, and module D. Module A is used to generate a human instance segmentation map, analyze the input human image, and mark the limbs of the person in the image with different numbers, such as filling the left arm area with 0, the right arm area with 1, the body torso with 2, etc. The prediction results of the human instance segmentation map are as follows: Figure 1a As shown in Figure 2; Module B is used to generate the segmentation map of the deformed clothes. The segmentation map is a binary image. The deformed clothes area in the image is filled with 1, and the rest of the positions are filled with 0. The prediction result of the deformed clothes segmentation map is as follows: Figure 1b As shown in Figure 2, the C module consists of two parts: the spatial transformer network STN (SpatialTransformer Network) and the correction network. The STN outputs a rough transformed clothing image, and the correction network corrects the output of the STN and outputs the final transformed clothing image. The prediction result of the transformed clothing image is shown in Figure 2. Figure 1c As shown; Module D integrates the outputs of the above modules to generate the final virtual try-on image, the effect is as follows Figure 1d However, in deep learning tasks, the segmentation of human instances involves the prediction of numerous detailed areas, such as hands, feet, hair, face, and torso. Obtaining highly refined predictions is extremely challenging. Consequently, errors in module A in the ACGPN algorithm further lead to errors in module B. Consequently, the errors in modules A and B accumulate in modules C and D, resulting in inaccurate virtual try-on results.

[0095] In order to solve the technical problems mentioned in the background technology, according to one aspect of the embodiment of the present application, the present application provides a virtual fitting system, such as Figure 2 As shown, including:

[0096] The deformation segmentation unit 201 is configured to extract the human features of the target object using a preset key point connection topology map as a limb connection constraint, extract the clothing features of the target clothing, fuse the human features with the clothing features, and generate a clothing segmentation image of the target clothing that matches the target object's appearance based on the fused features. The preset key point connection topology map is used to constrain the object feature extraction model to recognize the target object's limbs when the target object's limbs are occluded.

[0097] a clothing transformation unit 203 for transforming the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing;

[0098] The fitting unit 205 is configured to segment the target object according to the clothing segmentation image, extract pattern detail features of the target clothing using the detail feature extraction model as information supplement of the clothing transformation image, fit the clothing transformation image to the segmented region, and obtain a fitting image of the target object in the target clothing.

[0099] Based on the deformation segmentation unit 201, the human body features of the target object and the clothing features of the target clothing are extracted and fused, so that the neural network model predicts and outputs the clothing segmentation image of the target clothing that matches the appearance of the target object at one time, that is, the A module and the B module of the ACGPN algorithm in the related technology are merged to directly predict and output the deformed clothing image segmentation map, skipping the prediction of the human body instance segmentation map, and instead fusing the human body features and clothing features to predict the clothing segmentation image. This not only greatly simplifies the original scheme, reduces the number of modules of the original scheme, and enhances the integrity of the scheme, but also reduces the accumulation of prediction errors between multiple modules, further improving the image accuracy in subsequent steps.

[0100] Moreover, the technical solution of this application adds a connection topology relationship diagram between key points as a limb connection constraint when extracting human body features, so that even if the hands are crossed in front of the chest and blocked, the model can effectively identify the arm information of the left and right hands and distinguish them from the background clothes.

[0101] Furthermore, the technical solution of the present application also adds additional information branches to the fitting unit as a supplement to the information of the lost pattern details such as clothing patterns and logos, so that the model can effectively learn the lost detail information from the additional branches, thereby effectively improving the presentation effect of the final fitting image.

[0102] The technical solution of this application can be applied to the e-commerce sector, where the target object can be an e-commerce merchant's model, and the target outfit can be the merchant's clothing for sale. Based on this virtual fitting system, the merchant can batch-generate fitting images of their own clothing for sale simply by using the model's modeling photos. The target object can also be a consumer, who can generate fitting images of themselves trying on the clothing they are about to purchase by uploading their own photos. Therefore, in the e-commerce sector, good virtual fitting technology can not only provide consumers with a novel interactive experience, but also guide and stimulate consumers to make purchasing decisions more quickly.

[0103] The technical solution of this application can also be applied to the field of video platforms, where the target clothing can be the clothing of virtual characters in film and television works, game works, and animation works, and the target object can be the audience. The audience can upload their own photos and try on the clothes of the characters in the play. Therefore, in the field of video platforms, virtual fitting technology can bring a new immersive viewing experience to viewers and increase the interaction between fans and characters in the play. Not only that, the target object can also be a character in the play, and the target clothing can be a specific clothing selected by the audience, so that the audience can switch the clothes of the characters in the play at will, increasing the interactivity and fun of watching movies, and also making users active and retaining longer on the video platform.

[0104] Optionally, the deformation and segmentation unit includes:

[0105] a first feature extraction network, for extracting clothing features of the target clothing using an image of the target clothing;

[0106] a second feature extraction network for extracting body features of the target object using a preset segmentation map, a preset key point map, and a preset key point connection topology map of the target object, wherein the preset segmentation map is an image obtained by pre-identifying and segmenting each part of the target object one by one, and the preset key point map is used to mark key points of the body. The object feature extraction model includes the second feature extraction network;

[0107] The first feature decoding network is used to fuse the clothing features output by each convolutional layer of the first feature extraction network and the human body features output by each convolutional layer of the second feature extraction network, and upsample and decode the fused features throughout the fusion process to obtain a clothing segmentation image.

[0108] In the embodiment of the present application, the first feature extraction network takes the image of the target clothing as input and extracts the clothing features of the target clothing layer by layer through multiple convolutional layers.

[0109] Alternatively, as Figure 3 As shown in branch network 1 in , the first feature extraction network includes multiple convolutional layers and multiple downsampling layers. The convolutional layers and the downsampling layers are connected in series and alternately. The convolutional layers are used to extract the clothing features of the target clothing, and the downsampling layers are used to downsample and encode the clothing features output by the convolutional layers.

[0110] In this embodiment of the present application, the second feature extraction network takes a preset segmentation map and a preset keypoint map of the target object as input and extracts the human features of the target object layer by layer through multiple convolutional layers. The preset segmentation map of the target object can be obtained by processing a photograph of the target object using a human body parsing algorithm, such as RefineNet, which separately segments the left hand, right hand, head, hair, torso, left leg, and right leg of the human body. The preset keypoint map is used to mark key points of the human body.

[0111] Alternatively, as Figure 3 As shown in branch network 2 in , the second feature extraction network includes multiple convolutional layers and multiple downsampling layers. The convolutional layers and the downsampling layers are connected in series and alternately. The convolutional layers are used to extract the human features of the target object, and the downsampling layers are used to downsample and encode the human features output by the convolutional layers.

[0112] In the embodiment of this application, Figure 3 The “+” in the figure not only represents the fusion of the clothing features output by the last downsampling layer of the first feature extraction network and the human body features output by the last downsampling layer of the second feature extraction network, but also represents the feature fusion of the outputs of each convolutional layer in the first and second feature extraction networks.

[0113] Alternatively, as Figure 4 As shown, the first feature decoding network includes multiple convolutional layers and multiple upsampling layers, the convolutional layers and the upsampling layers are connected in series and alternately arranged, and the upsampling layers are used to upsample and decode the fusion features;

[0114] The input of the first target convolutional layer of the first feature decoding network includes: the clothing features output by the last downsampling layer in the first feature extraction network and the human body features output by the last downsampling layer in the second feature extraction network;

[0115] The input of the second target convolution layer of the first feature decoding network includes: the fused features output by the previous upsampling layer adjacent to the second target convolution layer, the clothing features output by the first convolution layer in the first feature extraction network, and the human body features output by the second convolution layer in the second feature extraction network;

[0116] The input of the third target convolution layer of the first feature decoding network includes: the fused features output by the previous upsampling layer adjacent to the third target convolution layer, the clothing features output by the third convolution layer in the first feature extraction network, and the human body features output by the fourth convolution layer in the second feature extraction network;

[0117] The input of the fourth target convolutional layer of the first feature decoding network includes: the fused features output by the previous upsampling layer adjacent to the fourth target convolutional layer, the clothing features output by the fifth convolutional layer in the first feature extraction network, and the human body features output by the sixth convolutional layer in the second feature extraction network;

[0118] Among them, the position order of the first target convolution layer to the fourth target convolution layer in the first feature decoding network is consistent with the ascending order, the position order of the first convolution layer, the third convolution layer and the fifth convolution layer in the first feature extraction network is consistent with the ascending order, and the position order of the second convolution layer, the fourth convolution layer and the sixth convolution layer in the second feature extraction network is consistent with the ascending order.

[0119] In the embodiment of this application, Figure 3 、 4 For example, the first target convolution layer to the fourth target convolution layer are convolution 6 to convolution 9 in the first feature decoding network, and their position order is consistent with the increasing order; the first convolution layer, the third convolution layer and the fifth convolution layer are Figure 3 The position order of convolution 2, convolution 3 and convolution 4 in branch network 1 (the first feature extraction network) is consistent with the increasing order; the second convolution layer, the fourth convolution layer and the sixth convolution layer are respectively Figure 3 The position order of convolution 2, convolution 3, and convolution 4 in branch network 2 (the second feature extraction network) is consistent with the increasing order.

[0120] The aforementioned network structure of the deformable segmentation unit fuses the clothing features and body features extracted at each stage of the feature extraction network, avoiding the loss of low-dimensional features after extracting high-dimensional features. This allows for a more comprehensive and accurate prediction of a clothing segmentation image of the target clothing that matches the target object's appearance. This clothing segmentation image represents the outline of the target object wearing the target clothing.

[0121] Optionally, the clothing changing unit includes:

[0122] A spatial transformation network is used to take the target clothing image and the clothing segmentation image as input, transform the target clothing according to the clothing segmentation image, and obtain the clothing transformation image;

[0123] The correction network is used to perform transformation correction on the clothing transformation image output by the spatial transformation network to obtain the final clothing transformation image.

[0124] In an embodiment of the present application, the Spatial Transformer Network (STN) is composed of three parts: a localization network, a grid generator, and a sampler. For the input image, the spatial transformer network first uses a localization network to predict the required transformation, that is, after several consecutive layers of calculations (including convolution and fully connected calculations), the grid generator and the sampler then transform the image. The spatial transformer network can adaptively perform spatial transformation and alignment on the data. In an embodiment of the present application, the spatial transformer network takes the image of the target dress and the dress segmentation image output by the above-mentioned deformation segmentation unit as input, and obtains the dress transformation image through adaptive transformation and alignment.

[0125] In the embodiment of the present application, the correction network performs transformation correction on the clothing transformation image output by the spatial transformation network to obtain the final clothing transformation image. Figure 5As shown, the correction network includes:

[0126] The first feature encoding network includes multiple convolutional layers and multiple downsampling layers, where the convolutional layers and the downsampling layers are connected in series and alternately arranged;

[0127] The second feature decoding network is connected in series with the first feature encoding network, and includes multiple convolutional layers and multiple upsampling layers, where the convolutional layers and the upsampling layers are connected in series and alternately arranged;

[0128] The input of the first feature encoding network includes: the clothing transformation image and clothing segmentation image output by the spatial transformation network;

[0129] The input of the fifth target convolutional layer of the second feature decoding network includes: the intermediate transformed image output by the last downsampling layer of the first feature encoding network;

[0130] The input of the sixth target convolutional layer of the second feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the sixth target convolutional layer and the intermediate transformed image output by the seventh convolutional layer in the first feature encoding network;

[0131] The input of the seventh target convolutional layer of the second feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the seventh target convolutional layer and the intermediate transformed image output by the eighth convolutional layer in the first feature encoding network;

[0132] The input of the eighth target convolutional layer of the second feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the eighth target convolutional layer and the intermediate transformed image output by the ninth convolutional layer in the first feature encoding network;

[0133] The input of the ninth target convolutional layer of the second feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the ninth target convolutional layer and the intermediate transformed image output by the tenth convolutional layer in the first feature encoding network;

[0134] Among them, the position order of the fifth target convolution layer to the ninth target convolution layer in the second feature decoding network is consistent with the ascending order, and the position order of the seventh convolution layer to the tenth convolution layer in the first feature encoding network is opposite to the ascending order.

[0135] In the embodiment of this application, Figure 5For example, the downsampling layer after convolution 1 to convolution 4 in the correction network is the first feature encoding network, and convolution 5 to convolution 9 is the second feature decoding network. The fifth to ninth target convolution layers correspond to convolution 5 to convolution 9 in the second feature decoding network, respectively, and their position order is consistent with the increasing order; the seventh to tenth convolution layers correspond to convolution 4, convolution 3, convolution 2, and convolution 1 in the first feature encoding network, respectively, and their position order is opposite to the increasing order.

[0136] Through the above-mentioned network structure setting of the clothing transformation unit, the correction network is used to further correct the clothing transformation image output by the spatial transformation network. In the correction network, the first encoded intermediate transformation image is input into the later convolutional layer for fusion. That is, the correction process is based on the features of each dimension, and ultimately a more accurate clothing transformation image is obtained.

[0137] Alternatively, as Figure 6 As shown, the test unit includes:

[0138] A branch network is used to extract pattern detail features in the target clothing through the first clothing transformation image, and the detail feature extraction model includes a branch network;

[0139] The backbone network is used to fit the second clothing transformation image to the segmented region and connect the pattern detail features during the fitting process to obtain a fitting image of the target object wearing the target clothing.

[0140] Optionally, the branch network includes:

[0141] The second feature encoding network includes multiple convolutional layers and multiple downsampling layers, where the convolutional layers and the downsampling layers are connected in series and alternately arranged;

[0142] The third feature decoding network is connected in series with the second feature encoding network, and includes multiple convolutional layers and multiple upsampling layers, where the convolutional layers and the upsampling layers are connected in series and alternately arranged;

[0143] The input of the second feature encoding network includes the first clothing transformation image.

[0144] Optionally, the backbone network includes:

[0145] The third feature encoding network includes multiple convolutional layers and multiple downsampling layers, where the convolutional layers and the downsampling layers are connected in series and alternately arranged;

[0146] a fourth feature decoding network, connected in series with the second feature encoding network, comprising a plurality of convolutional layers and a plurality of upsampling layers, wherein the convolutional layers and the upsampling layers are connected in series and alternately arranged;

[0147] The input of the third feature encoding network includes: the second clothing transformation image, the clothing segmentation image, and the target object's to-be-retained information, wherein the target object's to-be-retained information includes the target object's to-be-retained part obtained by segmenting the target object's image;

[0148] The input of the tenth target convolutional layer of the fourth feature decoding network includes: the intermediate encoded image output by the last downsampling layer of the third feature encoding network;

[0149] The input of the eleventh target convolutional layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the eleventh target convolutional layer, the intermediate encoded image output by the eleventh convolutional layer in the third feature encoding network, and the pattern detail features output by the twelfth convolutional layer in the third feature decoding network. The decoding scale of the eleventh target convolutional layer is the same as that of the twelfth convolutional layer.

[0150] The input of the twelfth target convolutional layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the twelfth target convolutional layer, the intermediate encoded image output by the thirteenth convolutional layer in the third feature encoding network, and the pattern detail features output by the fourteenth convolutional layer in the third feature decoding network. The decoding scale of the twelfth target convolutional layer is the same as that of the fourteenth convolutional layer.

[0151] The input of the thirteenth target convolutional layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the thirteenth target convolutional layer, the intermediate encoded image output by the fifteenth convolutional layer in the third feature encoding network, and the pattern detail features output by the sixteenth convolutional layer in the third feature decoding network. The decoding scale of the thirteenth target convolutional layer is the same as that of the sixteenth convolutional layer.

[0152] The input of the fourteenth target convolutional layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the fourteenth target convolutional layer, the intermediate encoded image output by the seventeenth convolutional layer in the third feature encoding network, and the pattern detail features output by the eighteenth convolutional layer in the third feature decoding network. The decoding scale of the fourteenth target convolutional layer is the same as that of the eighteenth convolutional layer.

[0153] Among them, the position order of the tenth target convolution layer to the fourteenth target convolution layer in the fourth feature decoding network is consistent with the ascending order, the position order of the eleventh convolution layer, the thirteenth convolution layer, the fifteenth convolution layer and the seventeenth convolution layer in the third feature encoding network is opposite to the ascending order, and the position order of the twelfth convolution layer, the fourteenth convolution layer, the sixteenth convolution layer and the eighteenth convolution layer in the third feature decoding network is consistent with the ascending order.

[0154] In the embodiment of this application, Figure 6For example, the test unit includes a branch network and a trunk network, and the branch network includes a second feature coding network ( Figure 6 The downsampling layer after convolution 1 to convolution 4 in the branch network) and the third feature decoding network ( Figure 6 Convolution 5 to convolution 9 in the branch network), the backbone network includes the third feature encoding network ( Figure 6 The downsampling layer after convolution 1 to convolution 4 in the backbone network) and the fourth feature decoding network ( Figure 6 Convolution 5 to Convolution 9 in the backbone network). The tenth to fourteenth target convolution layers correspond to Convolution 5 to Convolution 9 in the fourth feature decoding network, respectively, and their position order is consistent with the ascending order; the eleventh, thirteenth, fifteenth, and seventeenth convolution layers correspond to Convolution 4, Convolution 3, Convolution 2, and Convolution 1 in the third feature encoding network, respectively, and their position order is opposite to the ascending order; the twelfth, fourteenth, sixteenth, and eighteenth convolution layers correspond to Convolution 6, Convolution 7, Convolution 8, and Convolution 9 in the third feature decoding network, respectively, and their position order is consistent with the ascending order.

[0155] In an embodiment of the present application, the fourth feature decoding network of the backbone network will be connected to the shallow feature maps in the third feature encoding network in sequence according to the size of the feature maps. In addition, the high-level features (i.e., pattern detail features) of the corresponding size of the branch network will be additionally connected, so that the decoding part of the backbone network can contain a large amount of clear patterns and logos of the target clothes and other detailed information, helping the backbone network to recover the detailed information of the image.

[0156] In an embodiment of the present application, the third feature extraction network of the branch network does not need to be connected to the shallow features in the second feature encoding network. Although the shallow features contain a large amount of clothing detail information, the input of the second feature encoding network is the first clothing transformation image output by the above-mentioned spatial transformation network. At this time, there is still a large gap between the first clothing transformation image and the true value image of the target clothing after transformation. Excessively fine position information will interfere with the output of the third feature decoding network. Moreover, removing the feature connection between the third feature decoding network and the second feature encoding network in the branch network can also significantly reduce the computational complexity of the third feature decoding network.

[0157] In this embodiment of the present application, the target object's to-be-retained information includes the target object's to-be-retained parts obtained by segmenting the target object's image. For example, if the target object is wearing a short-sleeved shirt, the to-be-retained parts include the head, left and right arms, left and right legs, and left and right feet. If the target object is wearing pants, the to-be-retained parts include the head, left and right arms, chest, waist, and left and right feet.

[0158] Through the above-mentioned network structure setting of the fitting unit, the intermediate encoded image is input into the later convolutional layer for fusion, that is, the features of each dimension are fully integrated, and then additional pattern detail features are connected to finally obtain the fitting image of the target object trying on the target clothing.

[0159] Based on the same technical concept, according to one aspect of the embodiment of the present application, an embodiment of a virtual fitting method is provided, such as Figure 7 As shown, the method may include the following steps:

[0160] Step S702: extracting the human features of the target object using a preset key point connection topology as a limb connection constraint, extracting clothing features of the target clothing, fusing the human features and clothing features, and generating a clothing segmentation image of the target clothing that matches the target object's appearance based on the fused features. The preset key point connection topology is used to constrain the feature extraction model to recognize the target object's limbs when the target object's limbs are occluded.

[0161] Step S704, transforming the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing;

[0162] Step S706 , after segmenting the target object according to the clothing segmentation image, the pattern detail features of the target clothing extracted by the detail feature extraction model are used as information supplement of the clothing transformation image, and the clothing transformation image is fitted to the segmented region to obtain a fitting image of the target object in the target clothing.

[0163] Through steps S702 to S706, the present application integrates the human features of the target object and the clothing features of the target clothing, and directly predicts and outputs a deformed clothing segmentation image that matches the appearance of the target object, thereby reducing the accumulated prediction error of the related art of first predicting the model's human body instance segmentation image and then predicting the deformed clothing image segmentation map based on the human body instance segmentation image, further improving the accuracy of the fitting image, making the fitting effect more fitting and natural, and solving the technical problem of inaccurate virtual try-on results due to accumulated errors.

[0164] Moreover, the technical solution of this application adds a connection topology relationship diagram between key points as a limb connection constraint when extracting human body features, so that even if the hands are crossed in front of the chest and blocked, the model can effectively identify the arm information of the left and right hands and distinguish them from the background clothes.

[0165] Furthermore, the technical solution of the present application also adds additional information branches to the fitting unit as a supplement to the information of the lost pattern details such as clothing patterns and logos, so that the model can effectively learn the lost detail information from the additional branches, thereby effectively improving the presentation effect of the final fitting image.

[0166] In step S702, the human body features of the target object are extracted using the preset key point connection topology as the limb connection constraints, the clothing features of the target clothing are extracted, and the human body features and clothing features are fused to generate a clothing segmentation image of the target clothing that matches the appearance of the target object based on the fused features. The process includes:

[0167] Step 1, extracting clothing features of the target clothing using an image of the target clothing;

[0168] Step 2: extracting the human features of the target object using a preset segmentation map, a preset key point map, and a preset key point connection topology map of the target object, wherein the preset segmentation map is an image obtained by separately identifying and segmenting each part of the target object in advance, and the preset key point map is used to mark the key points of the human body;

[0169] Step 3: fuse the clothing features and human body features, and upsample and decode the fused features throughout the fusion process to obtain the clothing segmentation image.

[0170] In step S704, the target clothing is transformed according to the clothing segmentation image, and the clothing transformation image of the target clothing is obtained, including:

[0171] Step 1, obtain the image of the target clothing;

[0172] Step 2: transform the target clothing image according to the clothing segmentation image to obtain a clothing transformation image;

[0173] Step 3: Perform transformation correction on the clothing transformation image to obtain the final clothing transformation image.

[0174] In step S706, the pattern detail features of the target clothing extracted by the detail feature extraction model are used as information supplement of the clothing transformation image, and the clothing transformation image is fitted to the segmented region to obtain the fitting image of the target object wearing the target clothing, including:

[0175] Step 1: Input the first clothing transformation image into a detail feature extraction model to obtain pattern detail features of the target clothing;

[0176] Step 2: Decode the second clothing transformation image based on the clothing segmentation image and the target object's information to be retained. During the decoding process, connect the pattern detail features to fit the second clothing transformation image that incorporates the pattern detail features to the segmented area, thereby obtaining an image of the target object fitting the target clothing. The information to be retained of the target object includes the target object's part to be retained obtained by segmenting the target object's image.

[0177] Optionally, before inputting the first clothing transformation image into the detail feature extraction model, the method further includes training the detail feature extraction model in the following manner:

[0178] Step 1: Obtain a generated image and a ground-truth image corresponding to the generated image obtained by performing feature extraction on the training image using a detail feature extraction model, wherein both the generated image and the ground-truth image include images at multiple decoding scales, and the generated image and the corresponding ground-truth image at the same decoding scale are grouped together;

[0179] Step 2: Using the center of the generated image in the same group as the center of the cropping frame, crop the generated image and the true image to obtain a first generated image and a first true image;

[0180] Step 3: randomly determine the centers of multiple cropping frames on the first generated image and the first ground-truth image in the same group, and crop the first generated image and the first ground-truth image according to the centers of the multiple cropping frames to obtain multiple second generated images and multiple second ground-truth images, wherein the centers of the multiple cropping frames on the first generated image and the first ground-truth image are the same, and the second generated images with the same cropping frames correspond to the second ground-truth image, and a group of images with a larger decoding scale has more cropping frames;

[0181] Step 4: Use the target loss function to determine the L1 loss value between each second generated image and the corresponding second true image, and take the sum of all L1 loss values ​​as the final loss value of the current decoding scale;

[0182] Step 5: Use the final loss value to adjust the parameters of the detail feature extraction model to optimize the decoding accuracy of the detail feature extraction model for the current decoding scale.

[0183] In the embodiment of this application, the target loss function is the L1 PatchWise loss function. This loss function automatically filters out clothing in images during training and removes irrelevant information. It also uses multi-scale random cropping to automatically assign higher training weights to clothing, helping the model automatically discover details such as clothing patterns and logos during training and effectively recover them during the generation process. Multi-scale cropping is described below.

[0184] First do center cropping, such as Figure 8 As shown in , the center of the image to be cropped is used as the center of the cropping frame, and the true image and the generated image are cropped at the same time to remove irrelevant information in the image and filter out the clothing part in the image. Then multi-scale random cropping is performed, as shown in Figure 9As shown in the figure, the result image of the previous step of central cropping is used as the image to be cropped. A cropping box is randomly generated within the full image range, and it is ensured that the positions of the cropping boxes in the ground truth image and the image to be cropped are the same. The cropping box contains a total of n scales, forming a cropping box scale set S = {S1, S2, …, Sn}, and S1 > S2 > … > Sn is satisfied. The number of randomly cropped blocks corresponding to the S1 scale is M1, the number of randomly cropped blocks corresponding to the S2 scale is M2, and so on, forming a multi-scale randomly cropped block set M = {M1, M2, …, Mn}, and M1 < M2 < … < Mn is satisfied, that is, the larger the scale of the cropping box, the fewer the number of random cropping times. Finally, the L1 loss is calculated for each image block obtained in the previous step, and the average value is calculated by integrating all the L1 losses to obtain the final loss value, which is used for training optimization.

[0185] The target loss function can be specifically expressed as:

[0186]

[0187]

[0188] In the first formula, i represents the i-th scale cropping box, with a total of n, and j represents the j-th cropped block obtained under the i-th scale cropping box, with a total of M i cropped blocks. p generate represents the cropped block in the generated image, and p groundtruth represents the corresponding cropped block in the ground truth image. The cropping positions of the two are exactly the same. L1(p generate , p groundtruth ) represents calculating the per-pixel mean error between the two cropped blocks.

[0189] In the second formula, i represents the i-th element in the image block, with a total of N elements, and p1 and p2 have the same size and shape. L1(p1, p2) represents calculating the absolute error between the two image blocks for each element, summing them, and finally calculating the average value.

[0190] According to another aspect of the embodiments of the present application, the present application provides an electronic device, as Figure 10 shown, including a memory 1001, a processor 1003, a communication interface 1005, and a communication bus 1007. A computer program that can run on the processor 1003 is stored in the memory 1001. The memory 1001 and the processor 1003 communicate through the communication interface 1005 and the communication bus 1007. When the processor 1003 executes the computer program, the steps of the above method are implemented.

[0191] The memory and processor in the electronic device communicate via a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus may be divided into an address bus, a data bus, a control bus, and the like.

[0192] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0193] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0194] According to another aspect of the embodiments of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of any of the above embodiments.

[0195] Optionally, in an embodiment of the present application, the computer-readable medium is configured to store program codes for the processor to execute the following steps:

[0196] Extracting the human features of the target object using a preset key point connection topology as a limb connection constraint, extracting the clothing features of the target clothing, and fusing the human features and clothing features to generate a clothing segmentation image of the target clothing that matches the appearance of the target object based on the fused features, wherein the preset key point connection topology is used to constrain the feature extraction model to recognize the target object's limbs when the target object's limbs are occluded;

[0197] Transform the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing;

[0198] After the target object is segmented according to the clothing segmentation image, the pattern detail features of the target clothing obtained by the detail feature extraction model are used as information supplement of the clothing transformation image. The clothing transformation image is fitted to the segmented area to obtain the fitting image of the target object in the target clothing.

[0199] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0200] When implementing the embodiments of the present application, reference may be made to the above embodiments, which have corresponding technical effects.

[0201] It is understood that the embodiments described herein may be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or a combination thereof.

[0202] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0203] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0204] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0205] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0206] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0207] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0208] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application are essentially or partly contributed to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard drive, a ROM, a RAM, a magnetic disk, or an optical disk. It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0209] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

Claims

1. A virtual fitting system, characterized in that: The virtual fitting system is applied to a video platform, allowing viewers to use the clothes of characters in a play to try on virtual clothes, and / or allowing viewers to change the clothes of characters in the play. The system includes: a deformation segmentation unit, configured to extract human features of a target object using a preset key point connection topology map as a limb connection constraint, extract clothing features of a target garment, fuse the human features with the clothing features, and generate, based on the fused features, a clothing segmentation image of the target garment that matches the appearance of the target object, wherein the preset key point connection topology map is used to constrain the object feature extraction model to recognize the target object's limbs when the target object's limbs are occluded; a clothing transformation unit, configured to transform the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing; a fitting unit configured to segment the target object according to the clothing segmentation image, extract pattern detail features of the target clothing using a detail feature extraction model as information supplement for the clothing transformation image, fit the clothing transformation image to the segmented region, and obtain a fitting image of the target object wearing the target clothing.

2. The system according to claim 1, wherein: The deformation segmentation unit includes: a first feature extraction network, configured to extract the clothing features of the target clothing using the image of the target clothing; a second feature extraction network, configured to extract the human body features of the target object using a preset segmentation map, a preset key point map, and a preset key point connection topology map of the target object, wherein the preset segmentation map is an image obtained by pre-identifying and segmenting each part of the target object one by one, the preset key point map is used to mark key points of the human body, and the object feature extraction model includes the second feature extraction network; The first feature decoding network is used to fuse the clothing features output by each convolutional layer of the first feature extraction network and the human body features output by each convolutional layer of the second feature extraction network, and upsample and decode the fused features throughout the fusion process to obtain the clothing segmentation image.

3. The system according to claim 2, characterized in that The first feature extraction network includes multiple convolutional layers and multiple downsampling layers, which are connected in series and alternately arranged. The convolutional layers are used to extract the clothing features of the target clothing, and the downsampling layers are used to downsample and encode the clothing features output by the convolutional layers.

4. The system according to claim 2, wherein: The second feature extraction network includes multiple convolutional layers and multiple downsampling layers, which are connected in series and alternately arranged. The convolutional layers are used to extract the human body features of the target object, and the downsampling layers are used to downsample and encode the human body features output by the convolutional layers.

5. The system according to claim 2, wherein: The first feature decoding network includes multiple convolutional layers and multiple upsampling layers, the convolutional layers and the upsampling layers are connected in series and alternately arranged, and the upsampling layers are used to upsample and decode the fused features; The input of the first target convolutional layer of the first feature decoding network includes: the clothing features output by the last downsampling layer in the first feature extraction network and the human body features output by the last downsampling layer in the second feature extraction network; The input of the second target convolution layer of the first feature decoding network includes: the fusion feature output by the previous upsampling layer adjacent to the second target convolution layer, the clothing feature output by the first convolution layer in the first feature extraction network, and the human body feature output by the second convolution layer in the second feature extraction network; The input of the third target convolution layer of the first feature decoding network includes: the fusion feature output by the previous upsampling layer adjacent to the third target convolution layer, the clothing feature output by the third convolution layer in the first feature extraction network, and the human body feature output by the fourth convolution layer in the second feature extraction network; The input of the fourth target convolutional layer of the first feature decoding network includes: the fusion feature output by the previous upsampling layer adjacent to the fourth target convolutional layer, the clothing feature output by the fifth convolutional layer in the first feature extraction network, and the human body feature output by the sixth convolutional layer in the second feature extraction network; Among them, the position order of the first target convolution layer to the fourth target convolution layer in the first feature decoding network is consistent with the ascending order, the position order of the first convolution layer, the third convolution layer and the fifth convolution layer in the first feature extraction network is consistent with the ascending order, and the position order of the second convolution layer, the fourth convolution layer and the sixth convolution layer in the second feature extraction network is consistent with the ascending order.

6. The system according to claim 1, wherein: The clothing transformation unit includes: a spatial transformation network, configured to take the target clothing image and the clothing segmentation image as input, transform the target clothing according to the clothing segmentation image, and obtain a first clothing transformation image; The correction network is used to perform transformation correction on the clothing transformation image output by the spatial transformation network to obtain a final second clothing transformation image.

7. The system according to claim 6, characterized in that The correction network includes: A first feature encoding network includes a plurality of convolutional layers and a plurality of downsampling layers, wherein the convolutional layers and the downsampling layers are connected in series and alternately arranged; A second feature decoding network is connected in series with the first feature encoding network, comprising a plurality of convolutional layers and a plurality of upsampling layers, wherein the convolutional layers and the upsampling layers are connected in series and alternately arranged; The input of the first feature encoding network includes: the clothing transformation image and the clothing segmentation image output by the spatial transformation network; The input of the fifth target convolutional layer of the second feature decoding network includes: the intermediate transformed image output by the last downsampling layer of the first feature encoding network; The input of the sixth target convolutional layer of the second feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the sixth target convolutional layer and the intermediate transformed image output by the seventh convolutional layer in the first feature encoding network; The input of the seventh target convolutional layer of the second feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the seventh target convolutional layer and the intermediate transformed image output by the eighth convolutional layer in the first feature encoding network; The input of the eighth target convolutional layer of the second feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the eighth target convolutional layer and the intermediate transformed image output by the ninth convolutional layer in the first feature encoding network; The input of the ninth target convolutional layer of the second feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the ninth target convolutional layer and the intermediate transformed image output by the tenth convolutional layer in the first feature encoding network; Among them, the position order of the fifth target convolution layer to the ninth target convolution layer in the second feature decoding network is consistent with the ascending order, and the position order of the seventh convolution layer to the tenth convolution layer in the first feature encoding network is opposite to the ascending order.

8. The system according to claim 6, wherein: The trial installation unit comprises: a branch network, configured to extract the pattern detail features in the target clothing through the first clothing transformation image, wherein the detail feature extraction model includes the branch network; The backbone network is used to fit the second clothing transformation image to the segmented area, and connect the pattern detail features during the fitting process to obtain the fitting image of the target object wearing the target clothing.

9. The system according to claim 8, characterized in that The branch network includes: A second feature encoding network includes a plurality of convolutional layers and a plurality of downsampling layers, wherein the convolutional layers and the downsampling layers are connected in series and alternately arranged; A third feature decoding network is connected in series with the second feature encoding network, comprising a plurality of convolutional layers and a plurality of upsampling layers, wherein the convolutional layers and the upsampling layers are connected in series and alternately arranged; An input of the second feature encoding network includes the first clothing transformation image.

10. The system according to claim 9, characterized in that The backbone network includes: A third feature encoding network includes a plurality of convolutional layers and a plurality of downsampling layers, wherein the convolutional layers and the downsampling layers are connected in series and alternately arranged; a fourth feature decoding network, connected in series with the second feature encoding network, comprising a plurality of convolutional layers and a plurality of upsampling layers, wherein the convolutional layers and the upsampling layers are connected in series and alternately arranged; The input of the third feature encoding network includes: the second clothing transformation image, the clothing segmentation image, and the target object's information to be retained, wherein the target object's information to be retained includes the target object's part to be retained obtained by segmenting the target object's image; The input of the tenth target convolutional layer of the fourth feature decoding network includes: the intermediate coded image output by the last downsampling layer of the third feature encoding network; The input of the eleventh target convolutional layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the eleventh target convolutional layer, the intermediate encoded image output by the eleventh convolutional layer in the third feature encoding network, and the pattern detail features output by the twelfth convolutional layer in the third feature decoding network, and the decoding scale of the eleventh target convolutional layer is the same as that of the twelfth convolutional layer; The input of the twelfth target convolutional layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the twelfth target convolutional layer, the intermediate encoded image output by the thirteenth convolutional layer in the third feature encoding network, and the pattern detail features output by the fourteenth convolutional layer in the third feature decoding network, and the decoding scale of the twelfth target convolutional layer is the same as that of the fourteenth convolutional layer; The input of the thirteenth target convolutional layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the thirteenth target convolutional layer, the intermediate encoded image output by the fifteenth convolutional layer in the third feature encoding network, and the pattern detail features output by the sixteenth convolutional layer in the third feature decoding network, and the decoding scale of the thirteenth target convolutional layer is the same as that of the sixteenth convolutional layer; The input of the fourteenth target convolutional layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the fourteenth target convolutional layer, the intermediate encoded image output by the seventeenth convolutional layer in the third feature encoding network, and the pattern detail features output by the eighteenth convolutional layer in the third feature decoding network, and the decoding scale of the fourteenth target convolutional layer is the same as that of the eighteenth convolutional layer; Among them, the position order of the tenth target convolution layer to the fourteenth target convolution layer in the fourth feature decoding network is consistent with the ascending order, the position order of the eleventh convolution layer, the thirteenth convolution layer, the fifteenth convolution layer and the seventeenth convolution layer in the third feature encoding network is opposite to the ascending order, and the position order of the twelfth convolution layer, the fourteenth convolution layer, the sixteenth convolution layer and the eighteenth convolution layer in the third feature decoding network is consistent with the ascending order.

11. A virtual fitting method, applied to the virtual fitting system according to any one of claims 1 to 10, characterized in that: include: Extracting human features of a target object using a preset key point connection topology map as limb connection constraints, extracting clothing features of a target garment, and fusing the human features with the clothing features to generate, based on the fused features, a clothing segmentation image of the target garment that matches the appearance of the target object, wherein the preset key point connection topology map is used to constrain a feature extraction model to identify the target object's limbs when the target object's limbs are obscured; Transforming the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing; After the target object is segmented according to the clothing segmentation image, pattern detail features of the target clothing obtained by using a detail feature extraction model are used as information supplement of the clothing transformation image. The clothing transformation image is fitted to the segmented area to obtain an image of the target object fitting the target clothing.

12. The method according to claim 11, characterized in that Extracting the human body features of the target object using a preset key point connection topology as a limb connection constraint, extracting the clothing features of the target clothing, and fusing the human body features with the clothing features to generate a clothing segmentation image of the target clothing that matches the appearance of the target object based on the fused features, including: Extracting the clothing features of the target clothing using the image of the target clothing; Extracting the human body features of the target object using a preset segmentation map, a preset key point map, and a preset key point connection topology map of the target object, wherein the preset segmentation map is an image obtained by pre-identifying and segmenting each part of the target object one by one, and the preset key point map is used to mark key points of the human body; The clothing features and the human body features are fused, and the fused features are upsampled and decoded throughout the fusion process to obtain the clothing segmentation image.

13. The method according to claim 11, characterized in that Transforming the target clothing according to the clothing segmentation image to obtain the clothing transformation image of the target clothing includes: Acquire an image of the target's attire; transforming the target clothing image according to the clothing segmentation image to obtain a first clothing transformation image; Performing transformation correction on the first clothing transformation image to obtain a final second clothing transformation image.

14. The method according to claim 13, characterized in that Using the pattern detail features of the target clothing extracted by the detail feature extraction model as information supplement of the clothing transformation image, fitting the clothing transformation image to the segmented region, and obtaining a fitting image of the target object wearing the target clothing includes: Inputting the first clothing transformation image into the detail feature extraction model to obtain the pattern detail features of the target clothing; The second clothing transformation image is decoded based on the clothing segmentation image and the information to be retained of the target object. The pattern detail features are connected during the decoding process to fit the second clothing transformation image, which incorporates the pattern detail features, to the segmented region, thereby obtaining the fitting image of the target object wearing the target clothing. The information to be retained of the target object includes the target object portion to be retained obtained by segmenting the image of the target object.

15. The method according to claim 14, characterized in that Before inputting the first clothing transformation image into the detail feature extraction model, the method further includes training the detail feature extraction model in the following manner: Obtaining a generated image obtained by performing feature extraction on a training image by the detail feature extraction model and a ground-truth image corresponding to the generated image, wherein both the generated image and the ground-truth image include images at multiple decoding scales, and the generated image and the corresponding ground-truth image at the same decoding scale are grouped together; Using the center of the generated image in the same group as the center of the cropping frame, cropping the generated image and the true image to obtain a first generated image and a first true image; Randomly determining the centers of a plurality of cropping frames on the first generated image and the first ground-truth image in the same group, and cropping the first generated image and the first ground-truth image according to the centers of the plurality of cropping frames to obtain a plurality of second generated images and a plurality of second ground-truth images, wherein the centers of the plurality of cropping frames on the first generated image and the first ground-truth image are the same, and the second generated images and the second ground-truth images having the same centers of the cropping frames correspond to each other, and a group of images having a larger decoding scale has a greater number of cropping frames; Determine an L1 loss value between each second generated image and the corresponding second true image using a target loss function, and use the sum of all the L1 loss values ​​as a final loss value for the current decoding scale; The final loss value is used to adjust parameters of the detail feature extraction model to optimize the decoding accuracy of the detail feature extraction model for the current decoding scale.

16. An electronic device comprising a memory, a processor, a communication interface, and a communication bus, wherein the memory stores a computer program that can be run on the processor, and the memory and the processor communicate via the communication bus and the communication interface, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 11 to 15 are implemented.

17. A computer-readable medium having a non-volatile program code executable by a processor, characterized in that The program code enables the processor to execute the method according to any one of claims 11 to 15.

Images