Virtual trial assembly system, method, device and computer readable medium

Abstract

The present application relates to a virtual fitting system, method, device and computer-readable medium. The system includes: a deformation segmentation unit, which is used to extract the human body features of the target object and the dressing features of the target clothing, and after fusing the human body features and the dressing features, generates a clothing segmentation image of the target clothing that matches the appearance of the target object based on multi-level correction of the fused features; a clothing transformation unit, which is used to transform the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing; a fitting unit, which is used to segment the target object according to the clothing segmentation image, fit the clothing transformation image to the segmentation area, and obtain a fitting image of the target object for the target clothing. The present application fuses the human body features of the target object and the dressing features of the target clothing, directly predicts and outputs a deformed clothing segmentation image that matches the appearance of the target object, and solves the technical problem of inaccurate virtual fitting results due to cumulative errors.

Description

Technical Field

[0001] The present application relates to the field of multimedia intelligent processing technology, and in particular to a virtual trial installation system, method, device and computer-readable medium. Background Art

[0002] Virtual try-on involves generating a photo of a model wearing the target garment, given an image of a target garment and a model, to virtually try the garment on the model. The resulting try-on must meet four requirements: 1. Maintain the pose and shape of the person in the image; 2. Ensure the target garment fits the person's pose and shape; 3. Preserve all other information about the garment, such as undergarments; and 4. Ensure that creases, patterns, and logos remain clear and natural after the garment is worn. In e-commerce, effective virtual try-on technology not only provides consumers with a novel interactive experience but also guides and stimulates them to make faster purchasing decisions. In video platforms, virtual try-on technology can offer viewers a completely new viewing experience. For example, while watching a movie, viewers can upload their own photos and virtually wear the costumes of the characters in the show, creating an immersive viewing experience. Alternatively, viewers can switch between characters' outfits at will, increasing interactivity and fun, and potentially increasing user retention on the video platform.

[0003] At present, in related technologies, virtual fitting is usually performed through the ACGPN algorithm. The ACGPN algorithm consists of a total of four modules: module A, module B, module C, and module D. Module A is used to generate a human body instance segmentation map, parse the input human body image, and mark the limbs of the person in the image with different numbers, such as filling the left arm area with 0, the right arm area with 1, the body torso with 2, and so on; module B is used to generate a deformed clothing segmentation map, which is a binary image, with the deformed clothing area in the image filled with 1 and the rest of the positions filled with 0; module C consists of two parts: the spatial transformer network STN (Spatial Transformer Network) and the correction network, where STN outputs a rough transformed clothing image, and the correction network corrects the output of STN and outputs the final transformed clothing image; module D combines the outputs of the above modules to generate the final virtual fitting image. However, in deep learning tasks, the segmentation of human instances involves the prediction of many detailed areas such as hands, feet, hair, face, and torso. It is very difficult to obtain very precise prediction results. Therefore, the error in module A in the ACGPN algorithm further leads to the error in module B. As a result, the errors of modules A and B accumulate in modules C and D, resulting in inaccurate virtual try-on results in the final output.

[0004] Currently, no effective solution has been proposed to the problem of inaccurate virtual try-on results due to cumulative errors. Summary of the Invention

[0005] The present application provides a virtual fitting system, method, device and computer-readable medium to solve the technical problem of inaccurate virtual fitting results due to cumulative errors.

[0006] According to one aspect of an embodiment of the present application, the present application provides a virtual fitting system, comprising:

[0007] a deformation segmentation unit for extracting the human features of the target object and the clothing features of the target clothing, fusing the human features and the clothing features, and generating a clothing segmentation image of the target clothing that matches the appearance of the target object based on multi-level correction of the fused features;

[0008] a clothing transformation unit, configured to transform the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing;

[0009] The fitting unit is used to segment the target object according to the clothing segmentation image, fit the clothing transformation image to the segmentation area, and obtain a fitting image of the target object in the target clothing.

[0010] Optionally, the deformation and segmentation unit includes:

[0011] a first feature extraction network, for extracting clothing features of the target clothing using an image of the target clothing;

[0012] a second feature extraction network for extracting body features of the target object using a preset segmentation map and a preset key point map of the target object, wherein the preset segmentation map is an image obtained by pre-identifying and segmenting each part of the target object, and the preset key point map is used to mark key points of the body;

[0013] The first feature decoding network is used to fuse the clothing features output by each convolutional layer of the first feature extraction network and the human body features output by each convolutional layer of the second feature extraction network, and upsample and decode the fused features throughout the fusion process to obtain a basic clothing segmentation image;

[0014] The multi-level correction network is used to perform multi-level correction on the basic clothing segmentation image to obtain the final clothing segmentation image.

[0015] Optionally, the multi-stage correction network includes a plurality of sub-correction networks with the same structure and arranged in series; wherein,

[0016] The input of the first sub-correction network includes: the base clothing segmentation image;

[0017] The input of each remaining sub-correction network includes: the basic clothing segmentation image and the intermediate corrected image output by the previous sub-correction network;

[0018] The intermediate corrected image output by the last sub-correction network is the final clothing segmentation image.

[0019] Optionally, the sub-correction network includes:

[0020] The first feature encoding network includes multiple convolutional layers and multiple downsampling layers, where the convolutional layers and the downsampling layers are connected in series and alternately arranged;

[0021] The second feature decoding network is connected in series with the first feature encoding network, and includes multiple convolutional layers and multiple upsampling layers, where the convolutional layers and the upsampling layers are connected in series and alternately arranged;

[0022] The input of the first feature encoding network includes: the base clothing transformation image; or the base clothing transformation image and the intermediate correction image output by the previous sub-correction network;

[0023] The input of the first target convolutional layer of the second feature decoding network includes: the intermediate rectified image output by the last downsampling layer of the first feature encoding network;

[0024] The input of the second target convolutional layer of the second feature decoding network includes: the intermediate corrected image output by the previous upsampling layer adjacent to the second target convolutional layer and the intermediate corrected image output by the first convolutional layer in the first feature encoding network;

[0025] The input of the third target convolution layer of the second feature decoding network includes: the intermediate corrected image output by the previous upsampling layer adjacent to the third target convolution layer and the intermediate corrected image output by the second convolution layer in the first feature encoding network;

[0026] The input of the fourth target convolution layer of the second feature decoding network includes: the intermediate corrected image output by the previous upsampling layer adjacent to the fourth target convolution layer and the intermediate corrected image output by the third convolution layer in the first feature encoding network;

[0027] The input of the fifth target convolutional layer of the second feature decoding network includes: the intermediate corrected image output by the previous upsampling layer adjacent to the fifth target convolutional layer and the intermediate corrected image output by the fourth convolutional layer in the first feature encoding network;

[0028] Among them, the position order of the first target convolution layer to the fifth target convolution layer in the second feature decoding network is consistent with the ascending order, and the position order of the first convolution layer to the fourth convolution layer in the first feature encoding network is opposite to the ascending order.

[0029] Optionally, the first feature extraction network includes multiple convolutional layers and multiple downsampling layers, which are connected in series and alternately arranged. The convolutional layers are used to extract the clothing features of the target clothing, and the downsampling layers are used to downsample and encode the clothing features output by the convolutional layers.

[0030] Optionally, the second feature extraction network includes multiple convolutional layers and multiple downsampling layers, which are connected in series and alternately arranged. The convolutional layers are used to extract the human body features of the target object, and the downsampling layers are used to downsample and encode the human body features output by the convolutional layers.

[0031] Optionally, the first feature decoding network includes multiple convolutional layers and multiple upsampling layers, the convolutional layers and the upsampling layers are connected in series and alternately arranged, and the upsampling layers are used to upsample and decode the fused features;

[0032] The input of the sixth target convolutional layer of the first feature decoding network includes: the clothing features output by the last downsampling layer in the first feature extraction network and the human body features output by the last downsampling layer in the second feature extraction network;

[0033] The input of the seventh target convolutional layer of the first feature decoding network includes: the fused features output by the previous upsampling layer adjacent to the seventh target convolutional layer, the clothing features output by the fifth convolutional layer in the first feature extraction network, and the human body features output by the sixth convolutional layer in the second feature extraction network;

[0034] The input of the eighth target convolution layer of the first feature decoding network includes: the fused features output by the previous upsampling layer adjacent to the eighth target convolution layer, the clothing features output by the seventh convolution layer in the first feature extraction network, and the human body features output by the eighth convolution layer in the second feature extraction network;

[0035] The input of the ninth target convolution layer of the first feature decoding network includes: the fused features output by the previous upsampling layer adjacent to the ninth target convolution layer, the clothing features output by the ninth convolution layer in the first feature extraction network, and the human body features output by the tenth convolution layer in the second feature extraction network;

[0036] Among them, the position order of the sixth target convolution layer to the ninth target convolution layer in the first feature decoding network is consistent with the ascending order, the position order of the fifth convolution layer, the seventh convolution layer and the ninth convolution layer in the first feature extraction network is consistent with the ascending order, and the position order of the sixth convolution layer, the eighth convolution layer and the tenth convolution layer in the second feature extraction network is consistent with the ascending order.

[0037] Optionally, the clothing changing unit includes:

[0038] A spatial transformation network is used to take the target clothing image and the clothing segmentation image as input, transform the target clothing according to the clothing segmentation image, and obtain the clothing transformation image;

[0039] The correction network is used to perform transformation correction on the clothing transformation image output by the spatial transformation network to obtain the final clothing transformation image.

[0040] Optionally, the correction network includes:

[0041] The second feature encoding network includes multiple convolutional layers and multiple downsampling layers, where the convolutional layers and the downsampling layers are connected in series and alternately arranged;

[0042] The third feature decoding network is connected in series with the second feature encoding network, and includes multiple convolutional layers and multiple upsampling layers, where the convolutional layers and the upsampling layers are connected in series and alternately arranged;

[0043] The input of the second feature encoding network includes: the clothing transformation image and clothing segmentation image output by the spatial transformation network;

[0044] The input of the tenth target convolutional layer of the third feature decoding network includes: the intermediate transformed image output by the last downsampling layer of the second feature encoding network;

[0045] The input of the eleventh target convolutional layer of the third feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the eleventh target convolutional layer and the intermediate transformed image output by the eleventh convolutional layer in the second feature encoding network;

[0046] The input of the twelfth target convolution layer of the third feature decoding network includes: the intermediate transformation image output by the previous upsampling layer adjacent to the twelfth target convolution layer and the intermediate transformation image output by the twelfth convolution layer in the second feature encoding network;

[0047] The input of the thirteenth target convolution layer of the third feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the thirteenth target convolution layer and the intermediate transformed image output by the thirteenth convolution layer in the second feature encoding network;

[0048] The input of the fourteenth target convolution layer of the third feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the fourteenth target convolution layer and the intermediate transformed image output by the fourteenth convolution layer in the second feature encoding network;

[0049] Among them, the position order of the tenth target convolution layer to the fourteenth target convolution layer in the third feature decoding network is consistent with the ascending order, and the position order of the eleventh convolution layer to the fourteenth convolution layer in the second feature encoding network is opposite to the ascending order.

[0050] Optionally, the trial installation unit includes:

[0051] A third feature encoding network includes a plurality of convolutional layers and a plurality of downsampling layers, wherein the convolutional layers and the downsampling layers are connected in series and alternately arranged;

[0052] a fourth feature decoding network, connected in series with the third feature encoding network, comprising a plurality of convolutional layers and a plurality of upsampling layers, wherein the convolutional layers and the upsampling layers are connected in series and arranged alternately;

[0053] The input of the third feature encoding network includes: the clothing transformation image, the clothing segmentation image, and the target object's to-be-retained information, wherein the target object's to-be-retained information includes the target object's to-be-retained part obtained by segmenting the target object's image;

[0054] The input of the fifteenth target convolutional layer of the fourth feature decoding network includes: the intermediate encoded image output by the last downsampling layer of the third feature encoding network;

[0055] The input of the sixteenth target convolution layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the sixteenth target convolution layer and the intermediate encoded image output by the fifteenth convolution layer in the third feature encoding network;

[0056] The input of the seventeenth target convolutional layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the seventeenth target convolutional layer and the intermediate encoded image output by the sixteenth convolutional layer in the third feature encoding network;

[0057] The input of the eighteenth target convolution layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the eighteenth target convolution layer and the intermediate encoded image output by the seventeenth convolution layer in the third feature encoding network;

[0058] The input of the nineteenth target convolution layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the nineteenth target convolution layer and the intermediate encoded image output by the eighteenth convolution layer in the third feature encoding network;

[0059] Among them, the position order of the fifteenth target convolution layer to the nineteenth target convolution layer in the fourth feature decoding network is consistent with the ascending order, and the position order of the fifteenth convolution layer to the eighteenth convolution layer in the third feature encoding network is opposite to the ascending order.

[0060] According to another aspect of the embodiments of the present application, the present application provides a virtual fitting method, comprising:

[0061] Extracting the human features of the target object and the clothing features of the target clothing, fusing the human features and the clothing features, and generating a clothing segmentation image of the target clothing that matches the appearance of the target object based on multi-level correction of the fused features;

[0062] Transform the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing;

[0063] After the target object is segmented according to the clothing segmentation image, the clothing transformation image is fitted to the segmented region to obtain a fitting image of the target object in the target clothing.

[0064] Optionally, extracting the human features of the target object and the clothing features of the target clothing, fusing the human features and the clothing features, and generating a clothing segmentation image of the target clothing that matches the appearance of the target object based on the fused features includes:

[0065] Extracting clothing features of the target clothing using an image of the target clothing;

[0066] The human body features of the target object are extracted using a preset segmentation map and a preset key point map of the target object, wherein the preset segmentation map is an image obtained by pre-identifying and segmenting each part of the target object one by one, and the preset key point map is used to mark the key points of the human body;

[0067] Fuse clothing features and human body features, and upsample and decode the fused features throughout the fusion process to obtain a basic clothing segmentation image;

[0068] The basic clothing segmentation image is subjected to multi-level correction to obtain the final clothing segmentation image.

[0069] Optionally, transforming the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing includes:

[0070] Obtain an image of the target's clothing;

[0071] Transform the target clothing image according to the clothing segmentation image to obtain a clothing transformation image;

[0072] Perform transformation correction on the clothing transformation image to obtain the final clothing transformation image.

[0073] According to another aspect of an embodiment of the present application, the present application provides an electronic device, including a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, the memory and the processor communicate through the communication bus and the communication interface, and the steps of the above method are implemented when the processor executes the computer program.

[0074] According to another aspect of an embodiment of the present application, the present application further provides a computer-readable medium having a non-volatile program code executable by a processor, where the program code enables the processor to execute the above method.

[0075] The above technical solution provided by the embodiment of the present application has the following advantages compared with the related art:

[0076] The technical solution of the present application provides a virtual fitting system, comprising: a deformation segmentation unit for extracting the human body features of a target object and the clothing features of a target garment, and after fusing the human body features and the clothing features, generating a clothing segmentation image of the target garment that matches the appearance of the target object based on multi-level correction of the fused features; a clothing transformation unit for transforming the target garment according to the clothing segmentation image to obtain a clothing transformation image of the target garment; and a fitting unit for segmenting the target object according to the clothing segmentation image, fitting the clothing transformation image to the segmented region, and obtaining a fitting image of the target object for the target garment. The present application fuses the human body features of the target object and the clothing features of the target garment to directly predict and output a deformed clothing segmentation image that matches the appearance of the target object, thereby reducing the cumulative prediction error of the related art of first predicting the human body instance segmentation image of the target object and then predicting the deformed clothing image segmentation map based on the human body instance segmentation image. Moreover, the deformed clothing image segmentation map is subjected to multi-level correction, further improving the accuracy of the fitting image, making the fitting effect more fitting and natural, and solving the technical problem of inaccurate virtual fitting results due to cumulative error. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0079] Figure 1a Schematic diagram of a human body instance segmentation map predicted and output by module A of the ACGPN algorithm in the related art;

[0080] Figure 1b Schematic diagram of the segmentation map of the deformed clothing image predicted and output by the B module of the ACGPN algorithm in the related art;

[0081] Figure 1c Schematic diagram of the deformed clothing image predicted and output by the C module of the ACGPN algorithm in the related art;

[0082] Figure 1d Schematic diagram of the test results of the predicted output of the D module of the ACGPN algorithm in the related art;

[0083] Figure 2 Schematic diagram of an optional virtual trial assembly system structure provided according to an embodiment of the present application;

[0084] Figure 3 A schematic diagram of a multi-stage correction network structure of an optional deformation segmentation unit provided according to an embodiment of the present application;

[0085] Figure 4 A schematic structural diagram of an optional multi-stage correction network neutron correction network provided according to an embodiment of the present application;

[0086] Figure 5 Schematic diagram of a feature extraction network structure of an optional deformation segmentation unit provided according to an embodiment of the present application;

[0087] Figure 6 Schematic diagram of a feature decoding network structure of an optional deformation segmentation unit provided according to an embodiment of the present application;

[0088] Figure 7 This is a schematic diagram of a correction network structure of an optional clothing transformation unit provided according to an embodiment of the present application;

[0089] Figure 8 A schematic diagram of a network structure of an optional trial installation unit provided according to an embodiment of the present application;

[0090] Figure 9 A schematic flow chart of an optional virtual trial assembly method provided according to an embodiment of the present application;

[0091] Figure 10 A schematic diagram of an optional electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0092] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0093] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of this application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0094] In related technologies, virtual fitting is usually performed using the ACGPN algorithm. The ACGPN algorithm consists of four modules: module A, module B, module C, and module D. Module A is used to generate a human instance segmentation map, analyze the input human image, and mark the limbs of the person in the image with different numbers, such as filling the left arm area with 0, the right arm area with 1, the body torso with 2, etc. The prediction results of the human instance segmentation map are as follows: Figure 1a As shown in Figure 2; Module B is used to generate the segmentation map of the deformed clothes. The segmentation map is a binary image. The deformed clothes area in the image is filled with 1, and the rest of the positions are filled with 0. The prediction result of the deformed clothes segmentation map is as follows: Figure 1b As shown in Figure 2, the C module consists of two parts: the spatial transformer network STN (SpatialTransformer Network) and the correction network. The STN outputs a rough transformed clothing image, and the correction network corrects the output of the STN and outputs the final transformed clothing image. The prediction result of the transformed clothing image is shown in Figure 2. Figure 1c As shown; Module D combines the outputs of the above modules to generate the final virtual try-on image, the effect is as follows Figure 1d However, in deep learning tasks, the segmentation of human instances involves the prediction of numerous detailed areas, such as hands, feet, hair, face, and torso. Obtaining highly refined predictions is extremely challenging. Consequently, errors in module A in the ACGPN algorithm further lead to errors in module B. Consequently, the errors in modules A and B accumulate in modules C and D, resulting in inaccurate virtual try-on results.

[0095] In order to solve the technical problems mentioned in the background technology, according to one aspect of the embodiment of the present application, the present application provides a virtual fitting system, such as Figure 2 Shown, including:

[0096] The deformation segmentation unit 201 is configured to extract the human features of the target object and the clothing features of the target clothing, fuse the human features and the clothing features, and generate a clothing segmentation image of the target clothing that matches the appearance of the target object based on multi-level correction of the fused features;

[0097] a clothing transformation unit 203 for transforming the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing;

[0098] The fitting unit 205 is configured to segment the target object according to the clothing segmentation image, and then fit the clothing transformation image to the segmented region to obtain a fitting image of the target object in the target clothing.

[0099] Based on the deformation segmentation unit 201, the human body features of the target object and the clothing features of the target clothing are extracted and fused, so that the neural network model predicts and outputs the clothing segmentation image of the target clothing that matches the appearance of the target object at one time, that is, the A module and the B module of the ACGPN algorithm in the related technology are merged to directly predict and output the deformed clothing image segmentation map, skipping the prediction of the human body instance segmentation map, and instead fusing the human body features and clothing features to predict the clothing segmentation image. This not only greatly simplifies the original scheme, reduces the number of modules of the original scheme, and enhances the integrity of the scheme, but also reduces the accumulation of prediction errors between multiple modules, further improving the image accuracy in subsequent steps. In addition, the refinement of the clothing segmentation image is improved through multi-level correction of the fused features.

[0100] The technical solution of this application can be applied to the e-commerce sector, where the target object can be an e-commerce merchant's model, and the target outfit can be the merchant's clothing for sale. Based on this virtual fitting system, the merchant can batch-generate fitting images of their own clothing for sale simply by using the model's modeling photos. The target object can also be a consumer, who can generate fitting images of themselves trying on the clothing they are about to purchase by uploading their own photos. Therefore, in the e-commerce sector, good virtual fitting technology can not only provide consumers with a novel interactive experience, but also guide and stimulate consumers to make purchasing decisions more quickly.

[0101] The technical solution of this application can also be applied to the field of video platforms, where the target clothing can be the clothing of virtual characters in film and television works, game works, and animation works, and the target object can be the audience. The audience can upload their own photos and try on the clothes of the characters in the play. Therefore, in the field of video platforms, virtual fitting technology can bring a new immersive viewing experience to viewers and increase the interaction between fans and characters in the play. Not only that, the target object can also be a character in the play, and the target clothing can be a specific clothing selected by the audience, so that the audience can switch the clothes of the characters in the play at will, increasing the interactivity and fun of watching movies, and also making users active and retaining longer on the video platform.

[0102] Optionally, the deformation and segmentation unit includes:

[0103] a first feature extraction network, for extracting clothing features of the target clothing using an image of the target clothing;

[0104] a second feature extraction network for extracting body features of the target object using a preset segmentation map and a preset key point map of the target object, wherein the preset segmentation map is an image obtained by pre-identifying and segmenting each part of the target object, and the preset key point map is used to mark key points of the body;

[0105] The first feature decoding network is used to fuse the clothing features output by each convolutional layer of the first feature extraction network and the human body features output by each convolutional layer of the second feature extraction network, and upsample and decode the fused features throughout the fusion process to obtain a basic clothing segmentation image;

[0106] The multi-level correction network is used to perform multi-level correction on the basic clothing segmentation image to obtain the final clothing segmentation image. Figure 3 As shown, the multi-stage correction network includes a plurality of sub-correction networks with the same structure and arranged in series; wherein,

[0107] The input of the first sub-correction network includes: the base clothing segmentation image;

[0108] The input of each remaining sub-correction network includes: the basic clothing segmentation image and the intermediate corrected image output by the previous sub-correction network;

[0109] The intermediate corrected image output by the last sub-correction network is the final clothing segmentation image.

[0110] In the embodiment of the present application, one sub-correction network is a correction stage. The input of the first stage is the basic clothing segmentation image. The input of each subsequent stage includes the basic clothing segmentation image and the intermediate correction image output by the previous stage. The intermediate correction image is also a clothing segmentation image. The intermediate correction image output by the last stage is the final clothing segmentation image ( Figure 3 ). Therefore, the multi-stage correction structure proposed in this application can adaptively compensate for the lost accuracy of the module. The latter stage is responsible for correcting the segmentation results of the previous stage, from coarse to fine, and finally obtains a refined clothing segmentation image.

[0111] Alternatively, as Figure 4 As shown, the sub-correction network includes:

[0112] The first feature encoding network includes multiple convolutional layers and multiple downsampling layers, where the convolutional layers and the downsampling layers are connected in series and alternately arranged;

[0113] The second feature decoding network is connected in series with the first feature encoding network, and includes multiple convolutional layers and multiple upsampling layers, where the convolutional layers and the upsampling layers are connected in series and alternately arranged;

[0114] The input of the first feature encoding network includes: the base clothing transformation image; or the base clothing transformation image and the intermediate correction image output by the previous sub-correction network;

[0115] The input of the first target convolutional layer of the second feature decoding network includes: the intermediate rectified image output by the last downsampling layer of the first feature encoding network;

[0116] The input of the second target convolutional layer of the second feature decoding network includes: the intermediate corrected image output by the previous upsampling layer adjacent to the second target convolutional layer and the intermediate corrected image output by the first convolutional layer in the first feature encoding network;

[0117] The input of the third target convolution layer of the second feature decoding network includes: the intermediate corrected image output by the previous upsampling layer adjacent to the third target convolution layer and the intermediate corrected image output by the second convolution layer in the first feature encoding network;

[0118] The input of the fourth target convolution layer of the second feature decoding network includes: the intermediate corrected image output by the previous upsampling layer adjacent to the fourth target convolution layer and the intermediate corrected image output by the third convolution layer in the first feature encoding network;

[0119] The input of the fifth target convolutional layer of the second feature decoding network includes: the intermediate corrected image output by the previous upsampling layer adjacent to the fifth target convolutional layer and the intermediate corrected image output by the fourth convolutional layer in the first feature encoding network;

[0120] Among them, the position order of the first target convolution layer to the fifth target convolution layer in the second feature decoding network is consistent with the ascending order, and the position order of the first convolution layer to the fourth convolution layer in the first feature encoding network is opposite to the ascending order.

[0121] In the embodiment of this application, Figure 4 For example, the sub-correction network includes a first feature encoding network ( Figure 4 Downsampling layer after convolution 1 to convolution 4) and the second feature decoding network ( Figure 4 The first to fifth target convolutional layers correspond to convolutions 5 to 9 in the second feature decoding network, respectively, and their position order is consistent with the ascending order; the first to fourth convolutional layers correspond to convolutions 4, 3, 2, and 1 in the first feature encoding network, respectively, and their position order is opposite to the ascending order.

[0122] In the embodiment of the present application, the first feature extraction network takes the image of the target clothing as input and extracts the clothing features of the target clothing layer by layer through multiple convolutional layers.

[0123] Alternatively, as Figure 5 As shown in branch network 1 in , the first feature extraction network includes multiple convolutional layers and multiple downsampling layers. The convolutional layers and the downsampling layers are connected in series and alternately. The convolutional layers are used to extract the clothing features of the target clothing, and the downsampling layers are used to downsample and encode the clothing features output by the convolutional layers.

[0124] In this embodiment of the present application, the second feature extraction network takes a preset segmentation map and a preset keypoint map of the target object as input and extracts the human features of the target object layer by layer through multiple convolutional layers. The preset segmentation map of the target object can be obtained by processing a photograph of the target object using a human body parsing algorithm, such as RefineNet, which separately segments the left hand, right hand, head, hair, torso, left leg, and right leg of the human body. The preset keypoint map is used to mark key points of the human body.

[0125] Alternatively, as Figure 5 As shown in branch network 2 in , the second feature extraction network includes multiple convolutional layers and multiple downsampling layers. The convolutional layers and the downsampling layers are connected in series and alternately. The convolutional layers are used to extract the human features of the target object, and the downsampling layers are used to downsample and encode the human features output by the convolutional layers.

[0126] In the embodiment of this application, Figure 5 The “+” in the figure not only represents the fusion of the clothing features output by the last downsampling layer of the first feature extraction network and the human body features output by the last downsampling layer of the second feature extraction network, but also represents the feature fusion of the outputs of each convolutional layer in the first and second feature extraction networks.

[0127] Alternatively, as Figure 6 As shown, the first feature decoding network includes multiple convolutional layers and multiple upsampling layers, the convolutional layers and the upsampling layers are connected in series and alternately arranged, and the upsampling layers are used to upsample and decode the fusion features;

[0128] The input of the sixth target convolutional layer of the first feature decoding network includes: the clothing features output by the last downsampling layer in the first feature extraction network and the human body features output by the last downsampling layer in the second feature extraction network;

[0129] The input of the seventh target convolutional layer of the first feature decoding network includes: the fused features output by the previous upsampling layer adjacent to the seventh target convolutional layer, the clothing features output by the fifth convolutional layer in the first feature extraction network, and the human body features output by the sixth convolutional layer in the second feature extraction network;

[0130] The input of the eighth target convolution layer of the first feature decoding network includes: the fused features output by the previous upsampling layer adjacent to the eighth target convolution layer, the clothing features output by the seventh convolution layer in the first feature extraction network, and the human body features output by the eighth convolution layer in the second feature extraction network;

[0131] The input of the ninth target convolution layer of the first feature decoding network includes: the fused features output by the previous upsampling layer adjacent to the ninth target convolution layer, the clothing features output by the ninth convolution layer in the first feature extraction network, and the human body features output by the tenth convolution layer in the second feature extraction network;

[0132] Among them, the position order of the sixth target convolution layer to the ninth target convolution layer in the first feature decoding network is consistent with the ascending order, the position order of the fifth convolution layer, the seventh convolution layer and the ninth convolution layer in the first feature extraction network is consistent with the ascending order, and the position order of the sixth convolution layer, the eighth convolution layer and the tenth convolution layer in the second feature extraction network is consistent with the ascending order.

[0133] In the embodiment of this application, Figure 5 、 6 For example, the sixth to ninth target convolution layers are convolution 6 to convolution 9 in the first feature decoding network, and their position order is consistent with the increasing order; the fifth convolution layer, the seventh convolution layer, and the ninth convolution layer are Figure 3 Convolution 2, convolution 3, and convolution 4 in branch network 1 (the first feature extraction network) are in the same order of increasing position; the sixth convolution layer, the eighth convolution layer, and the tenth convolution layer are respectively Figure 3 The position order of convolution 2, convolution 3, and convolution 4 in branch network 2 (the second feature extraction network) is consistent with the increasing order.

[0134] The aforementioned network structure of the deformable segmentation unit fuses the clothing and body features extracted at each stage of the feature extraction network, avoiding the loss of low-dimensional features after extracting high-dimensional features. This allows for a more comprehensive and accurate prediction of a clothing segmentation image that matches the target object's appearance. Furthermore, the clothing segmentation image is corrected through a multi-stage correction network, from coarse to fine, to gradually obtain a more refined clothing segmentation image. This clothing segmentation image represents the outline of the target object wearing the target clothing.

[0135] Optionally, the clothing changing unit includes:

[0136] A spatial transformation network is used to take the target clothing image and the clothing segmentation image as input, transform the target clothing according to the clothing segmentation image, and obtain the clothing transformation image;

[0137] The correction network is used to perform transformation correction on the clothing transformation image output by the spatial transformation network to obtain the final clothing transformation image.

[0138] In an embodiment of the present application, the Spatial Transformer Network (STN) is composed of three parts: a localization network, a grid generator, and a sampler. For the input image, the spatial transformer network first uses a localization network to predict the required transformation, that is, after several consecutive layers of calculations (including convolution and fully connected calculations), the grid generator and the sampler then transform the image. The spatial transformer network can adaptively perform spatial transformation and alignment on the data. In an embodiment of the present application, the spatial transformer network takes the image of the target dress and the dress segmentation image output by the above-mentioned deformation segmentation unit as input, and obtains the dress transformation image through adaptive transformation and alignment.

[0139] In the embodiment of the present application, the correction network performs transformation correction on the clothing transformation image output by the spatial transformation network to obtain the final clothing transformation image. Figure 7 As shown, the correction network includes:

[0140] The second feature encoding network includes multiple convolutional layers and multiple downsampling layers, where the convolutional layers and the downsampling layers are connected in series and alternately arranged;

[0141] The third feature decoding network is connected in series with the second feature encoding network, and includes multiple convolutional layers and multiple upsampling layers, where the convolutional layers and the upsampling layers are connected in series and alternately arranged;

[0142] The input of the second feature encoding network includes: the clothing transformation image and clothing segmentation image output by the spatial transformation network;

[0143] The input of the tenth target convolutional layer of the third feature decoding network includes: the intermediate transformed image output by the last downsampling layer of the second feature encoding network;

[0144] The input of the eleventh target convolutional layer of the third feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the eleventh target convolutional layer and the intermediate transformed image output by the eleventh convolutional layer in the second feature encoding network;

[0145] The input of the twelfth target convolution layer of the third feature decoding network includes: the intermediate transformation image output by the previous upsampling layer adjacent to the twelfth target convolution layer and the intermediate transformation image output by the twelfth convolution layer in the second feature encoding network;

[0146] The input of the thirteenth target convolution layer of the third feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the thirteenth target convolution layer and the intermediate transformed image output by the thirteenth convolution layer in the second feature encoding network;

[0147] The input of the fourteenth target convolution layer of the third feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the fourteenth target convolution layer and the intermediate transformed image output by the fourteenth convolution layer in the second feature encoding network;

[0148] Among them, the position order of the tenth target convolution layer to the fourteenth target convolution layer in the third feature decoding network is consistent with the ascending order, and the position order of the eleventh convolution layer to the fourteenth convolution layer in the second feature encoding network is opposite to the ascending order.

[0149] In the embodiment of this application, Figure 7 For example, the downsampling layer after convolution 1 to convolution 4 in the correction network is the second feature encoding network, and convolution 5 to convolution 9 is the third feature decoding network. The tenth to fourteenth target convolution layers correspond to convolution 5 to convolution 9 in the third feature decoding network, respectively, and their position order is consistent with the increasing order; the eleventh to fourteenth convolution layers correspond to convolution 4, convolution 3, convolution 2, and convolution 1 in the second feature encoding network, respectively, and their position order is opposite to the increasing order.

[0150] Through the above-mentioned network structure setting of the clothing transformation unit, the correction network is used to further correct the clothing transformation image output by the spatial transformation network. In the correction network, the first encoded intermediate transformation image is input into the later convolutional layer for fusion. That is, the correction process is based on the features of each dimension, and ultimately a more accurate clothing transformation image is obtained.

[0151] Alternatively, as Figure 8 As shown, the test unit includes:

[0152] A third feature encoding network includes a plurality of convolutional layers and a plurality of downsampling layers, wherein the convolutional layers and the downsampling layers are connected in series and alternately arranged;

[0153] a fourth feature decoding network, connected in series with the third feature encoding network, comprising a plurality of convolutional layers and a plurality of upsampling layers, wherein the convolutional layers and the upsampling layers are connected in series and alternately arranged;

[0154] The input of the third feature encoding network includes: the clothing transformation image, the clothing segmentation image, and the target object's to-be-retained information, wherein the target object's to-be-retained information includes the target object's to-be-retained part obtained by segmenting the target object's image;

[0155] The input of the fifteenth target convolutional layer of the fourth feature decoding network includes: the intermediate encoded image output by the last downsampling layer of the third feature encoding network;

[0156] The input of the sixteenth target convolution layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the sixteenth target convolution layer and the intermediate encoded image output by the fifteenth convolution layer in the third feature encoding network;

[0157] The input of the seventeenth target convolutional layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the seventeenth target convolutional layer and the intermediate encoded image output by the sixteenth convolutional layer in the third feature encoding network;

[0158] The input of the eighteenth target convolution layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the eighteenth target convolution layer and the intermediate encoded image output by the seventeenth convolution layer in the third feature encoding network;

[0159] The input of the nineteenth target convolution layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the nineteenth target convolution layer and the intermediate encoded image output by the eighteenth convolution layer in the third feature encoding network;

[0160] Among them, the position order of the fifteenth target convolution layer to the nineteenth target convolution layer in the fourth feature decoding network is consistent with the ascending order, and the position order of the fifteenth convolution layer to the eighteenth convolution layer in the third feature encoding network is opposite to the ascending order.

[0161] In the embodiment of this application, Figure 8 For example, the test unit includes a third feature encoding network ( Figure 8 The downsampling layer after convolution 1 to convolution 4) and the fourth feature decoding network ( Figure 8 The fifteenth to nineteenth target convolution layers correspond to convolutions 5 to 9 in the fourth feature decoding network, respectively, and their position order is consistent with the ascending order; the fifteenth to eighteenth convolution layers correspond to convolutions 4, 3, 2, and 1 in the third feature encoding network, respectively, and their position order is opposite to the ascending order.

[0162] In this embodiment of the present application, the target object's to-be-retained information includes the target object's to-be-retained parts obtained by segmenting the target object's image. For example, if the target object is wearing a short-sleeved shirt, the to-be-retained parts include the head, left and right arms, left and right legs, and left and right feet. If the target object is wearing pants, the to-be-retained parts include the head, left and right arms, chest, waist, and left and right feet.

[0163] Through the above-mentioned network structure setting of the fitting unit, the intermediate encoded image is input into the later convolutional layer for fusion, that is, the features of each dimension are fully integrated, and finally a fitting image of the target object trying on the target clothing is obtained.

[0164] Based on the same technical concept, according to one aspect of the embodiment of the present application, an embodiment of a virtual fitting method is provided, such as Figure 9 As shown, the method may include the following steps:

[0165] Step S902: extracting the human features of the target object and the clothing features of the target clothing, fusing the human features and the clothing features, and generating a clothing segmentation image of the target clothing that matches the appearance of the target object based on multi-level correction of the fused features;

[0166] Step S904, transforming the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing;

[0167] Step S906 , after segmenting the target object according to the clothing segmentation image, the clothing transformation image is fitted to the segmented region to obtain a fitting image of the target object in the target clothing.

[0168] Through steps S902 to S906, the present application integrates the human body features of the target object and the clothing features of the target clothing, and directly predicts and outputs a deformed clothing segmentation image that matches the appearance of the target object, thereby reducing the accumulated prediction error of the related art of first predicting the human body instance segmentation image of the target object and then predicting the deformed clothing image segmentation map based on the human body instance segmentation image. Moreover, the deformed clothing image segmentation map is subjected to multi-level correction, further improving the accuracy of the fitting image, making the fitting effect more fitting and natural, and solving the technical problem of inaccurate virtual try-on results due to accumulated errors.

[0169] In step S902, extracting the human features of the target object and the clothing features of the target clothing, fusing the human features and the clothing features, and generating a clothing segmentation image of the target clothing that matches the appearance of the target object based on multi-level correction of the fused features includes:

[0170] Step 1, extracting clothing features of the target clothing using an image of the target clothing;

[0171] Step 2, using a preset segmentation map and a preset key point map of the target object to extract the human body features of the target object, wherein the preset segmentation map is an image obtained by separately identifying and segmenting each part of the target object in advance, and the preset key point map is used to mark the key points of the human body;

[0172] Step 3: fuse clothing features and human body features, and upsample and decode the fused features throughout the fusion process to obtain the basic clothing segmentation image;

[0173] The basic clothing segmentation image is subjected to multi-level correction to obtain the final clothing segmentation image.

[0174] In step S904, the target clothing is transformed according to the clothing segmentation image, and the clothing transformation image of the target clothing is obtained, including:

[0175] Obtain an image of the target's clothing;

[0176] Transform the target clothing image according to the clothing segmentation image to obtain a clothing transformation image;

[0177] Perform transformation correction on the clothing transformation image to obtain the final clothing transformation image.

[0178] In step S906, after the target object is segmented according to the clothing segmentation image, the human body information to be retained is determined, and the clothing transformation image is fitted to the segmented area to be spliced ​​with the above-mentioned human body information to be retained to obtain a fitting image of the target object in the target clothing.

[0179] According to another aspect of the embodiment of the present application, the present application provides an electronic device, such as Figure 10 As shown, it includes a memory 1001, a processor 1003, a communication interface 1005 and a communication bus 1007. The memory 1001 stores a computer program that can be run on the processor 1003. The memory 1001 and the processor 1003 communicate through the communication interface 1005 and the communication bus 1007. When the processor 1003 executes the computer program, the steps of the above method are implemented.

[0180] The memory and processor in the electronic device communicate via a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus may be divided into an address bus, a data bus, a control bus, and the like.

[0181] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0182] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0183] According to another aspect of the embodiments of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of any of the above embodiments.

[0184] Optionally, in an embodiment of the present application, the computer-readable medium is configured to store program codes for the processor to execute the following steps:

[0185] Extracting the human features of the target object and the clothing features of the target clothing, fusing the human features and the clothing features, and generating a clothing segmentation image of the target clothing that matches the appearance of the target object based on multi-level correction of the fused features;

[0186] Transform the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing;

[0187] After the target object is segmented according to the clothing segmentation image, the clothing transformation image is fitted to the segmented region to obtain a fitting image of the target object in the target clothing.

[0188] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0189] When implementing the embodiments of the present application, reference may be made to the above embodiments, which have corresponding technical effects.

[0190] It is understood that the embodiments described herein may be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or a combination thereof.

[0191] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0192] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0193] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0194] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0195] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0196] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0197] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application are essentially or partly contributed to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard drive, a ROM, a RAM, a magnetic disk, or an optical disk. It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0198] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

Claims

1. A virtual fitting system, characterized in that: The virtual fitting system is applied to a video platform, allowing viewers to use the clothes of characters in a play to try on virtual clothes, and / or allowing viewers to change the clothes of characters in the play. The system includes: a deformation segmentation unit, configured to extract the human features of the target object and the clothing features of the target clothing, fuse the human features and the clothing features, and generate a clothing segmentation image of the target clothing that matches the appearance of the target object based on multi-level correction of the fused features; a clothing transformation unit, configured to transform the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing; A fitting unit is configured to segment the target object according to the clothing segmentation image, fit the clothing transformation image to the segmented area, and obtain a fitting image of the target object in the target clothing.

2. The system according to claim 1, wherein: The deformation segmentation unit includes: a first feature extraction network, configured to extract the clothing features of the target clothing using the image of the target clothing; a second feature extraction network, configured to extract the body features of the target object using a preset segmentation map and a preset key point map of the target object, wherein the preset segmentation map is an image obtained by pre-identifying and segmenting each part of the target object one by one, and the preset key point map is used to mark key points of the body; a first feature decoding network, configured to fuse the clothing features output by each convolutional layer of the first feature extraction network with the human body features output by each convolutional layer of the second feature extraction network, and upsample and decode the fused features throughout the fusion process to obtain a basic clothing segmentation image; A multi-level correction network is used to perform multi-level correction on the basic clothing segmentation image to obtain the final clothing segmentation image.

3. The system according to claim 2, characterized in that The multi-stage correction network includes a plurality of sub-correction networks with the same structure and arranged in series; wherein, The input of the first sub-correction network includes: the basic clothing segmentation image; The input of each of the remaining sub-correction networks includes: the basic clothing segmentation image and the intermediate corrected image output by the previous sub-correction network; The intermediate corrected image output by the last sub-correction network is the final clothing segmentation image.

4. The system according to claim 3, characterized in that The sub-correction network includes: A first feature encoding network includes a plurality of convolutional layers and a plurality of downsampling layers, wherein the convolutional layers and the downsampling layers are connected in series and alternately arranged; A second feature decoding network is connected in series with the first feature encoding network, comprising a plurality of convolutional layers and a plurality of upsampling layers, wherein the convolutional layers and the upsampling layers are connected in series and alternately arranged; The input of the first feature encoding network includes: the basic clothing transformation image; or the basic clothing transformation image and the intermediate correction image output by the previous sub-correction network; The input of the first target convolutional layer of the second feature decoding network includes: the intermediate corrected image output by the last downsampling layer of the first feature encoding network; The input of the second target convolutional layer of the second feature decoding network includes: the intermediate corrected image output by the previous upsampling layer adjacent to the second target convolutional layer and the intermediate corrected image output by the first convolutional layer in the first feature encoding network; The input of the third target convolution layer of the second feature decoding network includes: the intermediate corrected image output by the previous upsampling layer adjacent to the third target convolution layer and the intermediate corrected image output by the second convolution layer in the first feature encoding network; The input of the fourth target convolutional layer of the second feature decoding network includes: the intermediate corrected image output by the previous upsampling layer adjacent to the fourth target convolutional layer and the intermediate corrected image output by the third convolutional layer in the first feature encoding network; The input of the fifth target convolutional layer of the second feature decoding network includes: the intermediate corrected image output by the previous upsampling layer adjacent to the fifth target convolutional layer and the intermediate corrected image output by the fourth convolutional layer in the first feature encoding network; Among them, the position order of the first target convolution layer to the fifth target convolution layer in the second feature decoding network is consistent with the ascending order, and the position order of the first convolution layer to the fourth convolution layer in the first feature encoding network is opposite to the ascending order.

5. The system according to claim 2, wherein: The first feature extraction network includes a plurality of convolutional layers and a plurality of downsampling layers, wherein the convolutional layers and the downsampling layers are connected in series and alternately arranged, the convolutional layers are used to extract the clothing features of the target clothing, and the downsampling layers are used to downsample and encode the clothing features output by the convolutional layers; The second feature extraction network includes multiple convolutional layers and multiple downsampling layers, which are connected in series and alternately arranged. The convolutional layers are used to extract the human body features of the target object, and the downsampling layers are used to downsample and encode the human body features output by the convolutional layers.

6. The system according to claim 2, wherein: The first feature decoding network includes multiple convolutional layers and multiple upsampling layers, the convolutional layers and the upsampling layers are connected in series and alternately arranged, and the upsampling layers are used to upsample and decode the fused features; The input of the sixth target convolutional layer of the first feature decoding network includes: the clothing features output by the last downsampling layer in the first feature extraction network and the human body features output by the last downsampling layer in the second feature extraction network; The input of the seventh target convolutional layer of the first feature decoding network includes: the fusion feature output by the previous upsampling layer adjacent to the seventh target convolutional layer, the clothing feature output by the fifth convolutional layer in the first feature extraction network, and the human body feature output by the sixth convolutional layer in the second feature extraction network; The input of the eighth target convolutional layer of the first feature decoding network includes: the fusion feature output by the previous upsampling layer adjacent to the eighth target convolutional layer, the clothing feature output by the seventh convolutional layer in the first feature extraction network, and the human body feature output by the eighth convolutional layer in the second feature extraction network; The input of the ninth target convolutional layer of the first feature decoding network includes: the fusion feature output by the previous upsampling layer adjacent to the ninth target convolutional layer, the clothing feature output by the ninth convolutional layer in the first feature extraction network, and the human body feature output by the tenth convolutional layer in the second feature extraction network; Among them, the position order of the sixth target convolution layer to the ninth target convolution layer in the first feature decoding network is consistent with the ascending order, the position order of the fifth convolution layer, the seventh convolution layer and the ninth convolution layer in the first feature extraction network is consistent with the ascending order, and the position order of the sixth convolution layer, the eighth convolution layer and the tenth convolution layer in the second feature extraction network is consistent with the ascending order.

7. The system according to claim 1, wherein: The clothing transformation unit includes: a spatial transformation network, configured to take the target clothing image and the clothing segmentation image as input, transform the target clothing according to the clothing segmentation image, and obtain the clothing transformation image; The correction network is used to perform transformation correction on the clothing transformation image output by the spatial transformation network to obtain a final clothing transformation image.

8. The system according to claim 7, characterized in that The correction network includes: A second feature encoding network includes a plurality of convolutional layers and a plurality of downsampling layers, wherein the convolutional layers and the downsampling layers are connected in series and alternately arranged; A third feature decoding network is connected in series with the second feature encoding network, comprising a plurality of convolutional layers and a plurality of upsampling layers, wherein the convolutional layers and the upsampling layers are connected in series and alternately arranged; The input of the second feature encoding network includes: the clothing transformation image and the clothing segmentation image output by the spatial transformation network; The input of the tenth target convolutional layer of the third feature decoding network includes: the intermediate transformed image output by the last downsampling layer of the second feature encoding network; The input of the eleventh target convolutional layer of the third feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the eleventh target convolutional layer and the intermediate transformed image output by the eleventh convolutional layer in the second feature encoding network; The input of the twelfth target convolution layer of the third feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the twelfth target convolution layer and the intermediate transformed image output by the twelfth convolution layer in the second feature encoding network; The input of the thirteenth target convolutional layer of the third feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the thirteenth target convolutional layer and the intermediate transformed image output by the thirteenth convolutional layer in the second feature encoding network; The input of the fourteenth target convolutional layer of the third feature decoding network includes: the intermediate transformed image output by the previous upsampling layer adjacent to the fourteenth target convolutional layer and the intermediate transformed image output by the fourteenth convolutional layer in the second feature encoding network; Among them, the position order of the tenth target convolution layer to the fourteenth target convolution layer in the third feature decoding network is consistent with the ascending order, and the position order of the eleventh convolution layer to the fourteenth convolution layer in the second feature encoding network is opposite to the ascending order.

9. The system according to claim 1, wherein: The trial installation unit comprises: A third feature encoding network includes a plurality of convolutional layers and a plurality of downsampling layers, wherein the convolutional layers and the downsampling layers are connected in series and alternately arranged; a fourth feature decoding network, connected in series with the third feature encoding network, comprising a plurality of convolutional layers and a plurality of upsampling layers, wherein the convolutional layers and the upsampling layers are connected in series and alternately arranged; The input of the third feature encoding network includes: the clothing transformation image, the clothing segmentation image, and the target object's information to be retained, wherein the target object's information to be retained includes the target object's part to be retained obtained by segmenting the target object's image; The input of the fifteenth target convolutional layer of the fourth feature decoding network includes: the intermediate coded image output by the last downsampling layer of the third feature encoding network; The input of the sixteenth target convolution layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the sixteenth target convolution layer and the intermediate coded image output by the fifteenth convolution layer in the third feature encoding network; The input of the seventeenth target convolutional layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the seventeenth target convolutional layer and the intermediate coded image output by the sixteenth convolutional layer in the third feature encoding network; The input of the eighteenth target convolutional layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the eighteenth target convolutional layer and the intermediate coded image output by the seventeenth convolutional layer in the third feature encoding network; The input of the nineteenth target convolutional layer of the fourth feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the nineteenth target convolutional layer and the intermediate coded image output by the eighteenth convolutional layer in the third feature encoding network; Among them, the position order of the fifteenth target convolution layer to the nineteenth target convolution layer in the fourth feature decoding network is consistent with the ascending order, and the position order of the fifteenth convolution layer to the eighteenth convolution layer in the third feature encoding network is opposite to the ascending order.

10. A virtual fitting method, characterized in that: A virtual fitting system according to any one of claims 1 to 9, comprising: extracting the human features of the target object and the clothing features of the target clothing, fusing the human features and the clothing features, and generating a clothing segmentation image of the target clothing that matches the appearance of the target object based on multi-level correction of the fused features; Transforming the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing; After the target object is segmented according to the clothing segmentation image, the clothing transformation image is fitted to the segmented region to obtain a fitting image of the target object in the target clothing.

11. The method according to claim 10, characterized in that Extracting the human body features of the target object and the clothing features of the target clothing, fusing the human body features and the clothing features, and generating a clothing segmentation image of the target clothing that matches the appearance of the target object based on the fused features includes: Extracting the clothing features of the target clothing using the image of the target clothing; Extracting the human body features of the target object using a preset segmentation map and a preset key point map of the target object, wherein the preset segmentation map is an image obtained by pre-identifying and segmenting each part of the target object one by one, and the preset key point map is used to mark key points of the human body; fusing the clothing features and the human body features, and upsampling and decoding the fused features throughout the fusion process to obtain a basic clothing segmentation image; Perform multi-level correction on the basic clothing segmentation image to obtain the final clothing segmentation image.

12. The method according to claim 10, characterized in that Transforming the target clothing according to the clothing segmentation image to obtain the clothing transformation image of the target clothing includes: Acquire an image of the target's attire; transforming the target clothing image according to the clothing segmentation image to obtain the clothing transformation image; Performing transformation correction on the clothing transformation image to obtain a final clothing transformation image.

13. An electronic device comprising a memory, a processor, a communication interface, and a communication bus, wherein the memory stores a computer program that can be run on the processor, and the memory and the processor communicate via the communication bus and the communication interface, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 10 to 12 are implemented.

14. A computer-readable medium having a non-volatile program code executable by a processor, characterized in that The program code enables the processor to execute the method according to any one of claims 10 to 12.

Images