Virtual trial installation system, method, device and computer readable medium

By using a multi-scale supervised deformation segmentation unit in virtual trial installation technology, the image of the target object wearing the target dress is directly predicted, which solves the problem of inaccurate virtual trial installation results and improves image accuracy and trial installation effect.

CN114881846BActive Publication Date: 2024-11-26BEIJING QIYI CENTURY SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210605215.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-30
Publication Date
2024-11-26
Estimated Expiration
2042-05-30

AI Technical Summary

Technical Problem

In the existing virtual trial installation technology, the error in human instance segmentation causes accumulation, resulting in inaccurate results of virtual trial installation and lack of effective solutions.

Method used

The human characteristics of the target object and the dress characteristics of the target dress are extracted through the deformation segmentation unit, and multiple loss functions are used for multi-scale supervision to generate the dress segmented image of the target dress that matches the appearance of the target object, and directly predict the deformed dress segmented image , reduce prediction error.

Benefits of technology

The accuracy of the virtual trial-on image is improved, making the trial-on effect more integrated and natural, and solving the problem of inaccurate virtual trial-on results caused by cumulative errors.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114881846B_ABST
    Figure CN114881846B_ABST
Patent Text Reader

Abstract

The present application relates to a virtual fitting system, method, device and computer-readable medium. The system includes: a deformation segmentation unit, which is used to extract the human features of the target object and the dressing features of the target clothing, and after fusing the human features and the dressing features, use multiple loss functions to perform multi-scale supervision on the decoding process of the fused features to generate a clothing segmentation image of the target clothing that matches the appearance of the target object; a clothing transformation unit, which is used to transform the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing; a fitting unit, which is used to segment the target object according to the clothing segmentation image, fit the clothing transformation image to the segmentation area, and obtain a fitting image of the target object for the target clothing. The present application performs multi-scale supervision on the intermediate decoding process of the clothing segmentation image to ensure that valid information is transmitted during the decoding process, thereby solving the technical problem of inaccurate virtual fitting results due to cumulative errors.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of multimedia intelligent processing technology, and in particular to a virtual trial installation system, method, device and computer-readable medium. Background Art

[0002] Virtual Try-on means: given a target clothing image and a model image, a photo of a model wearing the target clothing image is generated, so as to achieve the purpose of the model virtually trying on the target clothing. The try-on result must meet the following four requirements: 1. Keep the posture and body shape of the human body in the image unchanged; 2. The target clothing is adapted to the posture and body shape of the human body; 3. Keep other information except the target clothing, such as the lower garment; 4. Keep the creases, patterns, and logos of the target after wearing clear and natural. In the field of e-commerce, good virtual try-on technology can not only provide consumers with a novel interactive experience, but also guide and stimulate consumers to make purchase decisions faster. In the field of video playback platforms, virtual try-on technology can bring a new viewing experience to viewers. For example, while watching a movie, viewers can upload their own photos and virtually wear the clothes of the characters in the play, bringing an immersive viewing experience, or they can switch the clothes of the characters in the play at will, increasing the interactivity and fun of watching movies, and also invisibly allowing users to stay on the video playback platform for a longer time.

[0003] At present, in the related technology, the virtual fitting is usually carried out through the ACGPN algorithm. The ACGPN algorithm contains a total of 4 modules: module A, module B, module C and module D. The A module is used to generate a human body instance segmentation map, parse the input human body image, and mark the limbs of the person in the image with different numbers, such as filling the left arm area with 0, the right arm area with 1, the body torso with 2, etc.; the B module is used to generate a deformed clothing segmentation map, which is a binary image, and the deformed clothing area in the image is filled with 1, and the rest of the positions are filled with 0; the C module contains two parts: the spatial transformer network STN (Spatial Transformer Network) and the correction network, wherein the STN outputs a rough transformed clothing image, and the correction network corrects the output of the STN and outputs the final transformed clothing image; the D module combines the outputs of the above modules to generate the final virtual fitting image. However, in deep learning tasks, the division of human instances includes the prediction of many detailed areas such as hands, feet, hair, face, torso, etc. It is very difficult to obtain very precise prediction results. Therefore, the error of module A in the ACGPN algorithm further leads to the error of module B. As a result, the errors of modules A and B are accumulated to modules C and D, resulting in inaccurate virtual try-on results in the final output.

[0004] Currently, no effective solution has been proposed to the problem of inaccurate virtual try-on results due to cumulative errors. Summary of the invention

[0005] The present application provides a virtual fitting system, method, device and computer-readable medium to solve the technical problem of inaccurate virtual fitting results due to cumulative errors.

[0006] According to one aspect of an embodiment of the present application, the present application provides a virtual fitting system, including:

[0007] a deformation segmentation unit, for extracting the human features of the target object and the clothing features of the target clothing, and after fusing the human features and the clothing features, using multiple loss functions to perform multi-scale supervision on the decoding process of the fused features, so as to generate a clothing segmentation image of the target clothing that matches the appearance of the target object, wherein different loss functions correspond to different scales, and the multiple loss functions are used to gradually improve the matching degree between the intermediate segmentation image of the target clothing obtained in the decoding process of the fused features and the appearance of the target object;

[0008] a clothing transformation unit, used for transforming the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing;

[0009] The fitting unit is used to segment the target object according to the clothing segmentation image, fit the clothing transformation image to the segmentation area, and obtain a fitting image of the target object with respect to the target clothing.

[0010] Optionally, the deformation segmentation unit includes:

[0011] A first feature extraction network is used to extract clothing features of the target clothing using the image of the target clothing;

[0012] A second feature extraction network is used to extract the human body features of the target object by using a preset segmentation map and a preset key point map of the target object, wherein the preset segmentation map is an image obtained by separately identifying and segmenting each part of the target object in advance, and the preset key point map is used to mark the key points of the human body;

[0013] A first feature decoding network is used to fuse the clothing features output by each convolution layer of the first feature extraction network and the human body features output by each convolution layer of the second feature extraction network, and upsample and decode the fused features throughout the fusion process to obtain a clothing segmentation image, wherein the intermediate segmentation image is generated in the process of upsampling and decoding the fused features, and the clothing segmentation image is the last intermediate segmentation image;

[0014] The loss supervision network is used to calculate the loss values ​​of the intermediate segmentation images decoded at different scales and the true value segmentation images of the corresponding scales using multiple loss functions, and use the loss values ​​to adjust the parameters of the first feature decoding network until the loss value of the intermediate segmentation image and the true value segmentation image of the corresponding scale is less than the target loss threshold.

[0015] Optionally, the first feature decoding network includes a plurality of feature decoding units connected in series, each feature decoding unit includes two convolutional layers and an upsampling layer, the two convolutional layers are arranged one before and one after the upsampling layer, the upsampling layer is used to upsample and decode the fused features, and the convolutional layer arranged after the upsampling layer is used to output an intermediate segmented image corresponding to the current scale according to the features decoded by the upsampling layer;

[0016] The input of the first target convolutional layer of the first feature decoding network includes: the clothing features output by the last downsampling layer in the first feature extraction network and the human body features output by the last downsampling layer in the second feature extraction network;

[0017] The input of the second target convolution layer of the first feature decoding network includes: the intermediate segmented image output by the previous feature decoding unit adjacent to the second target convolution layer, the clothing features output by the first convolution layer in the first feature extraction network, and the human body features output by the second convolution layer in the second feature extraction network;

[0018] The input of the third target convolution layer of the first feature decoding network includes: the intermediate segmentation image output by the previous feature decoding unit adjacent to the third target convolution layer, the clothing features output by the third convolution layer in the first feature extraction network, and the human body features output by the fourth convolution layer in the second feature extraction network;

[0019] The input of the fourth target convolution layer of the first feature decoding network includes: the intermediate segmented image output by the previous feature decoding unit adjacent to the fourth target convolution layer, the clothing features output by the fifth convolution layer in the first feature extraction network, and the human body features output by the sixth convolution layer in the second feature extraction network;

[0020] Among them, the position order of the first target convolution layer to the fourth target convolution layer in the first feature decoding network is consistent with the ascending order, the position order of the first convolution layer, the third convolution layer and the fifth convolution layer in the first feature extraction network is consistent with the ascending order, and the position order of the second convolution layer, the fourth convolution layer and the sixth convolution layer in the second feature extraction network is consistent with the ascending order.

[0021] Optionally, the loss supervision network includes:

[0022] A first loss supervision unit is used to constrain an intermediate segmented image output by a previous feature decoding unit adjacent to the second target convolution layer through a first loss function, wherein the first loss function matches a segmentation scale of a current feature decoding unit;

[0023] A second loss supervision unit is used to constrain an intermediate segmented image output by a previous feature decoding unit adjacent to the third target convolutional layer through a second loss function, wherein the second loss function matches a segmentation scale of a current feature decoding unit;

[0024] A third loss supervision unit is used to constrain an intermediate segmented image output by a previous feature decoding unit adjacent to a fourth target convolutional layer through a third loss function, wherein the third loss function matches a segmentation scale of a current feature decoding unit.

[0025] Optionally, the first feature extraction network includes multiple convolutional layers and multiple downsampling layers, the convolutional layers and the downsampling layers are connected in series and alternately arranged, the convolutional layers are used to extract clothing features of the target clothing, and the downsampling layers are used to downsample and encode the clothing features output by the convolutional layers.

[0026] Optionally, the second feature extraction network includes multiple convolutional layers and multiple downsampling layers, the convolutional layers and the downsampling layers are connected in series and alternately arranged, the convolutional layers are used to extract human body features of the target object, and the downsampling layers are used to downsample and encode the human body features output by the convolutional layers.

[0027] Optionally, the clothing transformation unit includes:

[0028] A spatial transformation network is used to take the target clothing image and the clothing segmentation image as input, transform the target clothing according to the clothing segmentation image, and obtain the clothing transformation image;

[0029] The correction network is used to perform transformation correction on the clothing transformation image output by the spatial transformation network to obtain the final clothing transformation image.

[0030] Optionally, the correction network comprises:

[0031] A first feature encoding network includes a plurality of convolutional layers and a plurality of downsampling layers, wherein the convolutional layers and the downsampling layers are connected in series and arranged alternately;

[0032] A second feature decoding network is connected in series with the first feature encoding network, and includes a plurality of convolutional layers and a plurality of upsampling layers, wherein the convolutional layers and the upsampling layers are connected in series and arranged alternately;

[0033] The input of the first feature encoding network includes: the clothing transformation image and the clothing segmentation image output by the spatial transformation network;

[0034] The input of the fifth target convolutional layer of the second feature decoding network includes: the intermediate transformed image output by the last downsampling layer of the first feature encoding network;

[0035] The input of the sixth target convolution layer of the second feature decoding network includes: the intermediate transformation image output by the previous upsampling layer adjacent to the sixth target convolution layer and the intermediate transformation image output by the seventh convolution layer in the first feature encoding network;

[0036] The input of the seventh target convolution layer of the second feature decoding network includes: the intermediate transformation image output by the previous upsampling layer adjacent to the seventh target convolution layer and the intermediate transformation image output by the eighth convolution layer in the first feature encoding network;

[0037] The input of the eighth target convolution layer of the second feature decoding network includes: the intermediate transformation image output by the previous upsampling layer adjacent to the eighth target convolution layer and the intermediate transformation image output by the ninth convolution layer in the first feature encoding network;

[0038] The input of the ninth target convolution layer of the second feature decoding network includes: the intermediate transformation image output by the previous upsampling layer adjacent to the ninth target convolution layer and the intermediate transformation image output by the tenth convolution layer in the first feature encoding network;

[0039] Among them, the position order of the fifth target convolution layer to the ninth target convolution layer in the second feature decoding network is consistent with the ascending order, and the position order of the seventh convolution layer to the tenth convolution layer in the first feature encoding network is opposite to the ascending order.

[0040] Optionally, the test assembly unit comprises:

[0041] The second feature encoding network includes a plurality of convolutional layers and a plurality of downsampling layers, wherein the convolutional layers and the downsampling layers are connected in series and arranged alternately;

[0042] A third feature decoding network is connected in series with the second feature encoding network, and includes a plurality of convolutional layers and a plurality of upsampling layers, wherein the convolutional layers and the upsampling layers are connected in series and arranged alternately;

[0043] The input of the second feature encoding network includes: the clothing transformation image, the clothing segmentation image and the target object's information to be retained, wherein the target object's information to be retained includes the target object's part to be retained obtained by segmenting the target object's image;

[0044] The input of the tenth target convolutional layer of the third feature decoding network includes: the intermediate encoded image output by the last downsampling layer of the second feature encoding network;

[0045] The input of the eleventh target convolution layer of the third feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the eleventh target convolution layer and the intermediate encoded image output by the eleventh convolution layer in the second feature encoding network;

[0046] The input of the twelfth target convolution layer of the third feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the twelfth target convolution layer and the intermediate encoded image output by the twelfth convolution layer in the second feature encoding network;

[0047] The input of the thirteenth target convolution layer of the third feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the thirteenth target convolution layer and the intermediate encoded image output by the thirteenth convolution layer in the second feature encoding network;

[0048] The input of the fourteenth target convolution layer of the third feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the fourteenth target convolution layer and the intermediate encoded image output by the fourteenth convolution layer in the second feature encoding network;

[0049] Among them, the position order of the tenth target convolution layer to the fourteenth target convolution layer in the third feature decoding network is consistent with the ascending order, and the position order of the eleventh convolution layer to the fourteenth convolution layer in the second feature encoding network is opposite to the ascending order.

[0050] According to another aspect of the embodiment of the present application, the present application provides a virtual fitting method, comprising:

[0051] Extracting the human features of the target object and the clothing features of the target clothing, and fusing the human features and the clothing features, using multiple loss functions to perform multi-scale supervision on the decoding process of the fused features, so as to generate a clothing segmentation image of the target clothing that matches the appearance of the target object, wherein different loss functions correspond to different scales, and the multiple loss functions are used to gradually improve the matching degree between the intermediate segmentation image of the target clothing obtained in the decoding process of the fused features and the appearance of the target object;

[0052] Transform the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing;

[0053] After the target object is segmented according to the clothing segmentation image, the clothing transformation image is fitted to the segmented region to obtain a fitting image of the target object in the target clothing.

[0054] Optionally, extracting the human features of the target object and the clothing features of the target clothing, and fusing the human features and the clothing features, and using multiple loss functions to perform multi-scale supervision on the decoding process of the fused features to generate a clothing segmentation image of the target clothing that matches the appearance of the target object, including:

[0055] Extracting clothing features of the target clothing using the image of the target clothing;

[0056] The preset segmentation map and the preset key point map of the target object are used to extract the human body features of the target object, wherein the preset segmentation map is an image obtained by separately identifying and segmenting each part of the target object in advance, and the preset key point map is used to mark the key points of the human body;

[0057] The clothing features and the human body features are fused, and the fused features are upsampled and decoded throughout the fusion process to obtain a clothing segmentation image, wherein the intermediate segmentation image is generated in the process of upsampling and decoding the fused features, and the clothing segmentation image is the last intermediate segmentation image;

[0058] Among them, the decoding process also includes: using multiple loss functions to calculate the loss values ​​of the intermediate segmentation image decoded at different scales and the true value segmentation image of the corresponding scale, and using the loss value to adjust the parameters of the first feature decoding network until the loss value of the intermediate segmentation image and the true value segmentation image of the corresponding scale is less than the target loss threshold.

[0059] Optionally, transforming the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing includes:

[0060] Obtain an image of the target's clothing;

[0061] The target clothing image is transformed according to the clothing segmentation image to obtain a clothing transformation image;

[0062] The clothing transformation image is transformed and corrected to obtain the final clothing transformation image.

[0063] According to another aspect of an embodiment of the present application, the present application provides an electronic device, including a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, the memory and the processor communicate through the communication bus and the communication interface, and the steps of the above method are implemented when the processor executes the computer program.

[0064] According to another aspect of an embodiment of the present application, the present application also provides a computer-readable medium having a non-volatile program code executable by a processor, and the program code enables the processor to execute the above method.

[0065] The above technical solution provided by the embodiment of the present application has the following advantages compared with the related art:

[0066] The technical solution of the present application provides a virtual fitting system, including: a deformation segmentation unit, which is used to extract the human body features of the target object and the dressing features of the target clothing, and after fusing the human body features and the dressing features, generate a clothing segmentation image of the target clothing that matches the appearance of the target object based on the fusion features; a clothing transformation unit, which is used to transform the target clothing according to the clothing segmentation image to obtain the clothing transformation image of the target clothing; a fitting unit, which is used to segment the target object according to the clothing segmentation image, fit the clothing transformation image to the segmentation area, and obtain the fitting image of the target object to the target clothing. The present application fuses the human body features of the target object and the dressing features of the target clothing, directly predicts and outputs a deformed clothing segmentation image that matches the appearance of the target object, reduces the accumulated prediction error of first predicting the human body instance segmentation image of the target object in the related art, and then predicting the deformed clothing image segmentation map based on the human body instance segmentation image, further improves the accuracy of the fitting image, makes the fitting effect more fitting and more natural, and solves the technical problem of inaccurate virtual fitting results due to accumulated errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0069] Figure 1a A schematic diagram of a human body instance segmentation map predicted and output by module A of the ACGPN algorithm in the related art;

[0070] Figure 1b It is a schematic diagram of the segmentation diagram of the deformed clothing image predicted and output by the B module of the ACGPN algorithm in the related art;

[0071] Figure 1c A schematic diagram of a deformed clothing image predicted and output by the C module of the ACGPN algorithm in the related art;

[0072] Figure 1d It is a schematic diagram of the test installation result predicted by the D module of the ACGPN algorithm in the related art;

[0073] Figure 2 A schematic diagram of an optional virtual trial installation system structure provided according to an embodiment of the present application;

[0074] Figure 3A schematic diagram of a feature extraction network structure of an optional deformation segmentation unit provided according to an embodiment of the present application;

[0075] Figure 4 A schematic diagram of a feature decoding network structure of an optional deformation segmentation unit provided according to an embodiment of the present application;

[0076] Figure 5 A schematic diagram of a correction network structure of an optional clothing transformation unit provided according to an embodiment of the present application;

[0077] Figure 6 A schematic diagram of a network structure of an optional trial installation unit provided according to an embodiment of the present application;

[0078] Figure 7 A schematic diagram of an optional virtual trial installation method flow chart provided according to an embodiment of the present application;

[0079] Figure 8 A schematic diagram of an optional electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0081] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0082] In the related art, the virtual fitting is usually performed through the ACGPN algorithm. The ACGPN algorithm consists of 4 modules: module A, module B, module C and module D. Module A is used to generate a human body instance segmentation map, analyze the input human body image, and mark the limbs of the person in the image with different numbers, such as filling the left arm area with 0, the right arm area with 1, the body torso with 2, etc. The prediction result of the human body instance segmentation map is as follows: Figure 1a As shown in Figure 2; Module B is used to generate a segmentation map of the deformed clothing. The segmentation map is a binary image. The deformed clothing area in the image is filled with 1, and the rest of the positions are filled with 0. The prediction result of the deformed clothing segmentation map is shown in Figure 2. Figure 1bAs shown in Figure 2, the C module consists of two parts: the spatial transformer network STN (SpatialTransformer Network) and the correction network. The STN outputs a rough transformed clothing image, and the correction network corrects the output of the STN and outputs the final transformed clothing image. The prediction result of the transformed clothing image is shown in Figure 2. Figure 1c As shown; Module D combines the outputs of the previous modules to generate the final virtual try-on image. Figure 1d However, in deep learning tasks, the division of human instances includes the prediction of many detailed areas such as hands, feet, hair, face, torso, etc. It is very difficult to obtain very precise prediction results. Therefore, the error of module A in the ACGPN algorithm further leads to the error of module B. As a result, the errors of modules A and B are accumulated to modules C and D, resulting in inaccurate virtual try-on results.

[0083] In order to solve the technical problems mentioned in the background technology, according to one aspect of the embodiments of the present application, the present application provides a virtual fitting system, such as Figure 2 As shown, including:

[0084] The deformation segmentation unit 201 is used to extract the human features of the target object and the clothing features of the target clothing, and after fusing the human features and the clothing features, use multiple loss functions to perform multi-scale supervision on the decoding process of the fused features to generate a clothing segmentation image of the target clothing that matches the appearance of the target object, wherein different loss functions correspond to different scales, and the multiple loss functions are used to gradually improve the matching degree between the intermediate segmentation image of the target clothing obtained in the decoding process of the fused features and the appearance of the target object;

[0085] A clothing transformation unit 203, configured to transform the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing;

[0086] The fitting unit 205 is used to segment the target object according to the clothing segmentation image, and then fit the clothing transformation image to the segmentation area to obtain a fitting image of the target object with respect to the target clothing.

[0087] Based on the deformation segmentation unit 201, the human body features of the target object and the clothing features of the target clothing are extracted and integrated, so that the neural network model predicts and outputs the clothing segmentation image of the target clothing that matches the appearance of the target object at one time, that is, the A module and the B module of the ACGPN algorithm in the related technology are merged, and the deformed clothing image segmentation map is directly predicted and output, and the prediction of the human body instance segmentation map is skipped, and the human body features and clothing features are integrated to predict the clothing segmentation image. This not only greatly simplifies the original scheme, reduces the number of modules of the original scheme, and enhances the integrity of the scheme, but also reduces the accumulation of prediction errors between multiple modules, and further improves the image accuracy in subsequent steps.

[0088] The technical solution of the present application also makes further improvements to the deformation segmentation unit, that is, the feature information output by the upsampling part used to decode the fusion feature in the deformation segmentation unit is supervised in the middle layer, so that each upsampling decoding part outputs the segmentation result at the corresponding scale, and then the shallow feature information of the feature encoding part is fused to realize the correction of the segmentation result from coarse to fine, and finally a refined clothing segmentation image is obtained.

[0089] The technical solution of this application can be applied to the field of e-commerce, where the target object can be a model of an e-commerce merchant, and the target clothing can be the clothing to be sold by the merchant. Based on this virtual fitting system, the merchant can batch generate fitting images of its own clothing to be sold by using only the model's modeling photos. The target object can also be a consumer, and the consumer can generate a fitting image of himself trying on the clothing to be purchased by uploading his own photos. Therefore, in the field of e-commerce, a good virtual fitting technology can not only provide consumers with a novel interactive experience, but also guide and stimulate consumers to make purchase decisions faster.

[0090] The technical solution of this application can also be applied to the field of video platforms, where the target clothing can be the clothing of virtual characters in film and television works, game works, and animation works, and the target object can be the audience. The audience can upload their own photos to try on the clothes of the characters in the play. Therefore, in the field of video platforms, virtual fitting technology can bring a new immersive viewing experience to viewers and increase the interaction between fans and characters in the play. Not only that, the target object can also be a character in the play, and the target clothing can be a specific clothing selected by the audience, so that the audience can switch the clothes of the characters in the play at will, increase the interactivity and fun of watching movies, and also virtually make users active and stay longer on the video platform.

[0091] Optionally, the deformation segmentation unit includes:

[0092] A first feature extraction network is used to extract clothing features of the target clothing using the image of the target clothing;

[0093] A second feature extraction network is used to extract the human body features of the target object by using a preset segmentation map and a preset key point map of the target object, wherein the preset segmentation map is an image obtained by separately identifying and segmenting each part of the target object in advance, and the preset key point map is used to mark the key points of the human body;

[0094] A first feature decoding network is used to fuse the clothing features output by each convolution layer of the first feature extraction network and the human body features output by each convolution layer of the second feature extraction network, and upsample and decode the fused features throughout the fusion process to obtain a clothing segmentation image, wherein the intermediate segmentation image is generated in the process of upsampling and decoding the fused features, and the clothing segmentation image is the last intermediate segmentation image;

[0095] The loss supervision network is used to calculate the loss values ​​of the intermediate segmentation images decoded at different scales and the true value segmentation images of the corresponding scales using multiple loss functions, and use the loss values ​​to adjust the parameters of the first feature decoding network until the loss value of the intermediate segmentation image and the true value segmentation image of the corresponding scale is less than the target loss threshold.

[0096] In the embodiment of the present application, the first feature extraction network takes the image of the target clothing as input and extracts the clothing features of the target clothing layer by layer through multiple convolutional layers.

[0097] Alternatively, if Figure 3 As shown in branch network 1 in the figure, the first feature extraction network includes multiple convolutional layers and multiple downsampling layers. The convolutional layers and the downsampling layers are connected in series and alternately arranged. The convolutional layers are used to extract the clothing features of the target clothing, and the downsampling layers are used to downsample and encode the clothing features output by the convolutional layers.

[0098] In the embodiment of the present application, the second feature extraction network takes the preset segmentation map and the preset key point map of the target object as input, and extracts the human body features of the target object layer by layer through multiple convolutional layers. The preset segmentation map of the target object can be obtained by processing the photo of the target object by a human body parsing algorithm, and the human body parsing algorithm is to separate the left hand, right hand, head, hair, torso, left leg, and right leg of the human body, such as RefineNet. The preset key point map is used to mark the key points of the human body.

[0099] Alternatively, if Figure 3 As shown in branch network 2 in the figure, the second feature extraction network includes multiple convolutional layers and multiple downsampling layers. The convolutional layers and the downsampling layers are connected in series and alternately arranged. The convolutional layers are used to extract the human features of the target object, and the downsampling layers are used to downsample and encode the human features output by the convolutional layers.

[0100] In the embodiment of the present application, Figure 3The “+” in the figure not only represents the fusion of the clothing features output by the last downsampling layer of the first feature extraction network and the human features output by the last downsampling layer of the second feature extraction network, but also represents the feature fusion of the outputs of each convolutional layer in the first feature extraction network and the second feature extraction network.

[0101] Alternatively, if Figure 4 As shown, the first feature decoding network includes a plurality of feature decoding units connected in series, each feature decoding unit includes two convolutional layers and an upsampling layer, the two convolutional layers are arranged one before and one after the upsampling layer, the upsampling layer is used to upsample and decode the fusion feature, and the convolutional layer arranged after the upsampling layer is used to output an intermediate segmented image corresponding to the current scale according to the features decoded by the upsampling layer;

[0102] The input of the first target convolutional layer of the first feature decoding network includes: the clothing features output by the last downsampling layer in the first feature extraction network and the human body features output by the last downsampling layer in the second feature extraction network;

[0103] The input of the second target convolution layer of the first feature decoding network includes: the intermediate segmented image output by the previous feature decoding unit adjacent to the second target convolution layer, the clothing features output by the first convolution layer in the first feature extraction network, and the human body features output by the second convolution layer in the second feature extraction network;

[0104] The input of the third target convolution layer of the first feature decoding network includes: the intermediate segmentation image output by the previous feature decoding unit adjacent to the third target convolution layer, the clothing features output by the third convolution layer in the first feature extraction network, and the human body features output by the fourth convolution layer in the second feature extraction network;

[0105] The input of the fourth target convolution layer of the first feature decoding network includes: the intermediate segmented image output by the previous feature decoding unit adjacent to the fourth target convolution layer, the clothing features output by the fifth convolution layer in the first feature extraction network, and the human body features output by the sixth convolution layer in the second feature extraction network;

[0106] Among them, the position order of the first target convolution layer to the fourth target convolution layer in the first feature decoding network is consistent with the ascending order, the position order of the first convolution layer, the third convolution layer and the fifth convolution layer in the first feature extraction network is consistent with the ascending order, and the position order of the second convolution layer, the fourth convolution layer and the sixth convolution layer in the second feature extraction network is consistent with the ascending order.

[0107] In the present application embodiment, Figure 3 , 4For example, the first target convolution layer to the fourth target convolution layer are convolution 6 to convolution 9 in the first feature decoding network, and their position order is consistent with the increasing order; the first convolution layer, the third convolution layer and the fifth convolution layer are Figure 3 The position order of convolution 2, convolution 3 and convolution 4 in branch network 1 (the first feature extraction network) is consistent with the ascending order; the second convolution layer, the fourth convolution layer and the sixth convolution layer are respectively Figure 3 The position order of convolution 2, convolution 3 and convolution 4 in branch network 2 (the second feature extraction network) is consistent with the ascending order.

[0108] Alternatively, if Figure 4 As shown, the loss supervision network includes:

[0109] A first loss supervision unit is used to constrain an intermediate segmented image output by a previous feature decoding unit adjacent to the second target convolution layer through a first loss function, wherein the first loss function matches a segmentation scale of a current feature decoding unit;

[0110] A second loss supervision unit is used to constrain an intermediate segmented image output by a previous feature decoding unit adjacent to the third target convolutional layer through a second loss function, wherein the second loss function matches a segmentation scale of a current feature decoding unit;

[0111] A third loss supervision unit is used to constrain an intermediate segmented image output by a previous feature decoding unit adjacent to a fourth target convolutional layer through a third loss function, wherein the third loss function matches a segmentation scale of a current feature decoding unit.

[0112] like Figure 4 As shown, each feature decoding unit includes two convolutional layers and one upsampling layer, and the two convolutional layers are arranged one before and one after the upsampling layer, such as the first target convolutional layer ( Figure 4 The convolution 6 in the image is as follows: upsampling, 1*1 convolution is the first feature decoding unit, the second target convolution layer, upsampling, 1*1 convolution is the second feature decoding unit, the third target convolution layer, upsampling, 1*1 convolution is the third feature decoding unit, and the fourth target convolution layer, upsampling, 1*1 convolution is the fourth feature decoding unit. The first loss supervision unit is connected to the 1*1 convolution of the first feature decoding unit to constrain the intermediate segmentation image output by the 1*1 convolution, the second loss supervision unit is connected to the 1*1 convolution of the second feature decoding unit to constrain the intermediate segmentation image output by the 1*1 convolution, and the third loss supervision unit is connected to the 1*1 convolution of the third feature decoding unit to constrain the intermediate segmentation image output by the 1*1 convolution.

[0113] Among them, the upsampling layer is used to upsample and decode the fused features, and the 1*1 convolution layer set after the upsampling layer is used to output the intermediate segmentation image corresponding to the current scale according to the features decoded by the upsampling layer. Specifically, the fused feature f is input into the first target convolution layer to further extract features, and then upsampled by 2 times. The output features are reduced in dimension through 1*1 convolution, and a 2-dimensional segmentation result probability map (i.e., the intermediate segmentation image) is output. During the training process, the result will be supervised by the segmentation loss alone to constrain the result to be as close as possible to the true value of the segmentation result of the corresponding scale, that is, the segmentation result at this scale is output. Since the feature has only passed through a 2-fold upsampling layer at this time, the resolution of the predicted segmentation result is lower than the original image size, and it can only be regarded as a rough segmentation result. After that, the segmentation result will be connected to the shallow features output by the convolution 2 of the first feature extraction network (branch network 1) and the second feature extraction network (branch network 2) through a channel. The intermediate segmentation image output by the feature decoding unit is used as a supplementary feature information at high resolution, so as to refine the output of the segmentation result of the previous layer in the subsequent feature extraction process. Similarly, after convolution 10, the model outputs a refined segmentation result map after multi-scale correction, that is, the final clothing segmentation image.

[0114] Through the above network structure setting of the deformation segmentation unit, the clothing features and human body features extracted at each stage of the feature extraction network are fused to avoid forgetting low-dimensional features after extracting high-dimensional features, thereby more comprehensively and accurately predicting the clothing segmentation image of the target clothing that matches the appearance of the target object. The clothing segmentation image represents the outline of the target object wearing the target clothing.

[0115] Optionally, the clothing transformation unit includes:

[0116] A spatial transformation network is used to take the target clothing image and the clothing segmentation image as input, transform the target clothing according to the clothing segmentation image, and obtain the clothing transformation image;

[0117] The correction network is used to perform transformation correction on the clothing transformation image output by the spatial transformation network to obtain the final clothing transformation image.

[0118] In an embodiment of the present application, the Spatial Transformer Network (STN) is composed of three parts: a localisation network, a grid generator and a sampler. For the input image, the spatial transformer network first uses a localisation network to predict the required transformation, that is, after several consecutive layers of calculations (including convolution and fully connected calculations), the grid generator and the sampler then transform the image. The spatial transformer network can adaptively perform spatial transformation and alignment on the data. In an embodiment of the present application, the spatial transformer network takes the image of the target dress and the dress segmentation image output by the above-mentioned deformation segmentation unit as input, and obtains the dress transformation image through adaptive transformation and alignment.

[0119] In the embodiment of the present application, the correction network performs transformation correction on the clothing transformation image output by the spatial transformation network to obtain the final clothing transformation image. Figure 5 As shown, the correction network includes:

[0120] A first feature encoding network includes a plurality of convolutional layers and a plurality of downsampling layers, wherein the convolutional layers and the downsampling layers are connected in series and arranged alternately;

[0121] A second feature decoding network is connected in series with the first feature encoding network, and includes a plurality of convolutional layers and a plurality of upsampling layers, wherein the convolutional layers and the upsampling layers are connected in series and arranged alternately;

[0122] The input of the first feature encoding network includes: the clothing transformation image and the clothing segmentation image output by the spatial transformation network;

[0123] The input of the fifth target convolutional layer of the second feature decoding network includes: the intermediate transformed image output by the last downsampling layer of the first feature encoding network;

[0124] The input of the sixth target convolution layer of the second feature decoding network includes: the intermediate transformation image output by the previous upsampling layer adjacent to the sixth target convolution layer and the intermediate transformation image output by the seventh convolution layer in the first feature encoding network;

[0125] The input of the seventh target convolution layer of the second feature decoding network includes: the intermediate transformation image output by the previous upsampling layer adjacent to the seventh target convolution layer and the intermediate transformation image output by the eighth convolution layer in the first feature encoding network;

[0126] The input of the eighth target convolution layer of the second feature decoding network includes: the intermediate transformation image output by the previous upsampling layer adjacent to the eighth target convolution layer and the intermediate transformation image output by the ninth convolution layer in the first feature encoding network;

[0127] The input of the ninth target convolution layer of the second feature decoding network includes: the intermediate transformation image output by the previous upsampling layer adjacent to the ninth target convolution layer and the intermediate transformation image output by the tenth convolution layer in the first feature encoding network;

[0128] Among them, the position order of the fifth target convolution layer to the ninth target convolution layer in the second feature decoding network is consistent with the ascending order, and the position order of the seventh convolution layer to the tenth convolution layer in the first feature encoding network is opposite to the ascending order.

[0129] In the present application embodiment, Figure 5 For example, the downsampling layer after convolution 1 to convolution 4 in the correction network is the first feature encoding network, and convolution 5 to convolution 9 are the second feature decoding network. The fifth to ninth target convolution layers correspond to convolution 5 to convolution 9 in the second feature decoding network, respectively, and their position order is consistent with the increasing order; the seventh to tenth convolution layers correspond to convolution 4, convolution 3, convolution 2 and convolution 1 in the first feature encoding network, respectively, and their position order is opposite to the increasing order.

[0130] Through the above-mentioned network structure setting of the clothing transformation unit, the correction network is used to further correct the clothing transformation image output by the spatial transformation network. In the correction network, the first encoded intermediate transformation image is input to the later convolutional layer for fusion. That is, the correction process is based on the features of each dimension, and finally a more accurate clothing transformation image is obtained.

[0131] Alternatively, if Figure 6 As shown, the test assembly unit includes:

[0132] The second feature encoding network includes a plurality of convolutional layers and a plurality of downsampling layers, wherein the convolutional layers and the downsampling layers are connected in series and arranged alternately;

[0133] A third feature decoding network is connected in series with the second feature encoding network, and includes a plurality of convolutional layers and a plurality of upsampling layers, wherein the convolutional layers and the upsampling layers are connected in series and arranged alternately;

[0134] The input of the second feature encoding network includes: the clothing transformation image, the clothing segmentation image and the target object's information to be retained, wherein the target object's information to be retained includes the target object's part to be retained obtained by segmenting the target object's image;

[0135] The input of the tenth target convolutional layer of the third feature decoding network includes: the intermediate encoded image output by the last downsampling layer of the second feature encoding network;

[0136] The input of the eleventh target convolution layer of the third feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the eleventh target convolution layer and the intermediate encoded image output by the eleventh convolution layer in the second feature encoding network;

[0137] The input of the twelfth target convolution layer of the third feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the twelfth target convolution layer and the intermediate encoded image output by the twelfth convolution layer in the second feature encoding network;

[0138] The input of the thirteenth target convolution layer of the third feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the thirteenth target convolution layer and the intermediate encoded image output by the thirteenth convolution layer in the second feature encoding network;

[0139] The input of the fourteenth target convolution layer of the third feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the fourteenth target convolution layer and the intermediate encoded image output by the fourteenth convolution layer in the second feature encoding network;

[0140] Among them, the position order of the tenth target convolution layer to the fourteenth target convolution layer in the third feature decoding network is consistent with the ascending order, and the position order of the eleventh convolution layer to the fourteenth convolution layer in the second feature encoding network is opposite to the ascending order.

[0141] In the present application embodiment, Figure 6 For example, the test unit includes a second feature encoding network ( Figure 6 The downsampling layer after convolution 1 to convolution 4) and the third feature decoding network ( Figure 6 The tenth to fourteenth target convolution layers correspond to convolutions 5 to 9 in the third feature decoding network, respectively, and their position order is consistent with the increasing order; the eleventh to fourteenth convolution layers correspond to convolutions 4, 3, 2, and 1 in the second feature encoding network, respectively, and their position order is opposite to the increasing order.

[0142] In the embodiment of the present application, the information to be retained of the target object includes the parts to be retained of the target object obtained by segmenting the image of the target object. For example, if the target clothing is a short-sleeved shirt, the parts to be retained include the head, left and right arms, left and right legs, left and right feet, etc. If the target clothing is pants, the parts to be retained include the head, left and right arms, chest, waist, left and right feet, etc.

[0143] Through the above-mentioned network structure setting of the fitting unit, the first encoded intermediate encoded image is input into the later convolutional layer for fusion, that is, the features of each dimension are fully integrated, and finally a fitting image of the target object trying on the target clothing is obtained.

[0144] Based on the same technical concept, according to one aspect of the embodiments of the present application, an embodiment of a virtual fitting method is provided, such as Figure 7 As shown, the method may include the following steps:

[0145] Step S702, extracting the human features of the target object and the clothing features of the target clothing, and after fusing the human features and the clothing features, using multiple loss functions to perform multi-scale supervision on the decoding process of the fused features to generate a clothing segmentation image of the target clothing that matches the appearance of the target object, wherein different loss functions correspond to different scales, and the multiple loss functions are used to gradually improve the matching degree between the intermediate segmentation image of the target clothing obtained in the decoding process of the fused features and the appearance of the target object;

[0146] Step S704, transforming the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing;

[0147] Step S706, after segmenting the target object according to the clothing segmentation image, the clothing transformation image is fitted to the segmented area to obtain a fitting image of the target object for the target clothing.

[0148] Through steps S702 to S706, the present application fuses the human features of the target object and the clothing features of the target clothing, directly predicts and outputs a deformed clothing segmentation image that matches the appearance of the target object, reduces the accumulated prediction error of the related art in first predicting the human body instance segmentation image of the model, and then predicting the deformed clothing image segmentation map based on the human body instance segmentation image, further improves the accuracy of the fitting image, makes the fitting effect more fitting and natural, and solves the technical problem of inaccurate virtual try-on results due to accumulated errors. In addition, the technical solution of the present application further improves the deformation segmentation unit, that is, the feature information output by the upsampling part used to decode the fused features in the deformation segmentation unit is supervised in the middle layer, so that each upsampling decoding part outputs the segmentation result at the corresponding scale, and then fuses the shallow feature information of the feature encoding part to achieve coarse-to-fine segmentation result correction, and finally obtains a refined clothing segmentation image.

[0149] In step S702, extracting the human features of the target object and the clothing features of the target clothing, and fusing the human features and the clothing features, using multiple loss functions to perform multi-scale supervision on the decoding process of the fused features to generate a clothing segmentation image of the target clothing that matches the appearance of the target object, including:

[0150] Step 1, extracting clothing features of the target clothing using an image of the target clothing;

[0151] Step 2, using a preset segmentation map and a preset key point map of the target object to extract the human body features of the target object, wherein the preset segmentation map is an image obtained by separately identifying and segmenting each part of the target object in advance, and the preset key point map is used to mark the key points of the human body;

[0152] Step 3, fusing the clothing features and the human body features, and upsampling and decoding the fused features throughout the fusion process to obtain a clothing segmentation image, wherein the intermediate segmentation image is generated in the process of upsampling and decoding the fused features, and the clothing segmentation image is the last of the intermediate segmentation images;

[0153] Among them, the decoding process also includes: using multiple loss functions to calculate the loss values ​​of the intermediate segmentation image decoded at different scales and the true value segmentation image of the corresponding scale, and using the loss value to adjust the parameters of the first feature decoding network until the loss value of the intermediate segmentation image and the true value segmentation image of the corresponding scale is less than the target loss threshold.

[0154] In step S704, the target clothing is transformed according to the clothing segmentation image, and the clothing transformation image of the target clothing is obtained, including:

[0155] Obtain an image of the target's clothing;

[0156] The target clothing image is transformed according to the clothing segmentation image to obtain a clothing transformation image;

[0157] The clothing transformation image is transformed and corrected to obtain the final clothing transformation image.

[0158] In step S706, after the target object is segmented according to the clothing segmentation image, the human body information to be retained is determined, and the clothing transformation image is fitted to the segmented area to be spliced ​​with the above human body information to be retained to obtain a fitting image of the target object for the target clothing.

[0159] According to another aspect of the embodiments of the present application, the present application provides an electronic device, such as Figure 8 As shown, it includes a memory 801, a processor 803, a communication interface 805 and a communication bus 807. The memory 801 stores a computer program that can be run on the processor 803. The memory 801 and the processor 803 communicate through the communication interface 805 and the communication bus 807. When the processor 803 executes the computer program, the steps of the above method are implemented.

[0160] The memory and processor in the above electronic device communicate via a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc.

[0161] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0162] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0163] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any of the above embodiments.

[0164] Optionally, in an embodiment of the present application, the computer-readable medium is configured to store program codes for the processor to perform the following steps:

[0165] Extracting the human features of the target object and the clothing features of the target clothing, and fusing the human features and the clothing features, using multiple loss functions to perform multi-scale supervision on the decoding process of the fused features, so as to generate a clothing segmentation image of the target clothing that matches the appearance of the target object, wherein different loss functions correspond to different scales, and the multiple loss functions are used to gradually improve the matching degree between the intermediate segmentation image of the target clothing obtained in the decoding process of the fused features and the appearance of the target object;

[0166] Transform the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing;

[0167] After the target object is segmented according to the clothing segmentation image, the clothing transformation image is fitted to the segmented region to obtain a fitting image of the target object in the target clothing.

[0168] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0169] When the embodiments of the present application are specifically implemented, reference may be made to the above-mentioned embodiments, which have corresponding technical effects.

[0170] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0171] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0172] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0173] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0174] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0175] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0176] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0177] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk. It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0178] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A virtual fitting system, characterized in that: include: a deformation segmentation unit, used for extracting the human features of the target object and the clothing features of the target clothing, and after fusing the human features and the clothing features, using multiple loss functions to perform multi-scale supervision on the decoding process of the fused features, so as to generate a clothing segmentation image of the target clothing that matches the appearance of the target object, wherein different loss functions correspond to different scales, and the multiple loss functions are used to gradually improve the matching degree between the intermediate segmentation image of the target clothing obtained in the decoding process of the fused features and the appearance of the target object; a clothing transformation unit, configured to transform the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing; A fitting unit, configured to segment the target object according to the clothing segmentation image, fit the clothing transformation image to the segmentation area, and obtain a fitting image of the target object wearing the target clothing; The deformation segmentation unit comprises: A first feature extraction network, configured to extract the clothing features of the target clothing using the image of the target clothing; A second feature extraction network is used to extract the human body features of the target object by using a preset segmentation map and a preset key point map of the target object, wherein the preset segmentation map is an image obtained by separately identifying and segmenting each part of the target object in advance, and the preset key point map is used to mark key points of the human body; A first feature decoding network is used to fuse the clothing features output by each convolution layer of the first feature extraction network and the human body features output by each convolution layer of the second feature extraction network, and upsample and decode the fused features throughout the fusion process to obtain the clothing segmentation image, wherein the intermediate segmentation image is generated in the process of upsampling and decoding the fused features, and the clothing segmentation image is the last of the intermediate segmentation images; The loss supervision network is used to calculate the loss values ​​of the intermediate segmented image decoded at different scales and the true value segmented image of the corresponding scale using multiple loss functions, and use the loss values ​​to adjust the parameters of the first feature decoding network until the loss value of the intermediate segmented image and the true value segmented image of the corresponding scale is less than a target loss threshold.

2. The system according to claim 1, characterized in that The first feature decoding network includes a plurality of feature decoding units connected in series, each of the feature decoding units includes two convolutional layers and an upsampling layer, the two convolutional layers are arranged one before and one after the upsampling layer, the upsampling layer is used to upsample and decode the fused features, and the convolutional layer arranged after the upsampling layer is used to output the intermediate segmented image corresponding to the current scale according to the features decoded by the upsampling layer; The input of the first target convolutional layer of the first feature decoding network includes: the clothing features output by the last downsampling layer in the first feature extraction network and the human body features output by the last downsampling layer in the second feature extraction network; The input of the second target convolution layer of the first feature decoding network includes: the intermediate segmented image output by the previous feature decoding unit adjacent to the second target convolution layer, the clothing features output by the first convolution layer in the first feature extraction network, and the human body features output by the second convolution layer in the second feature extraction network; The input of the third target convolution layer of the first feature decoding network includes: the intermediate segmented image output by the previous feature decoding unit adjacent to the third target convolution layer, the clothing features output by the third convolution layer in the first feature extraction network, and the human body features output by the fourth convolution layer in the second feature extraction network; The input of the fourth target convolution layer of the first feature decoding network includes: the intermediate segmented image output by the previous feature decoding unit adjacent to the fourth target convolution layer, the clothing features output by the fifth convolution layer in the first feature extraction network, and the human body features output by the sixth convolution layer in the second feature extraction network; Among them, the position order of the first target convolution layer to the fourth target convolution layer in the first feature decoding network is consistent with the ascending order, the position order of the first convolution layer, the third convolution layer and the fifth convolution layer in the first feature extraction network is consistent with the ascending order, and the position order of the second convolution layer, the fourth convolution layer and the sixth convolution layer in the second feature extraction network is consistent with the ascending order.

3. The system according to claim 2, characterized in that The loss supervision network includes: A first loss supervision unit, configured to constrain the intermediate segmented image output by a previous feature decoding unit adjacent to the second target convolutional layer through a first loss function, wherein the first loss function matches a segmentation scale of a current feature decoding unit; A second loss supervision unit is used to constrain the intermediate segmented image output by the previous feature decoding unit adjacent to the third target convolution layer through a second loss function, wherein the second loss function matches the segmentation scale of the current feature decoding unit; A third loss supervision unit is used to constrain the intermediate segmented image output by the previous feature decoding unit adjacent to the fourth target convolutional layer through a third loss function, wherein the third loss function matches the segmentation scale of the current feature decoding unit.

4. The system according to claim 1, characterized in that The first feature extraction network includes multiple convolutional layers and multiple downsampling layers, the convolutional layers and the downsampling layers are connected in series and alternately arranged, the convolutional layers are used to extract the clothing features of the target clothing, and the downsampling layers are used to downsample and encode the clothing features output by the convolutional layers.

5. The system according to claim 1, characterized in that The second feature extraction network includes multiple convolutional layers and multiple downsampling layers, the convolutional layers and the downsampling layers are connected in series and alternately arranged, the convolutional layers are used to extract the human body features of the target object, and the downsampling layers are used to downsample and encode the human body features output by the convolutional layers.

6. The system according to claim 1, characterized in that The clothing transformation unit comprises: a spatial transformation network, for taking the target clothing image and the clothing segmentation image as input, transforming the target clothing according to the clothing segmentation image, and obtaining the clothing transformation image; The correction network is used to perform transformation correction on the clothing transformation image output by the spatial transformation network to obtain a final clothing transformation image.

7. The system according to claim 6, characterized in that The correction network comprises: A first feature encoding network includes a plurality of convolutional layers and a plurality of downsampling layers, wherein the convolutional layers and the downsampling layers are connected in series and arranged alternately; A second feature decoding network is connected in series with the first feature encoding network, comprising a plurality of convolutional layers and a plurality of upsampling layers, wherein the convolutional layers and the upsampling layers are connected in series and arranged alternately; The input of the first feature encoding network includes: the clothing transformation image and the clothing segmentation image output by the spatial transformation network; The input of the fifth target convolutional layer of the second feature decoding network includes: the intermediate transformed image output by the last downsampling layer of the first feature encoding network; The input of the sixth target convolution layer of the second feature decoding network includes: the intermediate transformation image output by the previous upsampling layer adjacent to the sixth target convolution layer and the intermediate transformation image output by the seventh convolution layer in the first feature encoding network; The input of the seventh target convolutional layer of the second feature decoding network includes: the intermediate transformation image output by the previous upsampling layer adjacent to the seventh target convolutional layer and the intermediate transformation image output by the eighth convolutional layer in the first feature encoding network; The input of the eighth target convolution layer of the second feature decoding network includes: the intermediate transformation image output by the previous upsampling layer adjacent to the eighth target convolution layer and the intermediate transformation image output by the ninth convolution layer in the first feature encoding network; The input of the ninth target convolutional layer of the second feature decoding network includes: the intermediate transformation image output by the previous upsampling layer adjacent to the ninth target convolutional layer and the intermediate transformation image output by the tenth convolutional layer in the first feature encoding network; Among them, the position order of the fifth target convolution layer to the ninth target convolution layer in the second feature decoding network is consistent with the ascending order, and the position order of the seventh convolution layer to the tenth convolution layer in the first feature encoding network is opposite to the ascending order.

8. The system according to claim 1, characterized in that The test assembly unit comprises: A second feature encoding network includes a plurality of convolutional layers and a plurality of downsampling layers, wherein the convolutional layers and the downsampling layers are connected in series and arranged alternately; A third feature decoding network is connected in series with the second feature encoding network, comprising a plurality of convolutional layers and a plurality of upsampling layers, wherein the convolutional layers and the upsampling layers are connected in series and arranged alternately; The input of the second feature encoding network includes: the clothing transformation image, the clothing segmentation image and the target object's information to be retained, wherein the target object's information to be retained includes the target object's part to be retained obtained by segmenting the target object's image; The input of the tenth target convolutional layer of the third feature decoding network includes: the intermediate coded image output by the last downsampling layer of the second feature encoding network; The input of the eleventh target convolution layer of the third feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the eleventh target convolution layer and the intermediate encoded image output by the eleventh convolution layer in the second feature encoding network; The input of the twelfth target convolution layer of the third feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the twelfth target convolution layer and the intermediate encoded image output by the twelfth convolution layer in the second feature encoding network; The input of the thirteenth target convolution layer of the third feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the thirteenth target convolution layer and the intermediate encoded image output by the thirteenth convolution layer in the second feature encoding network; The input of the fourteenth target convolution layer of the third feature decoding network includes: the intermediate decoded image output by the previous upsampling layer adjacent to the fourteenth target convolution layer and the intermediate encoded image output by the fourteenth convolution layer in the second feature encoding network; Among them, the position order of the tenth target convolution layer to the fourteenth target convolution layer in the third feature decoding network is consistent with the ascending order, and the position order of the eleventh convolution layer to the fourteenth convolution layer in the second feature encoding network is opposite to the ascending order.

9. A virtual fitting method, characterized in that: include: Extracting the human features of the target object and the clothing features of the target clothing, and fusing the human features and the clothing features, using multiple loss functions to perform multi-scale supervision on the decoding process of the fused features to generate a clothing segmentation image of the target clothing that matches the appearance of the target object, wherein different loss functions correspond to different scales, and the multiple loss functions are used to gradually improve the matching degree between the intermediate segmentation image of the target clothing obtained in the decoding process of the fused features and the appearance of the target object; Transforming the target clothing according to the clothing segmentation image to obtain a clothing transformation image of the target clothing; After segmenting the target object according to the clothing segmentation image, fitting the clothing transformation image to the segmentation region to obtain a fitting image of the target object with respect to the target clothing; The step of extracting the human features of the target object and the clothing features of the target clothing, fusing the human features and the clothing features, and using multiple loss functions to perform multi-scale supervision on the decoding process of the fused features to generate a clothing segmentation image of the target clothing that matches the appearance of the target object includes: Extracting the clothing features of the target clothing using the image of the target clothing; The human body features of the target object are extracted using a preset segmentation map and a preset key point map of the target object, wherein the preset segmentation map is an image obtained by separately identifying and segmenting each part of the target object in advance, and the preset key point map is used to mark key points of the human body; The clothing feature and the human body feature are fused, and the fused feature is upsampled and decoded throughout the fusion process to obtain the clothing segmentation image, wherein the intermediate segmentation image is generated in the process of upsampling and decoding the fused feature, and the clothing segmentation image is the last of the intermediate segmentation images; Among them, the decoding process also includes: using multiple loss functions to calculate the loss values ​​of the intermediate segmented image decoded at different scales and the true value segmented image of the corresponding scale, and using the loss value to adjust the parameters of the first feature decoding network until the loss value of the intermediate segmented image and the true value segmented image of the corresponding scale is less than the target loss threshold.

10. The method according to claim 9, characterized in that Transforming the target clothing according to the clothing segmentation image to obtain the clothing transformation image of the target clothing includes: Acquire an image of the target's attire; transforming the target clothing image according to the clothing segmentation image to obtain the clothing transformation image; The clothing transformation image is subjected to transformation correction to obtain a final clothing transformation image.

11. An electronic device, comprising a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, and the memory and the processor communicate through the communication bus and the communication interface, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 9 to 10 are implemented.

12. A computer-readable medium having a non-volatile program code executable by a processor, characterized in that: The program code enables the processor to execute the method according to any one of claims 9 to 10.