Two-stage suit style generation and style editing method in virtual fitting, electronic device and computer readable storage medium

Abstract

The two-stage suit style generation and style editing method in virtual try-on, electronic equipment and computer readable storage medium belong to the field of artificial intelligence, in order to solve the problem that the suitable matching result is given and edited in the visual mode of virtual try-on, and the suit output with unified material and style is uniform, including migrating the style of clothes in the style reference picture to the try-on picture, wherein the style of clothes in the try-on picture is editable, and the effect is that the related style editing and synthesis output can be realized on the basis of completing the style migration.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically involving a deep learning method for a two-stage kit from color and texture style generation to pattern block automatic editing. Background Technology

[0002] Virtual try-on is a technology that uses computer vision algorithms and computer image processing to allow users to try on different clothes remotely via the internet, eliminating the need for them to visit physical stores. While it doesn't achieve the same effect as trying on clothes in person, in today's era of prevalent online shopping, this technology allows users to visually experience how they would look in an outfit before purchasing, providing a sense of how well it suits their needs and offering necessary intuitive and visual assistance for product selection. Research and applications in virtual try-on generally employ common methods that process styles and designs separately, differing mainly in the deep learning networks used, leading to variations in the generated virtual try-on results. Furthermore, current literature on editing complete outfits primarily relies on big data analysis to provide styling suggestions. There is an urgent need to address the need for a visual approach to virtual try-on, providing and editing suitable outfits with uniform materials and styles. Summary of the Invention

[0003] To address the issue of providing and editing suitable outfit combinations through a visual virtual try-on method, and outputting suits with uniform materials and styles, a two-stage suit style generation and pattern editing method in virtual try-on, according to some embodiments of this application, includes transferring the style of clothing in a style reference image to a try-on image, wherein the style of clothing in the try-on image is editable.

[0004] The method for generating and editing two-stage suit styles in virtual try-on according to some embodiments of this application includes:

[0005] Based on the image input by the user, obtain the input style characteristics;

[0006] Based on the style reference image, obtain the reference style characteristics;

[0007] The reference style features are fused with the input style features to obtain the style transfer features of the image;

[0008] Based on the style transfer characteristics of the image, obtain a style editing reference;

[0009] Obtain the pattern block editing template corresponding to the pattern control word;

[0010] The pattern block editing is obtained according to the pattern block editing template. The pattern block editing includes the pattern custom editing area and the pattern retention area of ​​the corresponding pattern block editing template.

[0011] The pattern editing reference and the pattern block editing are feature-fused to obtain pattern editing block suggestion features;

[0012] The style editing block suggestion features are fused with the style transfer features of the user's input image to obtain the output image.

[0013] According to some embodiments of this application, the virtual try-on method for generating and editing two-stage suit styles includes a style control word with a high number of digits for the upper garment and a low number of digits for the lower garment. The high number of digits for the upper garment corresponds to an editing template in the upper garment template, and the low number of digits for the lower garment corresponds to an editing template in the lower garment template.

[0014] According to some embodiments of the present application, the virtual try-on method for generating and editing two-stage suit styles includes a top template, a collar template, a sleeve template, and a front placket template; the bottom template includes a random editing template, a shorts template, a skirt template, a long skirt template, and trousers template.

[0015] According to some embodiments of this application, the virtual try-on method for generating and editing two-stage suit styles involves generating a style in the custom editing area of ​​the corresponding style block editing template based on the editing area specified by the template. This operation is performed through the style editing network, which includes...

[0016] The encoding network comprises a 3-layer convolutional neural network with convolutional kernels of 1×1, 3×3 and 1×1 respectively;

[0017] The decoding network comprises a 3-layer convolutional neural network with convolutional kernels of 1×1, 3×3 and 1×1 respectively;

[0018] The decoding network, the network fusion network, includes a 4-layer fully connected network, with each convolutional kernel being 1×1.

[0019] According to some embodiments of this application, in the virtual try-on method for generating and editing two-stage suit styles, the style block editing template corresponding to the style control word is input into a first convolutional neural network composed of three layers of convolutions, each layer having a 3×3 convolutional kernel, and the first convolutional neural network outputs style block editing.

[0020] The style transfer features of the image are input into a fully connected network consisting of four fully connected layers, each with a 1×1 convolutional kernel. The fully connected network outputs a style editing reference.

[0021] The pattern editing reference and the pattern block editing feature fusion are merged. The pattern editing merge is input into a second convolutional neural network composed of 3 layers of convolutions, each layer having a 3×3 convolutional kernel. The second convolutional neural network outputs pattern editing block suggestion features.

[0022] According to some embodiments of the present application, the method for generating and editing a two-stage suit style in virtual try-on includes a step of fusing the style editing block suggestion features with the style transfer features of the try-on user's input image. The feature fusion operation is performed by the style transformation fusion unit, which includes a normalization unit, a convolution unit, and a fusion unit.

[0023] The virtual try-on method for generating and editing two-stage suit styles in some embodiments of this application includes eight style transformation and fusion units.

[0024] This application also provides an electronic device, which includes: one or more processors, a memory, and one or more programs; wherein the one or more programs are stored in the memory, and the one or more programs include instructions that, when executed by the electronic device, cause the electronic device to perform the technical solution of the first aspect of this application and any possible design of the first aspect.

[0025] This application also provides a computer-readable storage medium, which includes a computer program that, when run on an electronic device, causes the electronic device to execute the first aspect of this application and any possible design of the first aspect.

[0026] Beneficial Effects: This invention addresses the need for both style and editable outfits by proposing a deep learning neural network for outfit generation with two-stage style generation and pattern editing. This network targets outfits with consistent style and materials. It uses user input, style references, and pattern control words as inputs, applying color and texture information from the style references to the user's clothing. The pattern control words then edit the final outfit, enabling the method to perform style transfer and subsequent pattern editing and synthesis output. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the overall framework of the invention;

[0028] Figure 2 This is a diagram showing the correspondence between style control text and separate clothing editing templates;

[0029] Figure 3 It is a pattern information extraction network;

[0030] Figure 4 This is a schematic diagram of style transfer in Example 1;

[0031] Figure 5 This is a schematic diagram of the pattern editing in Example 2;

[0032] Figure 6 This is a schematic diagram of the kit generation in Example 3. Detailed Implementation

[0033] Embodiments of this application are described in detail below with reference to the accompanying drawings, examples of which are shown in the drawings.

[0034] 1. Technical terminology conventions in this invention:

[0035] (1) Virtual try-on recipient: The service recipient of virtual try-on is the target person for trying on the suit in this invention.

[0036] (2) Reference model for virtual try-on: The reference person and clothing object for virtual try-on, that is, the model or clothing image information that provides reference virtual try-on clothes for the virtual try-on person in this invention.

[0037] (3) Upper garment: Upper body clothing, that is, the general term for clothing worn on the upper body.

[0038] (4) Lower garments: Lower body clothing, that is, the general term for clothing worn on the lower body.

[0039] (5) Style characteristics: The general term for non-structural characteristics of clothing, such as color, texture, and material.

[0040] (6) Style features: Virtual try-on clothing calculated through computer vision technology, specifically referring to the structural features of the clothing's appearance, outline, cut, shape, etc., which are not related to color and texture.

[0041] (7) Virtual try-on: With the assistance of computer software algorithms, a half-body or full-body photo of a person is used as a template, and clothing from other photos is transferred to the current person. The person does not need to try on the clothes in person. The clothing is changed by computer vision technology, realizing the effect of virtual try-on under visual conditions.

[0042] (8) Try-on user: Generally, the virtual result is the try-on object, which appears as the target of the system. All the generated clothes will eventually be displayed on this input try-on user.

[0043] (9) Style reference: The style reference is one picture, which is responsible for providing the wearer with reference information such as color and texture.

[0044] (10) Style Control Word: The purpose of the style control word is to provide the necessary editing requirements for the suit editing. This includes reasonable control over collar, sleeves, length, etc., to generate richer and more diverse suit output results.

[0045] (11) Clothing boundary: A person's clothing is usually divided into two parts, upper garment and lower garment, with the waist as the dividing line. The clothing boundary is the upper garment and lower garment near the waist.

[0046] (12) Style Blocks: This provides style suggestions for editing the style blocks of clothing. The clothing boundary is used as the dividing line. The upper garment and the lower garment are each processed with 16 clothing blocks of uniform size, for a total of 32 clothing blocks, covering the clothing areas of the entire human body's upper and lower garments.

[0047] (13) Style block editing template: Based on the clothing boundary line, the upper garment has 16 clothing blocks, which are divided into random editing template, collar editing template, sleeve editing template and front editing template according to the characteristics of the upper garment; the lower garment has 16 clothing blocks, which are divided into random editing template, shorts editing template, short skirt editing template, long skirt editing template and trousers editing template according to the characteristics of the lower garment.

[0048] (13) Style Editing Suggestions: Style editing suggestions are based on 16 clothing blocks for tops and 16 clothing blocks for bottoms. Suggestions are made based on the collar, sleeves, and front of the top, and shorts, skirts, long skirts, and trousers for the bottom. The suggested results are divided into two parts: a custom style editing area and a random style editing area.

[0049] (14) Style transfer: The process of transforming the clothing style and color texture of a clothing image, including one or both of the top and bottom garments, onto the body of a human model in another image through an algorithm. This process of transforming clothing from one image to another is called style transfer.

[0050] (15) Style Editing: The style and appearance of clothing from a reference image, including either the top or the bottom, are transformed onto the body of a model in another image to be transferred using an algorithm. Style transfer only retains the structural framework information of the clothing, without including the material and texture information.

[0051] (16) Two stages: The first stage is the style transfer of the bottom and the top, and the second stage is the style transfer of the bottom and the top.

[0052] (17) Set: Compared with non-set, the main feature of a set is that the top and bottom are generated as a set (or a single piece) of clothing with consistent style through style reference. The set can be divided into two parts, a top and a bottom, or it can be a single independent garment.

[0053] (18) Matching: A suit that is coordinated and unified in style and design gives people an overall impression of being generous and appropriate.

[0054] (19) Set generation: After two-stage joint generation calculation, a set generation output with certain matching relationships is formed.

[0055] 2. Technical Solution

[0056] Virtual try-on is a technology that uses computer vision algorithms and computer image processing to allow users to try on different clothes remotely via the internet, eliminating the need for them to visit physical stores. While it doesn't achieve the same effect as trying on clothes in person, in today's era of prevalent online shopping, this technology allows users to visually experience how they would look in an outfit before purchasing, providing a sense of how well it suits their needs and offering necessary intuitive and visual assistance for product selection. Research and applications in virtual try-on generally employ common methods that process styles and designs separately, differing mainly in the deep learning networks used, leading to variations in the generated virtual try-on results. Furthermore, current literature on editing complete outfits primarily relies on big data analysis to provide styling suggestions. There is an urgent need to address the need for a visual approach to virtual try-on, providing and editing suitable outfits with uniform materials and styles.

[0057] The technical solution of this invention is a method for generating the style and pattern of a suit through a two-stage approach. It aims to utilize a deep learning network architecture to decouple the features of style reference and pattern reference images. Taking the clothing structure on the virtual try-on model as a reference and the color and texture information of the style reference images as a basis, it provides a method for generating a virtual try-on result for the virtual try-on user. This method consists of two computational stages. The user inputs an image S, a style reference image F as a color and texture reference, and a pattern control word W to manage the suit pattern editing requirements. After feature decoupling and calculation, the mean and standard deviation of the style reference F are probabilistically fused with the features of the user S to obtain the style transfer result of the first stage. In the second stage, based on the style transfer result obtained in the first stage, the pattern is edited according to the pattern control word W to form a suit result with a unified style.

[0058] In one embodiment, a method for generating and editing a two-stage suit style in a virtual try-on includes obtaining input style features based on an image input by the try-on user;

[0059] Based on the style reference image, obtain the reference style characteristics;

[0060] The reference style features are fused with the input style features to obtain the style transfer features of the image;

[0061] Based on the style transfer characteristics of the image, obtain a style editing reference;

[0062] Obtain the pattern block editing template corresponding to the pattern control word;

[0063] The pattern block editing is obtained according to the pattern block editing template. The pattern block editing includes the pattern custom editing area and the pattern retention area of ​​the corresponding pattern block editing template.

[0064] The pattern editing reference and the pattern block editing are feature-fused to obtain pattern editing block suggestion features;

[0065] The style editing block suggestion features are fused with the style transfer features of the user's input image to obtain the output image.

[0066] In a further embodiment, the style control word includes a high number of digits for the upper garment and a low number of digits for the lower garment. The high number of digits for the upper garment corresponds to an edit template in the upper garment template, and the low number of digits for the lower garment corresponds to an edit template in the lower garment template.

[0067] In a further embodiment, the top template includes a random editing template, a collar editing template, a sleeve editing template, and a front editing template; the bottom template includes a random editing template, a shorts editing template, a short skirt editing template, a long skirt editing template, and a long pants editing template.

[0068] In a further embodiment, the style customization editing area of ​​the corresponding style block editing template performs a style generation operation based on the editing area specified by the template. This operation is performed through the style editing network, wherein the style editing network includes...

[0069] The encoding network comprises a 3-layer convolutional neural network with convolutional kernels of 1×1, 3×3 and 1×1 respectively;

[0070] The decoding network comprises a 3-layer convolutional neural network with convolutional kernels of 1×1, 3×3 and 1×1 respectively;

[0071] The decoding network, the network fusion network, includes a 4-layer fully connected network, with each convolutional kernel being 1×1.

[0072] In a further embodiment, the pattern block editing template corresponding to the pattern control word is input into a first convolutional neural network consisting of three layers of convolutions, each layer having a 3×3 convolutional kernel, and the first convolutional neural network outputs the pattern block editing.

[0073] The style transfer features of the image are input into a fully connected network consisting of four fully connected layers, each with a 1×1 convolutional kernel. The fully connected network outputs a style editing reference.

[0074] The pattern editing reference and the pattern block editing feature fusion are merged. The pattern editing merge is input into a second convolutional neural network composed of 3 layers of convolutions, each layer having a 3×3 convolutional kernel. The second convolutional neural network outputs pattern editing block suggestion features.

[0075] In a further embodiment, the step involves fusing the pattern editing block suggestion features with the style transfer features of the user's input image. The feature fusion operation is performed by the pattern transformation fusion unit, which includes a normalization unit, a convolution unit, and a fusion unit.

[0076] In a further embodiment, eight pattern transformation and fusion units are included.

[0077] The method is implemented by the deep learning-based virtual try-on system, which includes a style transfer module that outputs the style transfer features of the image.

[0078] The style editing module obtains style editing block suggestion features based on the style transfer features and style control words of the image, and obtains virtual try-on images based on the style editing block suggestion features.

[0079] In a further embodiment, the style transfer module includes

[0080] Style encoder E1, the input of which is the image input by the wearer;

[0081] Style encoder E2, the input of which is a style reference image;

[0082] Style fusion unit H includes a first input terminal and a second input terminal. The output of style encoder E1 is connected to the first input of style fusion unit H, and the output of style encoder E2 is connected to the second input of style fusion unit H.

[0083] Style decoder D, the output of style fusion unit H is connected to the input of style decoder D, and style decoder D outputs the style transfer features of the image.

[0084] In a further embodiment, the pattern editing module includes

[0085] The style block editing unit is used to obtain style editing block suggestion features based on the style transfer features and style control words of the image;

[0086] Upsampling unit R0 upsamples the pattern editing block suggestion features;

[0087] Feature decoupling unit L2 decouples the upsampled features;

[0088] The downsampling unit samples the decoupled features.

[0089] Normalization unit, which normalizes the normalized features;

[0090] Feature decoupling unit L1 decouples the style transfer features of the image;

[0091] Convolution, which involves convolving the decoupled features;

[0092] The pattern transformation fusion unit normalizes and convolves the convolved features, and then fuses the convolved features with the features normalized by the normalization unit to obtain the output image.

[0093] In a further embodiment, the pattern block editing unit includes...

[0094] The first convolutional neural network consists of three layers of convolutional kernels, with each layer having a 3×3 convolutional kernel.

[0095] A fully connected network consisting of four fully connected layers, each with a 1×1 convolutional kernel;

[0096] The second convolutional neural network consists of three layers, each with a 3×3 convolutional kernel.

[0097] In a further embodiment, the pattern transformation fusion unit includes a normalization unit, a convolution unit, and a fusion unit;

[0098] The downsampling unit includes downsampling units R1 to R7;

[0099] The normalization unit includes normalization units N1 to N8;

[0100] The pattern transformation and fusion unit includes pattern transformation and fusion units G1 to G8;

[0101] The convolution includes convolutions C1 to C8.

[0102] In a further embodiment, the input to the upsampling unit is the block-based proposed features. The upsampling unit R0 is connected to the input of the feature decoupling unit L2. The output of the feature decoupling unit L2 is connected to the input of the normalization unit N8 and the input of the downsampling unit R7, respectively. The output of the downsampling unit R7 is connected to the input of the normalization unit N7 and the downsampling unit R6, respectively. The output of the downsampling unit R6 is connected to the input of the normalization unit N6 and the downsampling unit R5, respectively. The output of the downsampling unit R5 is connected to the input of the normalization unit N5 and the downsampling unit R4, respectively. The output of the downsampling unit R4 is connected to the input of the normalization unit N4 and the downsampling unit R3, respectively. The output of the downsampling unit R3 is connected to the input of the normalization unit N3 and the downsampling unit R2, respectively. The output of the downsampling unit R2 is connected to the input of the normalization unit N2 and the downsampling unit R1, respectively. The output of the downsampling unit R1 is connected to the input of the normalization unit N1.

[0103] The input to the feature decoupling unit L1 is the style transfer features of the image. The feature decoupling unit L1 is connected in parallel with the inputs of convolution C1, convolution C2, convolution C3, convolution C4, convolution C5, convolution C6, convolution C7, and convolution C8.

[0104] In a further embodiment, the convolutional layer C1 is connected to the input of the normalization layer of the pattern transformation fusion unit G1, the output of the normalization layer of the pattern transformation fusion unit G1 is connected to the input of the convolutional layer of the pattern transformation fusion unit G1, the output of the convolutional layer of the pattern transformation fusion unit G1 is connected to the first input of the fusion layer of the pattern transformation fusion unit G1, the output of the normalization layer N1 is connected to the second input of the fusion layer of the pattern transformation fusion unit G1, and the output of the fusion layer of the pattern transformation fusion unit G1 is connected to the first input of the normalization layer of the pattern transformation fusion unit G2.

[0105] The convolutional layer C2 is connected to the second input of the normalization layer of the pattern transformation fusion unit G2, the output of the normalization layer of the pattern transformation fusion unit G2 is connected to the input of the convolutional layer of the pattern transformation fusion unit G2, the output of the convolutional layer of the pattern transformation fusion unit G2 is connected to the first input of the fusion layer of the pattern transformation fusion unit G2, the output of the normalization layer N2 is connected to the second input of the fusion layer of the pattern transformation fusion unit G2, and the output of the fusion layer of the pattern transformation fusion unit G2 is connected to the first input of the normalization layer of the pattern transformation fusion unit G3.

[0106] The convolutional layer C3 is connected to the second input of the normalization layer of the pattern transformation fusion unit G3, the output of the normalization layer of the pattern transformation fusion unit G3 is connected to the input of the convolutional layer of the pattern transformation fusion unit G3, the output of the convolutional layer of the pattern transformation fusion unit G3 is connected to the first input of the fusion layer of the pattern transformation fusion unit G3, the output of the normalization layer N3 is connected to the second input of the fusion layer of the pattern transformation fusion unit G3, and the output of the fusion layer of the pattern transformation fusion unit G3 is connected to the first input of the normalization layer of the pattern transformation fusion unit G4.

[0107] The convolutional layer C4 is connected to the second input of the normalization layer of the pattern transformation fusion unit G4, the output of the normalization layer of the pattern transformation fusion unit G4 is connected to the input of the convolutional layer of the pattern transformation fusion unit G4, the output of the convolutional layer of the pattern transformation fusion unit G4 is connected to the first input of the fusion layer of the pattern transformation fusion unit G4, the output of the normalization layer N4 is connected to the second input of the fusion layer of the pattern transformation fusion unit G4, and the output of the fusion layer of the pattern transformation fusion unit G4 is connected to the first input of the normalization layer of the pattern transformation fusion unit G5.

[0108] The convolutional layer C5 is connected to the second input of the normalization layer of the pattern transformation fusion unit G5, the output of the normalization layer of the pattern transformation fusion unit G5 is connected to the input of the convolutional layer of the pattern transformation fusion unit G5, the output of the convolutional layer of the pattern transformation fusion unit G5 is connected to the first input of the fusion layer of the pattern transformation fusion unit G5, the output of the normalization layer N5 is connected to the second input of the fusion layer of the pattern transformation fusion unit G5, and the output of the fusion layer of the pattern transformation fusion unit G5 is connected to the first input of the normalization layer of the pattern transformation fusion unit G6.

[0109] The convolutional layer C6 is connected to the second input of the normalization layer of the pattern transformation fusion unit G6, the output of the normalization layer of the pattern transformation fusion unit G6 is connected to the input of the convolutional layer of the pattern transformation fusion unit G6, the output of the convolutional layer of the pattern transformation fusion unit G6 is connected to the first input of the fusion layer of the pattern transformation fusion unit G6, the output of the normalization layer N6 is connected to the second input of the fusion layer of the pattern transformation fusion unit G6, and the output of the fusion layer of the pattern transformation fusion unit G6 is connected to the first input of the normalization layer of the pattern transformation fusion unit G7.

[0110] The convolutional layer C7 is connected to the second input of the normalization layer of the pattern transformation fusion unit G7, the output of the normalization layer of the pattern transformation fusion unit G7 is connected to the input of the convolutional layer of the pattern transformation fusion unit G7, the output of the convolutional layer of the pattern transformation fusion unit G7 is connected to the first input of the fusion layer of the pattern transformation fusion unit G7, the output of the normalization layer N7 is connected to the second input of the fusion layer of the pattern transformation fusion unit G7, and the output of the fusion layer of the pattern transformation fusion unit G7 is connected to the first input of the normalization layer of the pattern transformation fusion unit G8.

[0111] The convolutional layer C8 is connected to the second input of the normalization layer of the pattern transformation fusion unit G8. The output of the normalization layer of the pattern transformation fusion unit G8 is connected to the input of the convolutional layer of the pattern transformation fusion unit G8. The output of the convolutional layer of the pattern transformation fusion unit G8 is connected to the first input of the fusion layer of the pattern transformation fusion unit G8. The output of the normalization layer N8 is connected to the second input of the fusion layer of the pattern transformation fusion unit G8. The output of the fusion layer of the pattern transformation fusion unit G8 is the output image.

[0112] In a further manner, the style blender H outputs the style blended result H(E1(S), E2(F)).

[0113]

[0114] Where σ() function is the mean operation, μ() function is the standard deviation operation, ∈ is any positive decimal, style encoder E1 outputs style coding result E1(S), style encoder E2 outputs style coding result E2(F);

[0115] Style transfer features T of the output image of the style decoder D

[0116] T = D(H(E1(S), E2(F))).

[0117] In a further manner, the style editor merges B'

[0118] B′=T′+W′

[0119] Among them: style block editing W′, style editing reference T′;

[0120] The pattern editing merges B′ into a second convolutional neural network consisting of three layers of convolutions, each with a 3×3 convolutional kernel, and outputs the pattern editing block suggestion feature B.

[0121] In a further embodiment, Y represents the output image, where:

[0122] G1=G1(C1(L1(T)),N1(R1(R2(...R8(L2(B))))))

[0123] G2=G2(C2(L1(Q2)),N2(R2(R3(...R8(L2(B))))))

[0124] G3=G3(c3(L1(Q3)),N3(R3(R4(...R8(L2(B))))))

[0125] G4=G4(C4(L1(Q4)),N4(R4(R5(...R8(L2(B))))))

[0126] G5=G5(C5(L1(Q5)),N5(R5(R6(...R8(L2(B))))))

[0127] G6=G6(C6(L1(Q6)),N6(R6(R7(R8(L2(B)))))))

[0128] G7=G7(C7(L1(Q7)),N7(R7(R8(L2(B)))))

[0129] Y=G8=G8(C8(L1(Q8)), N8(R8(L2(B)))).

[0130] This invention addresses the need for both style and editable outfits by proposing a deep learning neural network for outfit generation with two stages: style generation and pattern editing. This network targets outfits with consistent style and materials. It uses user input, style references, and pattern control words as inputs, applying color and texture information from the style references to the user's clothing. The pattern control words then edit the final outfit, enabling the method to perform style transfer and subsequent pattern editing and synthesis output.

[0131] To further understand the above embodiments, this example uses data stream processing as a basis to provide a processing flow description from input to output.

[0132] (1) Overall technical solution

[0133] The invention relates to a two-stage method for generating and editing suit styles in a virtual try-on setting.

[0134] The method takes as input an image S (input from the user) and a style reference image F (input from the user's image), both of which are image information. The style control word W controls the specific structural information of the style and is a numerical variable. Besides the input and output, the core processing is divided into two modules: a style transfer module and a style editing module.

[0135] The system has two input terminals: a try-on input (S) and a style reference input (F). It also has one output terminal: a suit output (T). The image resolution dimensions of all three terminals are (3, W0, H0), which represent the three channels of RGB image information: red, green, and blue components. The width and height are (W0, H0) respectively. The input image size and the output image size are kept consistent.

[0136] (2) Style transfer

[0137] The main function of the style transfer module is to use a style reference F to perform a probabilistic transformation on the clothing style input S by the wearer, generating new tops and bottoms. To ensure that the clothing structure does not change or pixel features drift before and after the style transfer, the structural parameters of the tops and bottoms remain unchanged. That is, the style of the tops and bottoms does not change before and after the style transfer transformation; only the style information, including color and texture, changes.

[0138] (3) Template for editing style blocks

[0139] In this invention, the pattern block editing template provides pattern editing suggestions for clothing pattern migration. Its purpose is to provide block-based pattern editing suggestions to solve the problem of fine matching of clothing texture and color.

[0140] The pattern editing templates are divided into blocks, with 5 templates for tops and 6 templates for bottoms. The model's image is evenly divided into blocks, with clothing lines as the upper and lower boundaries. Each section is divided into 16 areas, called pattern blocks. The 16 clothing blocks for the top are further categorized based on their characteristics: "Maintain Editing Template," "Collar Editing Template," "Sleeve Editing Template," "Front Editing Template," and "Random Editing Template." Similarly, the 16 clothing blocks for the bottom are categorized based on their characteristics: "Maintain Editing Template," "Shorts Editing Template," "Skirt Editing Template," "Long Skirt Editing Template," "Long Pants Editing Template," and "Random Editing Template." The customizable areas in the pattern block editing templates correspond to the actual clothing areas. For example, in the collar editing template for the top, the collar maintains its spatial relationship relative to the body. The areas requiring editing are the custom editing areas, while the areas outside the collar are the "Maintain Style Areas," meaning the style in these areas remains unchanged.

[0141] (4) Style Editing Suggestions

[0142] The pattern is edited in blocks, and each pattern's custom editing area is used to generate patterns based on the editing area specified in the template. This operation is performed by a pattern editing network, which consists of three modules: an encoding network, a fusion network, and a decoding network, all of which are convolutional neural networks. The encoding and decoding networks are both 3-layer convolutional neural networks with 1×1, 3×3, and 1×1 kernels, respectively; the fusion network is a 4-layer fully connected network with 1×1 kernels for all layers.

[0143] The input to the encoding network is the result of style transfer. The input to the fusion network is the template initialization result given by the style block editing template, and the input is the style transfer result feature of the encoding network. After fusion, it is fed into the decoding network to obtain style editing suggestion B.

[0144] (5) Style Editing

[0145] The main task of the style editing module is to perform style migration processing on the result T processed by the style migration module and the style editing suggestion B. After calculation by the style transformation fusion unit 8 times, the final set output result Y is obtained.

[0146] 4. Constraints

[0147] (1) The input is the image S of the try-on user and the image F of the style reference. The output is the outfit output result T, which are both image information, and are (3, W0, H0). The width and height of the image are fixed at 512 pixels, i.e. W0 = 512, H0 = 512, and the unit is image pixels.

[0148] (2) The style control word W is the specific structural information that controls the style. This control word consists of the high-order bits of the upper garment. H 、Lower digits W L Composition. A control word is determined by two numerical values, where the higher-order bit W H The values ​​are (0, 1, 2, 3, 4), a total of 5 values, corresponding to the upper garment part, namely the random editing template, collar editing template, sleeve editing template, and front editing template; the lower digit W L The possible values ​​are (0, 1, 2, 3, 4, 5), a total of 6 values, corresponding to the following templates for lower garments: random editing template, shorts editing template, short skirt editing template, long skirt editing template, and long pants editing template. For example, if the front of the top needs to be edited, corresponding to the number 3, and the skirt needs to be edited, corresponding to the number 2, then the style editing control word W = 32.

[0149] (3) Style Transformation and Fusion Units: There are a total of 8 units. These 8 units have identical functions, differing only in the image feature size they process, ranging from 4×4 and 8×8 to 512×512 pixels. The size variation follows a 2...n ×2 n , where n∈[2,9].

[0150] (3) Style block editing template: There are 16 area blocks for the top and 16 for the bottom, for a total of 32 blocks. The size is (W0 / 16, H0 / 16), and the unit is image pixels.

[0151] Beneficial effects:

[0152] (1) Application of virtual try-on for overall or partial clothing change: The system of the present invention can complete the virtual try-on application of a single person's suit. At the same time, the upper or lower garment can be virtually tried on separately with the waist as the clothing boundary, so as to complete the overall or partial change of clothing.

[0153] (2) Applicable to the application of changing and editing suits in virtual try-on: This invention can complete the application of changing suits in virtual try-on, and the suit can maintain the consistency of style and texture. Furthermore, the style can also be edited and modified according to the input style control text to form a customized virtual try-on suit result.

[0154] (3) Application for changing clothes for video platform anchors: This invention allows video platform anchors to change clothes at any time during their programs. It can also be used as a plugin for various short videos to achieve the entertainment purpose of changing clothes.

[0155] (4) Applicable to clothing change application for clothing matching on shopping platforms: This invention can provide clothing change operation for clothing matching on shopping platforms, so that users can observe the effect of the clothes they want to buy after wearing them, thereby providing an intuitive and visual auxiliary function for purchasing clothes.

[0156] In one specific example, the method for generating and editing the style of a two-stage suit in virtual try-on includes:

[0157] Step 1: Style Transfer. Input the wearer's image (S), the style reference image (F), and the style control word (W).

[0158] To ensure that the user's input image provides style information from the style reference image, the mean and standard deviation parameters of the style reference image F need to be transferred to the user's input image S. First, both inputs need to be encoded. The user's input S is processed by style encoding E1 to obtain style encoding result E1(S), where all feature attributes of E1(S) are consistent with the user's input S. Similarly, the style reference F is processed by style encoding E2 to obtain style encoding result E2(F), where all feature attributes of E2(F) are consistent with the style reference F.

[0159] The style fusion H processing unit performs transfer processing calculations on the mean and standard deviation to obtain the style fusion result.

[0160]

[0161] Here, σ() is the mean calculation function, μ() is the standard deviation calculation function, and ∈ is any positive decimal. Calculated using this formula, the style features of the style reference F can be probabilistically fused and transferred to the wearer's input S.

[0162] After the user's input S is fused with the style reference F, it needs to be processed by style decoding D to obtain the final style transfer transformation result.

[0163] T = D(H(E1(S), E2(F)))

[0164] Thus, a complete style transfer processing result image T is obtained, with a resolution size of (3, W0, H0), which represents the 3-channel information of the RGB image: red, green, and blue components, with width and height of (W0, H0) respectively.

[0165] Step 2: Style Block Editing and Style Editing Suggestion Output. The style block editing templates for the top and bottom garments are merged and fed into a 3-layer convolutional neural network, each layer having a 3×3 convolutional kernel, to calculate the style block edit W′. Simultaneously, the style transfer output T is processed through a 4-layer fully connected network, each layer having a 1×1 convolutional kernel. The fusion result is:

[0166] B′=T′+W′

[0167] Then, if B′ is fed into a 3-layer convolutional neural network, with each layer having a 3×3 convolutional kernel, the resulting pattern editing suggestion B will be obtained.

[0168] Step 3: Style Transformation and Fusion. Style transformation and fusion consists of eight style transformation and fusion units: G1, G2, G3, G4, G5, G6, G7, and G8. Each style transformation and fusion unit has the same function, consisting of a normalization network, a convolutional network, and a fusion network. The purpose is to fuse style transfer features and style editing suggestion features to achieve style editing and generation. Specifically, the normalization network performs standard batch normalization; the convolutional network is a three-layer convolutional neural network with 1×1, 3×3, and 1×1 kernels; and the fusion network uses an additive fusion method.

[0169] Here, feature decoupling L1 and feature decoupling L2 have the same function and network, both consisting of a single fully connected network, equivalent to a 1×1 convolutional kernel. Convolutional layers C1-C8 consist of eight layers, each with a 3×3 convolutional kernel. Normalization N1-N8 uses batch normalization. Upsampling R0 increases the spatial size of the style editing suggestion B to 512×512. Downsampling R1-R8 ensures that the style editing suggestion received by each style transformation fusion unit has a consistent length, width, and resolution.

[0170] G1=G1(C1(L1(T)),N1(R1(R2(...R8(L2(B))))))

[0171] G2=G2(C2(L1(Q2)),N2(R2(R3(...R8(L2(B))))))

[0172] G3=G3(C3(L1(Q3)),N3(R3(R4(...R8(L2(B))))))

[0173] G4=G4(c4(L1(Q4)),N4(R4(R5(...R8(L2(B)))))))

[0174] G5=G5(C5(L1(Q5)),N5(R5(R6(...R8(L2(B))))))

[0175] G6=G6(C6(L1(Q6)),N6(R6(R7(R8(L2(B)))))))

[0176] G7=G7(C7(L1(Q7)),N7(R7(R8(L2(B)))))

[0177] Y=G8=G8(C8(L1(Q8)),N8(R8(L2(B))))

[0178] Here, B represents the output of the pattern block editing. Therefore, after calculation by the pattern transformation and fusion unit, Y is used as the final set output image to obtain the pattern and style visualization result of the final set output.

[0179] This invention uses a convolutional neural network encoder-decoder to extract texture features and employs probabilistic fusion to synthesize new style features. It uses images of the wearer as a base, utilizes style references to generate a unified style transfer result, and further edits the result using style control text to form the final outfit output. The innovation of this invention in solving the phased reference problem of style and pattern lies not in the understanding and processing of semantics, but rather in its focus on solving the problem of custom outfit pattern editing. This invention utilizes the image's pose estimation results for the recognition and segmentation of the human body contour. The clothing blocks and their sub-blocks only define the clothing area worn by the person, not the body itself. The existence of these clothing blocks and sub-blocks allows for more precise editing of the clothing pattern results.

[0180] Example 1: Clothing Style Transfer

[0181] Input the image S (for the wearer) and the style reference image F. Provide the style control word W=40. Only the style transfer is performed on the upper garment.

[0182] Example 2: Clothing Style Editing

[0183] Input the image of the wearer (S) and the style reference image (F). Provide the style control word W=10. Only edit the top garment.

[0184] Example 3: Kit Generation

[0185] Given the input image S of the user trying on the outfit and the generated outfit, the style control word W = 14.

[0186] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0187] This application is described with reference to flowchart illustrations and / or block diagrams of the methods, apparatus, and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0188] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0189] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0190] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for generating and editing a two-stage suit style in virtual try-on, characterized in that, Based on the style of the clothing in the style reference image, the style is transferred to the try-on image, wherein the style of the clothing in the try-on image is editable; Based on the image input by the user, obtain the input style characteristics; Based on the style reference image, obtain the reference style characteristics; The reference style features are fused with the input style features to obtain the style transfer features of the image; Based on the style transfer characteristics of the image, obtain a style editing reference; Obtain the pattern block editing template corresponding to the pattern control word; The pattern block editing is obtained according to the pattern block editing template. The pattern block editing includes the pattern custom editing area and the pattern retention area of ​​the corresponding pattern block editing template. The pattern editing reference and the pattern block editing are feature-fused to obtain pattern editing block suggestion features; The style editing block suggestion feature is fused with the style transfer feature of the user's input image to obtain an output image, wherein the image is a simulated fitting image; The style control word includes a high number of digits for the upper garment and a low number of digits for the lower garment. The high number of digits for the upper garment corresponds to an editing template in the upper garment template, and the low number of digits for the lower garment corresponds to an editing template in the lower garment template. The top templates include random editing templates, collar editing templates, sleeve editing templates, and front editing templates; the bottom templates include random editing templates, shorts editing templates, short skirt editing templates, long skirt editing templates, and long pants editing templates. The corresponding pattern block editing template's custom editing area performs pattern generation operations based on the editing area specified in the template. These operations are performed through the pattern editing network, which includes... The encoding network comprises a 3-layer convolutional neural network, with convolutional kernels respectively... , and ; The decoding network comprises a 3-layer convolutional neural network, with convolutional kernels respectively. , and ; The fusion network comprises a 4-layer fully connected network with all convolutional kernels being... ; The style editing is performed by the style editing module, including: The style block editing unit is used to obtain style editing block suggestion features based on the style transfer features and style control words of the image; Upsampling unit R0 upsamples the pattern editing block suggestion features; Feature decoupling unit L2 decouples the upsampled features; The downsampling unit downsamples the decoupled features; Normalization unit, which normalizes the normalized features; Feature decoupling unit L1 decouples the style transfer features of the image; The convolutional unit performs convolution on the decoupled features; The pattern transformation fusion unit normalizes and convolves the convolved features, and then fuses the convolved features with the features normalized by the normalization unit to obtain the output image; The pattern transformation fusion unit includes a normalization unit, a convolution unit, and a fusion unit; The downsampling unit includes a downsampling unit. R 1~ R 7; The normalization unit includes a normalization unit. N 1~ N 8; The pattern transformation fusion unit includes a pattern transformation fusion unit. G 1~ G 8; The convolution includes convolution. C 1~ C 8; The input to the upsampling unit is the block proposal feature. R 0 and feature decoupling unit L 2 input connection, the feature decoupling unit L 2 Output and Normalization Unit N 8 input and downsampling units R The inputs of 7 are respectively connected to the downsampling unit. R 7 Output and Normalization Unit N 7 and downsampling unit R The inputs of 6 are respectively connected to the downsampling unit. R 6 Output and Normalization Unit N 6 and downsampling unit R The inputs of 5 are respectively connected to the downsampling unit. R 5 Output and Normalization Unit N 5 and downsampling unit R The inputs of 4 are respectively connected to the downsampling unit. R 4 Output and Normalization Unit N 4 and downsampling unit R The inputs of 3 are respectively connected to the downsampling unit. R 3 Output and Normalization Unit N 3 and downsampling unit R The inputs of 2 are respectively connected to the downsampling unit. R 2 Output and Normalization Unit N 2 and downsampling unit R The inputs of 1 are respectively connected to the downsampling unit. R 1 Output and Normalization Unit N 1. Input connection; Feature decoupling unit L The input to 1 is the style transfer features of the image, and the feature decoupling unit... L 1 and convolution C 1 input, convolution C 2 inputs, convolution C 3 inputs, convolution C 4 inputs, convolution C 5 inputs, convolution C 6 inputs, convolution C 7 inputs and convolution C 8-input parallel connection; The convolution C 1. Integration unit with style transformation G The input connection of the normalization unit of 1, the pattern transformation fusion unit G The output of the normalization unit of 1 and the pattern transformation fusion unit G The input connection of the convolutional layer of 1, the pattern transformation fusion unit G The output of the convolutional layer 1 is fused with the pattern transformation unit. G The first input connection of the fusion layer of 1, the normalization unit N The output of 1 and the pattern transformation fusion unit G The second input connection of the fusion layer of 1, the pattern transformation fusion unit G Output of the fusion layer and the pattern transformation fusion unit of 1 G The first input connection of the normalization unit of 2; The convolution C 2. Integration unit with style transformation G The second input connection of the normalization unit of 2, the pattern transformation fusion unit G The output of the normalization unit of 2 and the pattern transformation fusion unit G The input connection of the convolutional layer of 2, the pattern transformation fusion unit G The output of the convolutional layer 2 is fused with the pattern transformation unit. G The first input connection of the fusion layer of 2, the normalization unit N The output of 2 is fused with the pattern transformation unit. G The second input connection of the fusion layer of 2, the pattern transformation fusion unit G Output of the fusion layer and the pattern transformation fusion unit of 2 G The first input connection of the normalization unit of 3; The convolution C 3. Integration unit with style transformation G The second input connection of the normalization unit of 3, the pattern transformation fusion unit G The output of the normalization unit of 3 and the pattern transformation fusion unit G The input connection of the convolutional layer of layer 3, the pattern transformation fusion unit G The output of the convolutional layer 3 is fused with the pattern transformation unit. G The first input connection of the fusion layer of 3, the normalization unit N The output of 3 is fused with the pattern transformation unit. G The second input connection of the fusion layer of 3, the pattern transformation fusion unit G Output of the fusion layer of layer 3 and the pattern transformation fusion unit G The first input connection of the normalization unit of 4; The convolution C 4. Integration unit with style transformation G The second input connection of the normalization unit of 4, the pattern transformation fusion unit G The output of the normalization unit of 4 and the pattern transformation fusion unit G The input connection of the convolutional layer of 4, the pattern transformation fusion unit G The output of the 4th convolutional layer is fused with the pattern transformation unit. G The first input connection of the fusion layer of 4, the normalization unit N The output of 4 is fused with the pattern transformation unit. G The second input connection of the fusion layer of 4, the pattern transformation fusion unit G Output of the fusion layer and the pattern transformation fusion unit of 4 G The first input connection of the normalization unit of 5; The convolution C 5. Integration unit with style transformation G The second input connection of the normalization unit of 5, the pattern transformation fusion unit G The output of the normalization unit of 5 and the pattern transformation fusion unit G The input connection of the convolutional layer of 5, the pattern transformation fusion unit G The output of the convolutional layer 5 is fused with the pattern transformation unit. G The first input connection of the fusion layer of 5, the normalization unit N The output of 5 is fused with the pattern transformation unit. G The second input connection of the fusion layer of 5, the pattern transformation fusion unit G Output of the fusion layer and the pattern transformation fusion unit of 5 G The first input connection of the normalization unit of 6; The convolution C 6. Integration unit with style transformation G The second input connection of the normalization unit of 6, the pattern transformation fusion unit G The output of the normalization unit of 6 and the pattern transformation fusion unit G The input connection of the convolutional layer of 6, the pattern transformation fusion unit G The output of the convolutional layer 6 is fused with the pattern transformation unit. G The first input connection of the fusion layer of 6, the normalization unit N The output of 6 is fused with the pattern transformation unit. G The second input connection of the fusion layer of 6, the pattern transformation fusion unit G Output of the fusion layer of 6 and the pattern transformation fusion unit G The first input connection of the normalization unit of 7; The convolution C 7. Integration unit with style transformation G The second input connection of the normalization unit of 7, the pattern transformation fusion unit G The output of the normalization unit of 7 and the pattern transformation fusion unit G The input connection of the convolutional layer of 7, the pattern transformation fusion unit G The output of the convolutional layer 7 is fused with the pattern transformation unit. G The first input connection of the fusion layer of 7, the normalization unit N The output of 7 and the pattern transformation fusion unit G The second input connection of the fusion layer of 7, the pattern transformation fusion unit G Output of the fusion layer of 7 and the pattern transformation fusion unit G The first input connection of the 8-bit normalization unit; The convolution C 8. Integration unit with style transformation G The second input connection of the normalization unit of 8, the pattern transformation fusion unit G The output of the normalization unit of 8 and the pattern transformation fusion unit G The input connection of the 8-convolutional layer, the pattern transformation fusion unit G The output of the 8-convolutional layer is fused with the pattern transformation unit. G The first input connection of the fusion layer of 8, the normalization unit N The output of 8 and the pattern transformation fusion unit G The second input connection of the fusion layer of 8, the pattern transformation fusion unit G The output of the fusion layer of layer 8 is the output image.

2. The method for generating and editing two-stage suit styles in virtual try-on according to claim 1, characterized in that, The pattern block editing template corresponding to the pattern control word is input into the first convolutional neural network consisting of three layers of convolutions, each layer being... Convolutional kernels, the first convolutional neural network outputs a block-based editing pattern; The style transfer features of the image are input into a fully connected network consisting of four fully connected layers, each layer being... Convolutional kernels, the output pattern editing reference of the fully connected network; The style editing reference and the style editing obtained by feature fusion of the style block editing are merged. The merged style editing is then input into a second convolutional neural network consisting of three layers, each of which is... The convolutional kernel is the output pattern editing block suggestion feature of the second convolutional neural network.

3. An electronic device, the electronic device comprising: One or more processors, a memory, and one or more programs; wherein the one or more programs are stored in the memory, and the one or more programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method of any one of claims 1-2.

4. A computer-readable storage medium comprising a computer program that, when executed on an electronic device, causes the electronic device to perform the method of any one of claims 1-2.

Images