System and method for automatically generating three-dimensional virtual garment models using product descriptions

Through computing devices, identify clothing types and attributes, segment images, fit them to 3D templates and register them to 3D mannequins, the problem of lack of 3D clothing models on e-commerce platforms is solved, and high-quality 3D clothing model generation is achieved, which improves consumption Experience it.

CN112970047BActive Publication Date: 2024-11-26BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980072771.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-10-25
Filing Date
2019-11-14
Publication Date
2024-11-26
Estimated Expiration
2039-11-14

AI Technical Summary

Technical Problem

When e-commerce platforms sell fashion products, they lack the representation of three-dimensional (3D) clothing models, and cannot effectively display the shape and structure of clothing, affecting consumers' purchasing decisions.

Method used

Provide the description and images of clothing through computing devices, identify clothing types and attributes, segment images, determine landmarks, fit to 3D clothing templates, register to 3D mannequin models, and generate 3D clothing models through iterative projection and comparative refinement.

Benefits of technology

It realizes the automation of high-quality 3D clothing models, helping consumers better understand the shape and structure of clothing and improve the purchasing experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112970047B_ABST
    Figure CN112970047B_ABST
Patent Text Reader

Abstract

A method and system for generating a three-dimensional (3D) model of a garment. The method includes: providing an image of the garment; generating a 3D garment representation based on the image; registering the 3D garment representation to the 3D model to obtain a preliminary 3D garment model; projecting the preliminary 3D garment model worn on the 3D model to a 2D projection image; and comparing the 2D projection image with the image of the garment to refine the preliminary 3D garment model to obtain a final 3D model of the garment.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application Serial Nos. 62 / 767,214 and 62 / 767,220, filed on November 14, 2018, and U.S. Patent Application Serial No. 16 / 664,689, filed on October 25, 2019, which are incorporated herein by reference in their entirety.

[0003] Several references, which may include patents, patent applications, and various publications, are cited and discussed in the description of the present disclosure. The citation and / or discussion of these references is provided solely to clarify the description of the present disclosure, and is not an admission that any reference is "prior art" to the disclosure described herein. All references cited and discussed in this specification are incorporated herein by reference in their entirety, and to the same extent as if each reference were individually incorporated by reference. Technical Field

[0004] The present disclosure relates generally to generating three-dimensional (3D) virtual garment models, and more particularly to systems and methods for automatically generating 3D virtual garment models using both textual descriptions and image representations of products. Background Art

[0005] The background technology provided herein is for the purpose of generally presenting the context of the present disclosure. No admission is made, either explicitly or implicitly, that the work of the presently named inventors within the scope described in the background technology section, and aspects of the description that may not be prior art at the time of filing, are prior art against the present invention.

[0006] E-commerce is already an important part of social life, and there are millions of fashion products for sale on online retail platforms such as amazon.com, jd.com and alibaba.com. Although the platforms usually provide pictures of clothes and pictures of clothes worn on models, there is no 3D representation of clothing on fashion models.

[0007] Therefore, there is a need in the art to address the above-mentioned deficiencies and inadequacies. Summary of the invention

[0008] In some aspects, the present disclosure relates to a method for generating a three-dimensional (3D) model of a garment (i.e., a 3D garment model). The method comprises:

[0009] providing, by a computing device, a description and at least one image of the garment;

[0010] identifying, by a computing device, the type and attributes of the garment based on the description;

[0011] classifying, by a computing device, the at least one image to obtain a category of the at least one image as a garment image including a garment or a model image including a garment worn on a first model;

[0012] segmenting, by a computing device, at least one image based on the type of clothing and the category of the image to obtain a mask of the clothing;

[0013] determining, by a computing device, landmarks of a garment in at least one image using the mask, and parsing the garment in at least one image into a plurality of parts, wherein the parsing may or may not use the landmarks;

[0014] fitting, by a computing device, the landmarks to a 3D garment template corresponding to the type of garment to obtain a 3D garment representation;

[0015] registering, by a computing device, the 3D garment representation to the second model to obtain a preliminary 3D garment model, wherein the 3D garment model is worn on the second model, and the second model is a 3D human body model;

[0016] projecting, by a computing device, the preliminary 3D clothing model worn by the second model onto at least one 2D projection image; and

[0017] The at least one 2D projection image is compared with a corresponding one of the at least one model image by a computing device to refine the preliminary 3D garment model to obtain a 3D garment model.

[0018] In some embodiments, the type of clothing is a category of clothing including sweaters, shirts, skirts, T-shirts, and pants, and the attributes of clothing include the color, material, neckline, and gender of the clothing. In some embodiments, the step of identifying the type and attributes of clothing is performed by retrieving the type and attributes from a product database or by extracting the type and attributes from a shopping web page for clothing.

[0019] In some embodiments, the step of classifying at least one image is performed by a neural network, the neural network comprising cascaded convolutional blocks for extracting semantic features from the at least one image, and each of the convolutional blocks comprises a 2D convolutional layer, a maximum pooling layer, and a batch normalization layer.

[0020] In some embodiments, segmenting the at least one image includes segmenting the at least one image using graph cuts when the at least one image is classified as an image of clothing.

[0021] In some embodiments, the step of segmenting the at least one image includes: when the at least one image is classified as a model image, using a faster regional convolutional neural network (Faster R-CNN) to locate a bounding box of a clothing region surrounding the at least one image, and using a fully convolutional neural network (FCN) to segment the at least one image.

[0022] In some embodiments, the step of determining landmarks is performed using a neural network.

[0023] In some embodiments, the steps of determining landmarks and resolving the garment include deforming a 2D garment template against a mask, wherein the 2D garment template is retrieved based on the type of garment and includes landmark points and a resolution of the 2D garment template.

[0024] In some embodiments, the method further comprises:

[0025] After classifying at least one image as a model image, estimating a pose of a first model in the model image; and

[0026] Before the projecting step, the second model is arranged in the estimated pose of the first model in the model image.

[0027] In some embodiments, the method further includes rendering textures on the preliminary 3D garment model, the final 3D garment model, and an intermediate 3D garment model between the preliminary 3D garment model and the final 3D garment model.

[0028] In some aspects, the present disclosure relates to a method for generating a three-dimensional (3D) garment model of a garment. In some embodiments, the method includes:

[0029] providing, by a computing device, at least one image of a garment;

[0030] generating, by a computing device, a 3D garment representation based on the at least one image;

[0031] registering the 3D garment representation to the 3D model by a computing device to obtain a preliminary 3D garment model worn on the 3D model;

[0032] projecting a preliminary 3D clothing model worn on the 3D model onto at least one 2D projection image; and

[0033] The at least one 2D projection image is compared with a corresponding one of the at least one image to refine the preliminary 3D garment model to obtain a final 3D garment model of the garment.

[0034] In some embodiments, the method further comprises: classifying at least one image of the garment into a garment image including only the garment and a model image including the garment worn on the first model. In some embodiments, the method further comprises: estimating a pose of the first model in the model image to obtain an estimated pose; and setting the 3D model in the estimated pose. In some embodiments, the first model is a 2D model and the 3D model is a 3D human body model.

[0035] In some embodiments, the method further comprises, before generating the 3D garment representation, determining landmarks of the garment in the at least one image, and fitting the landmarks to the 3D garment template to obtain the 3D garment representation.

[0036] In some embodiments, the method further includes identifying the type and attributes of the garment based on the description of the garment.

[0037] In some embodiments, the steps of generating, registering, projecting, and comparing are performed iteratively until the difference between the 2D projection image and the model image is less than a predetermined value.

[0038] In some aspects, the present disclosure relates to a system for generating a 3D clothing model. The system includes a computing device. The computing device has a processor and a storage device storing computer executable code. When the computer executable code is executed at the processor, it is configured to perform the above method.

[0039] In some aspects, the present disclosure relates to a non-transitory computer-readable medium storing computer executable code. When the computer executable code is executed at a processor of a computing device, it is configured to perform the above method.

[0040] These and other aspects of the disclosure will become more apparent from the following detailed description of the preferred embodiments taken in conjunction with the accompanying drawings, although various changes and modifications may be made therein without departing from the spirit and scope of the novel concepts of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings illustrate one or more embodiments of the present disclosure and, together with the written description, serve to explain the principles of the present disclosure. Wherever possible, the same reference numerals are used throughout the drawings to refer to the same or similar elements of the embodiments.

[0042] Figure 1 A system for automatically generating a 3D virtual clothing model according to certain embodiments of the present disclosure is schematically depicted.

[0043] Figure 2A A clothing image containing only clothing is shown.

[0044] Figure 2B An image of a model including clothing worn on the model is shown.

[0045] Figure 2C A neural network architecture for image classification according to certain embodiments of the present disclosure is schematically shown.

[0046] Figure 3A A neural network architecture for segmenting clothing images according to certain embodiments of the present disclosure is schematically illustrated.

[0047] Figure 3B Schematically illustrates the segmentation of a model image through a neural network architecture according to certain embodiments of the present disclosure.

[0048] Figure 4A Landmark recognition and garment parsing for garment images are schematically depicted in accordance with certain embodiments of the present disclosure.

[0049] Figure 4B Schematically depicted deforming a garment template for a garment mask in accordance with certain embodiments of the present disclosure.

[0050] Figure 5 Schematically depicted are landmark recognition and garment parsing for model images, according to certain embodiments of the present disclosure.

[0051] Figure 6 Schematically depicts pose estimation of a model image according to certain embodiments of the present disclosure.

[0052] Figure 7 Garment representations according to certain embodiments of the present disclosure are schematically shown.

[0053] Figure 8 Schematically depicts texture mapping from a 2D flat texture to a 3D texture model according to certain embodiments of the present disclosure.

[0054] Fig. 9 A method for generating a 3D clothing model according to certain embodiments of the present disclosure is schematically depicted.

[0055] Fig.10 The workflow of automatically generating a 3D clothing model according to some embodiments of the present disclosure is schematically depicted.

[0056] Fig.11 The workflow of automatically generating a 3D clothing model according to other embodiments of the present disclosure is schematically depicted. DETAILED DESCRIPTION

[0057] The present disclosure is described in more detail in the following examples intended only for illustration, as several modifications and variations therein will be apparent to those skilled in the art. Various embodiments of the present disclosure will now be described in detail. With reference to the accompanying drawings, similar reference numerals indicate similar components throughout the accompanying drawings. As used in the description herein and throughout the appended claims, unless the context clearly indicates otherwise, the meaning of "one", "an", and "the" includes plural indicators. In addition, as used in the description herein and throughout the appended claims, unless the context clearly indicates otherwise, the meaning of "in..." includes "in..." and "on...". In addition, for the convenience of the reader, a title or subtitle may be used in the specification, which does not affect the scope of the present disclosure. In addition, some terms used in this specification are defined more specifically below.

[0058] The terms used in this specification generally have their ordinary meaning in the art in the context of the present disclosure and in the specific context in which each term is used. Certain terms used to describe the present disclosure are discussed below or elsewhere in the specification to provide additional guidance to implementers related to the description of the present disclosure. It should be recognized that the same thing can be stated in more than one way. Therefore, alternative language and synonyms can be used for any one or more of the terms discussed herein, and no special meaning is applied to whether the terms are elaborated or discussed in detail in this article. Synonyms for certain terms are provided. The enumeration of one or more synonyms does not exclude the use of other synonyms. The use of examples (including examples of any terms discussed herein) anywhere in this specification is only illustrative, and in no way limits the scope and meaning of the present disclosure or any exemplary term. Similarly, the present disclosure is not limited to the various embodiments given in this specification.

[0059] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the same meaning as in the context of the relevant art and the present disclosure, and should not be interpreted as an ideal or overly formal meaning unless explicitly so defined herein.

[0060] As used herein, "approximately", "about", "substantially" or "approximately" shall generally mean within 20%, preferably 10%, and more preferably 5% of a given value or range. The numerical values ​​given herein are approximate, meaning that if not explicitly stated, the term "approximately", "about", "substantially" or "approximately" can be inferred.

[0061] As used herein, "plurality" means two or more.

[0062] As used herein, the terms "include", "comprising", "carrying", "having", "containing", "involving", etc. should be understood as open-ended, meaning including but not limited to.

[0063] As used herein, the phrase at least one of A, B, and C should be construed to mean a logical (A or B or C), using a non-exclusive logical "or." It should be understood that one or more steps within a method can be performed in a different order (or simultaneously) without altering the principles of the present disclosure. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0064] As used herein, the term "module" may refer to, belong to or include an application specific integrated circuit (ASIC), an electronic circuit, a combinational logic circuit, a field programmable gate array (FPGA), a processor (shared, dedicated or group) that executes code, other suitable hardware components that provide the functionality, or a combination of some or all of the above, such as in the form of a system on a chip. The term "module" may include a memory (shared, dedicated or group) that stores code executed by a processor.

[0065] The term "code" as used herein may include software, firmware and / or microcode, and may refer to a program, a routine, a function, a class and / or an object. The term "shared" as used above means that a single (shared) processor may be used to execute some or all codes from multiple modules. In addition, some or all codes from multiple modules may be stored by a single (shared) memory. The term "group" as used above means that a group of processors may be used to execute some or all codes from a single module. In addition, a group of memories may be used to store some or all codes from a single module.

[0066] The term "interface" as used herein generally refers to a communication tool or component located at an interaction point between components for performing data communication between components. Typically, an interface can be applicable to both the hardware and software levels, and can be a unidirectional or bidirectional interface. Examples of physical hardware interfaces can include electrical connectors, buses, ports, cables, terminals, and other I / O devices or components. For example, the components that communicate with the interface can be multiple components or peripheral devices of a computer system.

[0067] The present disclosure relates to computer systems. As depicted in the accompanying drawings, computer components may include physical hardware components (which are shown as solid-line boxes) and virtual software components (which are shown as dashed-line boxes). Those of ordinary skill in the art will recognize that, unless otherwise stated, these computer components may be implemented in the form of software, firmware or hardware components, or a combination thereof, but are not limited to the above forms.

[0068] The devices, systems, and methods described herein may be implemented by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions stored on a non-transitory tangible computer-readable medium. The computer programs may also include stored data. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory, magnetic storage, and optical storage.

[0069] Hereinafter, the present disclosure will be described in more detail with reference to the accompanying drawings showing embodiments of the present disclosure. However, the present disclosure can be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein; on the contrary, these embodiments are provided to make the present disclosure comprehensive and complete and to fully convey the scope of the present disclosure to those skilled in the art.

[0070] Figure 1 A system for automatically generating a 3D virtual clothing model according to certain embodiments of the present disclosure is schematically depicted. Figure 1 As shown, system 100 includes computing device 110. In some embodiments, Figure 1 The computing device 110 shown may be a server computer, a cluster, a cloud computer, a general-purpose computer, a headless computer, or a special-purpose computer that provides a posture tracking service. The computing device 110 may include, but is not limited to, a processor 112, a memory 114, and a storage device 116. In certain embodiments, the computing device 110 may include other hardware components and software components (not shown) for performing its corresponding tasks. Examples of these hardware and software components may include, but are not limited to, other required memories, interfaces, buses, input / output (I / O) modules or devices, network interfaces, and peripherals.

[0071] Processor 112 may be a central processing unit (CPU) configured to control the operation of computing device 110. Processor 112 may execute an operating system (OS) or other applications of computing device 110. In some embodiments, computing device 110 may have more than one CPU as a processor (e.g., two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs).

[0072] The memory 114 may be volatile memory (e.g., random access memory (RAM)) used to store data and information during operation of the computing device 110. In some embodiments, the memory 114 may be a volatile memory array. In some embodiments, the computing device 110 may operate on more than one memory 114.

[0073] In some embodiments, computing device 114 may also include a graphics card to assist processor 112 and memory 114 with image processing and display.

[0074] Storage device 116 is a non-volatile data storage medium for storing an OS (not shown) and other applications of computing device 110. Examples of storage device 116 may include non-volatile memory, such as flash memory, a memory card, a USB drive, a hard drive, a floppy disk, an optical drive, or any other type of data storage device. In some embodiments, computing device 110 may have multiple storage devices 116, which may be the same storage device or different types of storage devices, and applications of computing device 110 may be stored in one or more of the storage devices 116 of computing device 110.

[0075] In this embodiment, processor 112, memory 114, storage device 116 are components of computing device 110, such as a server computing device. In other embodiments, computing device 110 may be a distributed computing device, and processor 112, memory 114, and storage device 116 are shared resources from multiple computers in a predetermined area.

[0076] The storage device 116 includes a 3D clothing model generation application 118, etc. The 3D clothing model generation application 118 includes a clothing recognition module 120, an image type classification module 122, a clothing segmentation module 124, a landmark identification and clothing parsing module 126, a landmark identification and model clothing parsing module 128, a posture estimation module 130, a clothing representation module 132, a model representation module 134, a texture mapping module 136, and a reprojection module 138. In some embodiments, the storage device 116 may include other applications or modules required for the operation of the 3D clothing model generation application 118. It should be noted that the modules 120-138 are all implemented by computer executable code or instructions, or data tables or databases, wherein the computer executable code or instructions, or data tables or databases together form an application. In some embodiments, each of the modules may also include sub-modules. Alternatively, some of the modules can be combined into a stack, for example, the landmark recognition and clothing parsing module 126 and the landmark recognition and model clothing parsing module 128 can be combined into one module, and the landmark recognition and model clothing parsing module 128 and the posture estimation module 130 can be combined because they share information and can be trained jointly. In other embodiments, certain modules can be implemented as circuits rather than executable code. In some embodiments, the 3D clothing model generation application 118 may also include: a user interface for a user to monitor and adjust parameters for the 3D clothing model generation application 118; and / or a scheduler for managing images and text processed by different modules of the 3D clothing model generation application 118; and / or a clothing dataset, image dataset, or other type of dataset for training one or more of the modules 120-138.

[0077] The products processed by the 3D clothing model generation application 118 are clothing provided by an e-commerce platform. The data of clothing on the e-commerce platform includes product information and images. The product information of clothing may include clothing types and attributes of clothing. The clothing type indicates the type of clothing, for example, "sweater", "shirt", "pants". The attributes of clothing characterize clothing by gender, neckline, material, color, etc., and examples of clothing attributes may include "women's (gender characteristics)", "V-neck (neckline characteristics)", "wool (material characteristics)" and "red (color characteristics)". The image of clothing may include one or more clothing images showing only clothing, and / or one or more model images showing clothing worn by a model. In some embodiments, the product information and images of clothing are stored in a corresponding product database, and the clothing type, attributes and images of clothing can be retrieved from the database using the identification of the clothing (e.g., its stock keeping unit (SKU)). In some embodiments, the product information and images of clothing are displayed in the homepage on the e-commerce platform, the title statement or description part may include clothing type and attributes, and the product information and images can be retrieved or crawled from the homepage.

[0078] The clothing identification module 120 is configured to retrieve or receive product information of clothing from a corresponding clothing entry in a product database or from a homepage of clothing in an e-commerce platform. When the clothing information is stored in a product database and the database is accessible by the clothing identification module 120, the clothing information is directly retrieved from the product database. In some embodiments, the clothing identification module 120 includes an application programming interface (API) for querying and retrieving clothing types and attributes from a product database. In one example, the product types and attributes (inputs) in the product database are as follows:

[0079] Clothing Type: Sweater

[0080] Gender: Women's

[0081] Neckline: V-neck

[0082] Material: Wool

[0083] Color: Red

[0084] The output of the clothing recognition module 120 after querying the database is:

[0085] Clothing Type: Sweater

[0086] Gender: Women's

[0087] Neckline: V-neck

[0088] Material: Wool

[0089] Color: Red

[0090] When the product database is inaccessible, the clothing identification module 120 is configured to retrieve clothing types and attributes from the homepage of the clothing. In some embodiments, the data retrieved from the homepage is the title sentence of the clothing. In other embodiments, the data retrieved from the homepage may also include other descriptions of the clothing. In some embodiments, after obtaining the title sentence of the clothing, a slot filling algorithm is used to obtain the type and attributes from the product description. In some embodiments, a slot lot filling algorithm based on a simple template is used, where the slot corresponds to the specified entity (i.e., clothing type and attribute) that is expected to be extracted. In other embodiments, more advanced algorithms may also be used, for example, a knowledge base population (KBP). The public content of KBP (https: / / nlp.stanford.edu / projects / kbp / ) is incorporated herein by reference in its entirety. In one embodiment, a dictionary is maintained for a list of clothing types (e.g., {sweater, shirt, skirt, T-shirt, pants, ...}, and a specified entity of clothing type (sweater) is identified by directly matching the noun of the sentence with the words in the catalog. Other attributes are extracted in a similar manner. For example, red is extracted as the color and wool is extracted as the material.

[0091] In one example, the product title is the text "Women's Red V-Neck Wool Sweater," and the clothing type and attributes determined by slot filling are:

[0092] Clothing Type: Sweater

[0093] Gender: Women Neckline: V-neck

[0094] Material: Wool

[0095] Color: Red

[0096] The clothing recognition module 120 is also configured to send the information to the landmark recognition and clothing parsing module 126, the landmark recognition and model clothing parsing module 128, and the clothing representation module 132 after obtaining the clothing type and attributes. In some embodiments, the obtained clothing type and attributes may not be sent to other related modules. Instead, the results are stored in a specified location and can be accessed by the module that needs the results.

[0097] The image type classification module 122 is configured to retrieve or receive an image of a garment from a corresponding garment entry in a product database or from a homepage of the garment in an e-commerce platform, and classify the image into a garment image that includes only the garment or a model image that includes the garment worn on a fashion model. The model image may also be referred to as a garment image on a model or a garment model image. In some embodiments, the classification of the image is performed by machine learning. Figure 2A A clothing image containing only clothing is shown. Figure 2B An image of a model including clothing worn on the model is shown. Figure 2C Schematically illustrates a neural network architecture for image classification according to some embodiments of the present disclosure. This neural network is also called an image classifier, which may correspond to the image type classification module 122 or a substantial part of the image type classification module 122. Figure 2C As shown, the image classifier 122 includes cascaded convolution blocks to classify the image (e.g., Figure 2A The clothing images shown and Figure 2B The image classifier 122 is a 2D convolutional layer that extracts semantic features from the model image shown in the figure. Each convolution block contains a 2D convolution layer, a maximum pooling layer, and a batch norm layer. The extracted features are then fed to two full convolution layers for feature transformation, and a probability distribution is generated using a softmax activation function. The image classifier 122 is trained using pre-labeled data before processing the image. In some embodiments, the input of the image classifier 122 is the RGB value of the image pixel, and the output is a probability distribution representing the probability that the input image belongs to category 0 or category 1, where 0 represents a clothing image containing only clothing, and 1 represents a model image containing clothing on a fashion model. When the image is classified, the image type classification module 122 is also configured to send the classification result and the image to the clothing segmentation module 124. In some embodiments, the image type classification module 122 is also configured to: when it is determined that the image is a model image, send the determination result and the model image to the posture estimation module 130.

[0098] The segmentation module 124 is configured to segment the image to obtain a clothing mask surrounding the clothing area in the clothing image or the model image when receiving the image category and the image from the image type classification module 122. In some embodiments, the segmentation of the image is performed by machine learning. Figure 3A Schematically illustrates a neural network architecture for image segmentation according to certain embodiments of the present disclosure. Figure 3AAs shown, the neural network architecture includes ResNet101 as the backbone network to extract shared low-level feature maps from the original image. Then, in order to obtain global context information, the Atrous Spatial Pyramid Pooling (ASPP) module consisting of multiple convolutional and pooling layers is applied after the last level output of ResNet 101, followed by 1x1 convolution and upsampling. The upsampled features are fed into the final 3x3 convolutional layer and a segmentation mask is generated as a binary 2D matrix. Figure 3A The segmentation of the clothing image is shown in Figure 2. The segmentation of the model image is similar. Figure 3B As shown, the processing of the model image by the segmentation module 124 outputs a mask of the clothing worn on the model. For clothing image segmentation, the parameters of the segmentation architecture are obtained by training using labeled clothing images; and for model image segmentation, the parameters of the segmentation architecture are obtained by training using labeled model images. In this embodiment, the architecture parameters used are consistent with the image classification results. In some embodiments, the 3D clothing model generation application 118 can provide two segmentation modules, one for the segmentation of clothing images and one for the segmentation of model images. The two segmentation modules have the same module structure but different specific parameters. After obtaining the segmentation result as a mask of clothing from a clothing image or a model image, the clothing segmentation module 124 is also configured to send the clothing mask and the clothing image to the landmark recognition and clothing parsing module 126 or send the clothing mask and the model image to the landmark recognition and model clothing parsing module 128.

[0099] The landmark recognition and clothing parsing module 126 is configured to: upon receiving the clothing image and the mask of the clothing in the clothing image from the clothing segmentation module 124 and the clothing type from the clothing recognition module 120, identify the landmarks of the clothing and parse the mask into sub-masks. Each sub-mask is a portion of the mask corresponding to a specific portion of the clothing (e.g., left sleeve, right sleeve, collar, body, etc.). In some embodiments, the output of the landmark recognition and clothing parsing module 126 is a list of landmark points for defining the clothing image and mask marks for defining the sub-mask area or portion in the mask. In some embodiments, the 3D clothing model generation application 118 provides predefined templates for each clothing type. Each predefined template is marked with landmark points and is segmented into different parts. When the clothing type is obtained from the clothing recognition module 120, the corresponding predefined template is retrieved. The landmark recognition and clothing parsing module 126 performs a deformable image registration of the template to the clothing mask. After the deformable registration, the clothing mask has marked landmark points and segmented parts. Figure 4A Schematic depiction of landmark recognition and clothing parsing for clothing images. Figure 4AAs shown, the input is a garment image and a corresponding mask, and the output is the landmarks of the garment and the resolved different parts of the mask. In some embodiments, the landmark points are defined by their coordinates in the garment image, and each landmark has a specific identifier based on the type of garment; the pixels in the resolved different parts can be characterized by different values. For example, the pixel value of the left sleeve can be characterized as 1, the pixel value of the right sleeve can be characterized as 2, the pixel value of the collar can be characterized as 3, and the pixel value of the body can be characterized as 4. The values ​​can be stored in independent pixel channels, and the values ​​can be displayed by color for viewing. In one example, a garment image of a long-sleeved shirt is provided, and a mask of the long-sleeved shirt is obtained using the garment segmentation module 124. As shown Figure 4B As shown on the left side of the figure, the long-sleeved shirt template is marked with landmark points and divided into separate parts. Figure 4B After the clothing mask shown on the right side of , the clothing mask is marked with the fitted landmark points and divided into different parts (or sub-masks). After identifying the landmark points and parsing the mask into different parts, the landmark identification and clothing parsing module 126 is also configured to send the results to the clothing representation module 132.

[0100] The functions of the landmark recognition and model clothing parsing module 128 are similar to those of the landmark recognition and model clothing parsing module 126, except that the former processes the model image using the model clothing template, while the latter processes the clothing image using the clothing template. Specifically, the landmark recognition and model clothing parsing module 128 is configured to: upon receiving the model image from the clothing segmentation module 124 and the mask of the model clothing in the model image and the clothing type from the clothing recognition module 120, identify the landmarks of the model clothing and parse the mask into sub-masks. Each sub-mask is a portion of the mask corresponding to a specific portion of the clothing (e.g., left sleeve, right sleeve, collar, body, etc.). In some embodiments, the output of the landmark recognition and model clothing parsing module 128 is a list of mask markers for defining landmark points of the model image and for defining sub-mask areas or portions of the mask. In some embodiments, the 3D clothing model generation application 118 can provide predefined templates with different poses for each clothing type. In other embodiments, the template may not include a pose, and the landmark recognition and model garment parsing module 128 may use only the template corresponding to the garment as a starting point, and the fitting from the garment template to the model garment mask may be more difficult. As described above, each predefined template is marked with landmark points and segmented into different parts. When the garment type is obtained from the garment recognition module 120, the corresponding predefined template is retrieved. The landmark recognition and model garment parsing module 128 performs a deformable image registration of the template onto the model garment mask. After the deformable registration, the model garment mask then has marked landmark points and segmented parts. Figure 5 Schematic depiction of landmark recognition and model garment parsing for model images. Figure 5 As shown, the input is a model image and a corresponding mask, and the output is landmarks of the model garment and the resolved different parts of the mask. In some embodiments, the landmark points are defined by their coordinates in the model image, and each landmark marker has a specific identity based on the type of garment; pixels in different resolved parts can be characterized by different values, for example, 1 for the left sleeve, 2 for the right sleeve, 3 for the collar, and 4 for the body. In one example, a model image of a long-sleeved shirt is provided, and a mask of the long-sleeved shirt is obtained using the garment segmentation module 124. As shown in FIG. Figure 5 As shown, the right sleeve portion of the garment is overlaid on the body portion, and the overlap indicates the pose of the model in the model image. After identifying the landmark points and parsing the mask into different parts, the landmark identification and model garment parsing module 128 is also configured to send the results to the garment representation module 132.

[0101] The pose estimation module 130 is configured to estimate the pose of the model in the model image upon receiving the classification result (the image is a model image) from the image type classification module 122 and the model image. In some embodiments, the pose is estimated immediately after the classifier determines that the image is a model image. In some embodiments, the pose estimation neural network is jointly trained with the clothing landmark recognition because they share information. Figure 6 As shown, the input of the pose estimation module 130 is the clothing on the fashion model, and the output is the skeleton of the fashion model estimated from the input image, which can be a list of joint points. There are many different existing technologies for estimating the human pose of the fashion model in the image. In some embodiments, the pose estimation module 130 can use a pose machine (http: / / www.cs.cmu.edu / ~vramakri / poseMachines.html), in which the spatial interaction between multiple parts and the information of parts across different scales are incorporated for pose estimation. In some embodiments, the pose estimation module 130 can use an open pose (OpenPose) algorithm, in which a deep learning-based algorithm is used to jointly learn the position of the part and its association (https: / / arxiv.org / abs / 1611.08050). After obtaining the pose, the pose estimation module 130 is also configured to send the estimated pose to the reprojection module 138.

[0102] The clothing representation module 132 is configured to fit the clothing to a specific clothing template for 3D modeling upon receiving a landmark point from at least one of the landmark recognition and clothing parsing modules 126 and / or the landmark recognition and model clothing parsing modules 128, and a clothing type from the clothing recognition module 120. The 3D clothing model generation application 118 provides a clothing template based on a specific clothing type. In some embodiments, the clothing template used herein is a 3D clothing template. The input of the clothing representation module 132 includes: a clothing type, for example, "women's sweater"; and a landmark of the clothing, for example, the coordinates of "left neckline point, right neckline point, central neckline point, left shoulder point, right shoulder point, left outer cuff point, left inner cuff point, right outer cuff point, right inner cuff point, left armpit point, right armpit point, left upper pleat point (hem point), right upper pleat point", where the coordinates can be related to the origin of the predefined clothing. The output of the garment representation module 132 includes a series of parameters for modeling a specific garment: (1) chest circumference, neck circumference, raglan, armhole depth, body length, natural waist circumference, back waist length, back width, hip circumference, sleeve length, center back neck to cuff length, upper arm circumference, wrist circumference and elbow circumference; (2) a collection of vertices and surfaces representing a 3D model of the garment.

[0103] In some embodiments, a combination of image processing and geometric calculations can be used to calculate all or a subset of parameters. For example, to calculate sleeve length, if the sleeve is flat in the image, the distance between the left shoulder point and the left outer cuff point can be simply calculated. If it is not flat in the image of a fashion model wearing clothing, a contour tracking algorithm can be used to track the clothing contour from the shoulder point to the cuff point. In another example, a simple geometric structure can be used to calculate the chest circumference. An ellipse can be fitted to the horizontal part of a person's body. Then, the distance between the two armpit points is the length of the major axis. The length of the minor axis can be estimated in advance based on the major axis and a common body shape. The chest circumference can then be calculated based on the perimeter of the fitted ellipse. In some embodiments, when images of several clothing are available, the clothing representation module 132 can use the average value, median value, or weighted average from the image to calculate the parameters. In some embodiments, when both clothing images and model images are available, the clothing representation module 132 can use the greater weight of the clothing image to calculate the parameters.

[0104] Note that not all measurements can be computed from the image, but not all of them are needed for modeling. The parameters are redundant. Also note that the above measurements are all in pixels. To obtain real-world measurements, the user needs to specify the scale (e.g., a simple real sleeve length is enough to compute the scale from image space to real space).

[0105] The garment representation module 132 is configured to use these parameters to create a 3D model of the garment. One method is to use these parameters to create 2D pattern blocks, and then create a 3D model in 3D space based on these 2D blocks. This process mimics the real process of garment design. Another method is to use a template garment mesh defined for the garment type, and then deform the template mesh model in 3D space based on the size measurements. Figure 7 The garment representation according to some embodiments of the present disclosure is schematically shown, wherein the representation is a long-sleeved shirt. After obtaining the garment representation, the garment representation module 132 is further configured to send the representation to the model representation module 134.

[0106] The model representation module 134 is configured to align the landmark points (joints) of the model representation with the joints of the human body model when receiving the garment representation in the 3D space. The 3D garment model generation application 118 provides the human body model. The human body model is established based on a set of parameters. In some embodiments, the human body model is defined using the method of Li et al. (Jituo Li Et al., Fitting 3D garment models onto individual human models, Computers & Graphics, Volume 34, Issue 6, December 2010, Pages 742-755), which is incorporated herein by reference in its entirety. After establishing the human body model, the model representation module 134 is configured to align the garment representation to the human body model by aligning the joints of the human body model and the garment template. In some embodiments, for example, different body shapes can also be fitted using the method of Brouet et al. (Remi Brouet et al., Design preserving garment transfering, ACM Transactions on Graphics, Association for Computing Machinery, 2012, SIGGRAPH 2012), which is incorporated herein by reference in its entirety. The input of the model representation module 134 is the pre-selected human body parameters and the estimated clothing parameters, and the output is the clothing and body alignment. At this point, the 3D clothing model is obtained and is ready to be viewed by the user. In some embodiments, after aligning and obtaining the 3D clothing model, the model representation module 134 is also configured to: send the model representation (the human body model with the aligned clothing on the human body model) to the texture mapping module 136 for providing texture to the 3D clothing model, and / or to the reprojection module 138 for refining the 3D clothing model. In some embodiments, the 3D clothing model also needs to be refined by the reprojection module 138, and therefore the 3D clothing model is referred to as a preliminary 3D clothing model.

[0107] The texture mapping module 136 is configured to, upon receiving the model representation of the garment, the original image containing the garment, and the parsing result, map the parsed garment blocks to the flattened 2D texture image. Each parsed block is mapped to Figure 8 In some embodiments, mapping the resolved blocks to the 2D texture image requires deformation. When creating the 2D texture image, the texture mapping module 136 is also configured to map the 2D texture image to Figure 8The 3D model shown on the right side of is represented. At this point, a 3D clothing model with texture is obtained. In some embodiments, a reference texture is obtained from a clothing template, and the clothing template comes with the 3D model and the 2D flattened reference mesh. In some embodiments, in each reprojection cycle in the reprojection cycle, the intermediate 3D clothing model is processed as having texture, and the texture information is used to help refine the model. In some embodiments, texturing of the 3D clothing model is performed according to the method of Xu et al. (Xu et al., 3D Virtual Garment Modeling from RGB Images, arXiv:1908.00114v1, 31, July 2019), which is incorporated herein by reference in its entirety.

[0108] The reprojection module 138 is configured to refine the preliminary 3D clothing model with or without texture upon receiving the preliminary 3D clothing model from the model representation module 134 (aligned 3D clothing and predefined human body) or the textured preliminary 3D clothing model from the texture mapping module 136, the original model image with clothing on the fashion model and the posture from the posture estimation module 130, or the original clothing image with clothing only, and the landmarks from the landmark recognition and clothing parsing module 126. Specifically, the reprojection model 138 is configured to fit the preliminary 3D clothing model to the posture determined by the posture estimation module 130 to obtain a 2D projection of the clothing; compare the 2D projection of the clothing with the 2D model image (which is used to determine the posture); and use the difference in the comparison to improve the 3D clothing model estimation. The improvement can be performed by adjusting the parameters of the human body model parameters, the posture estimation parameters and the clothing construction parameters or by adjusting the parameters of the clothing landmark recognition parameters, the clothing parsing parameters and the clothing construction parameters. In some embodiments, the process of reprojection, comparison and parameter adjustment is an iterative process, which is iteratively performed until the difference between the 2D source image (model image) and the 2D projection image according to the estimated 3D garment model reset posture is minimized or within the difference range. In some embodiments, when the difference between the 2D source image and the 2D projection is less than a threshold, the refinement process using reprojection is determined as final. In some embodiments, the threshold is a 20% difference. In some embodiments, the threshold is a 10% difference. In some embodiments, the threshold is a 5% difference. In some embodiments, the threshold is a 1% difference. The final output is an adjusted or final 3D garment model. In some embodiments, the method described by Li et al. (Jituo Li et al., Fitting 3D garment models onto individual human models, Computers & Graphics, Volume 34, Issue 6, December 2010, Pages 742-755) is used to perform the repositioning of the 3D garment model to the estimated posture determined by the posture estimation module 130.

[0109] As described above, reprojection can be performed for a model image using posture information, or reprojection can be performed for a clothing image using landmarks. For a model image, the posture of a 2D model is estimated based on the model image, and a corresponding 2D projection is performed by setting a 3D human body model with the posture of the 2D model. For clothing images, the posture is unavailable and reprojection can still be completed similarly. Specifically, in order to iteratively improve, the 3D clothing model can be reprojected to a 2D clothing shape in the same manner as the 3D clothing model is reprojected to the projection image considering the posture. In this case, the 3D clothing model is reprojected to a 2D space based on the landmarks and the parsing results. For example, a front view of a 3D clothing model can be obtained and compared with an input front view clothing image. This may be particularly useful if there are multiple input clothing images obtained from different angles (e.g., front, back, left, right...) as described below, at which point some comparisons can be made in one iteration.

[0110] In some embodiments, the image of the garment includes: one or more garment images each including only the garment; and one or more model images each including the garment worn on a model (e.g., a two-dimensional (2D) model). When the image of the garment includes multiple images, the 2D projection image may also be one or more. The number of 2D projection images may be equal to or less than the number of images of the garment. Comparison is performed between a 2D projection image in the 2D projection image and a corresponding one of the images in the image of the garment. For example, if one of the images of the garment is a front view image, the corresponding 2D projection image for comparison is also a front view of the garment projected from the preliminary 3D garment model. Multiple images may be compared in one round of refinement. When the image of the garment includes multiple garment images: a front view image, a rear view image, a left view image, and a right view image, the corresponding 2D projection image may also include multiple projection images accordingly: a front view 2D projection image, a rear view 2D projection image, a left view 2D projection image, and a right view 2D projection image. The corresponding image-projection image pairs are compared, and the comparison results are combined to refine the 3D garment model. When both garment images and model images are available, reprojection refinement can combine the comparison results for both garment images and model images.

[0111] In the above description, the image is processed and the result is passed from one model to another. In some embodiments, the 3D clothing model generation application 118 also includes a scheduler that coordinates the process. For example, when the application 118 is in operation, the application retrieves the image and places the image in the memory 114, and then the scheduler instructs different modules to process the image, and stores the processing results in the memory 114 for the corresponding image. The processing results of one module can be accessed by other modules. The processing results can include image type, mask of clothing, landmark points of clothing, parsed blocks of clothing, posture of the model in the model image, clothing representation, set human body model, texture of the parsed blocks, etc. These processing results are then combined to provide a refined 3D clothing model for the user to view. Because the final result is a 3D clothing model, the user can view the clothing from different angles and rotate the 3D clothing model to understand the clothing well before determining whether to buy the clothing.

[0112] In addition, as described above, the application 118 provides templates (both 3D templates and 2D reference meshes) for the clothing representation module 132 and the landmark identification modules 126 and 128, and provides a human body model to the model representation module 134. This information may be pre-stored to the application 118 or stored in a database accessible by the application 118.

[0113] Fig. 9 A method for generating a 3D clothing model according to some embodiments of the present disclosure is schematically described. In some embodiments, the method comprises: Figure 1 The computing device 110 shown is implemented. It should be particularly noted that unless otherwise stated in the present disclosure, the steps of the method can be arranged in a different order and are therefore not limited to Fig. 9 The order shown.

[0114] In some embodiments, the description and image of the garment and the results of processing the description and image are sequentially passed from one module to another. In other embodiments, the description and image of the garment and the processing are loaded into or stored in the memory 114 during the operation of the application 118, and the scheduler can be used to coordinate the operation of the modules. The following method 900 is described by using the scheduler to coordinate the operation of the modules, but the present disclosure is not limited to this.

[0115] In some embodiments, the 3D garment model generation application 118 provides a garment template (and associated 2D reference mesh) for each garment type. Figure 4BOn the left side of the figure, each garment template shows the shape of the garment, which is divided into different parts or portions, for example, the left sleeve, right sleeve, collar and body of the long-sleeved shirt template. The template also includes landmark points of the garment used to characterize different portions of the template. Different portions are marked differently and can be displayed using different colors based on the markings.

[0116] In some embodiments, the 3D clothing model generation application 118 provides a 3D clothing template for each type of clothing. The 3D clothing template is a parameter for generating a 3D clothing model. For example, for a long-sleeved shirt, the 3D clothing template may include the following parameters: chest circumference, neck circumference, raglan, armhole depth, body length, natural waist circumference, back waist length, back width, high waist circumference (high girth), sleeve length, back neck center to cuff length, upper arm circumference, wrist circumference, elbow circumference, and a collection of vertices and surfaces representing the 3D clothing model. When the landmark points of the target clothing match the 3D clothing template, the parameters are updated. Therefore, a new 3D clothing model can be generated based on the updated parameters.

[0117] In some embodiments, the 3D clothing model generation application 118 provides different human body models. The human body model can be limited by gender, height, weight, age, etc. In some embodiments, the 3D clothing model generation application 118 can define several standard human body models as a starting point, for example, a child body model, a girl body model, a boy body model, a woman body model, and a man body model, and the parameters of these models can be adjusted as needed. The human body model is three-dimensional, and joints are marked in the 3D human body model. The joints of the 3D human body model can be defined to correspond to the landmark points of the clothing, and the joint / landmark point correspondence can be used to wear the clothing on the human body model.

[0118] refer to Figure 4B On the left side of the figure, each template shows the shape of a garment, and the garment template is divided into different parts or portions, for example, the left sleeve, right sleeve, collar and body of a long-sleeved shirt template. The template also includes landmark points of the garment used to characterize different portions of the template. Different portions are labeled differently and can be displayed using different colors based on the labeling.

[0119] like Fig. 9As shown, in process 902, a description of clothing and an image of clothing are provided. Specifically, the scheduler can use the identification of clothing to retrieve the description or image of clothing from the product database, or extract the description (e.g., the title of clothing) and image from the homepage of clothing. Alternatively, the scheduler can instruct the clothing recognition module 120 to retrieve or extract the clothing description, and instruct the image type classification module 122 to retrieve or extract the image of clothing. In some embodiments, the description and image of the retrieved or extracted clothing are stored in a dedicated part of the memory 114 for processing. In some embodiments, the description and image can be limited by clothing entries (e.g., "clothing 1"), and the description, image, and the following processing are all stored in the same clothing entry. In some embodiments, multiple clothing are processed in parallel, and each of the clothing is marked using a specific clothing entry.

[0120] At process 904, upon receiving the description of the garment, the scheduler loads the garment identification module 120 into the memory 114 and executes the module. The garment identification module 120, when executed, identifies the type and attributes of the garment. The type and attributes of the garment are stored in the entry for "Garment 1" in the memory 114. In some embodiments, after determining the type and attributes of the garment, the description can be deleted from "Garment 1".

[0121] At process 906, upon receiving the image of the garment, the scheduler loads the image type classification module 122 into the memory 114 and executes the module. The image type classification module 122, when executed, classifies the image as a garment image or a model image, wherein the garment image contains only garments and the model image contains garments worn on a model. The image type classification module 122 then stores the category of the image as "garment image" or "model (garment) image" to the entry "Garment 1" corresponding to the image being processed.

[0122] At process 908, when classifying the image, the scheduler loads the clothing segmentation module 124 into the memory 114 and executes the module. When the clothing segmentation module 124 is executed, the image is segmented based on the category of the image to obtain a mask of the clothing. Specifically, when the image is a clothing image, the clothing segmentation module 124 identifies the clothing area from the clothing image; when the image is a model image, the clothing segmentation module 124 can use a bounding box to surround the clothing and identify the clothing area based on the bounding box. The mask is used to define the identified clothing. The clothing segmentation module 124 then stores the mask in the entry "clothing 1" together with the image.

[0123] At process 910, when a mask of a garment is obtained, when the image is classified as a garment image, the scheduler loads the landmark recognition and garment parsing module 126 and executes the module, and when the image is classified as a model image (or a model garment image that includes a garment worn on a model), the scheduler loads the landmark recognition and model garment parsing module 128 and executes the module.

[0124] When the image is a clothing image, the landmark recognition and clothing parsing module 126 selects a 2D clothing template based on the type of clothing, deforms the 2D clothing template to fit the mask of the clothing derived from the clothing image. After the 2D clothing template is deformed, the landmark points and different parts are also moved or deformed to fit the mask. Then, the landmark recognition and clothing parsing module 126 marks the mask with the moved landmark points and deformed clothing parts.

[0125] When the image is a model image, similarly, the landmark recognition and model clothing parsing module 128 selects a 2D clothing template based on the type of clothing, deforming the 2D clothing template to fit the mask of the clothing derived from the clothing image. After the 2D clothing template is deformed, the landmark points and different parts are also moved or deformed to fit the mask. Then, the landmark recognition and model clothing parsing module 128 marks the mask with the moved landmark points and the deformed clothing parts. In some embodiments, because the 2D clothing template is very different from the mask of the clothing worn on the model, the deformation is inaccurate. In order to improve landmark recognition and parsing, in some embodiments, the landmark recognition and model clothing parsing module 128 can use the pose estimation of the model clothing to assist the process. For example, if the pose estimation module 130 determines that the right sleeve portion is covered on the body portion, the landmark recognition and model clothing parsing module 128 will fold the right sleeve portion of the 2D clothing template onto the body portion of the 2D clothing template, and then deform the 2D clothing template for the mask. In some embodiments, the 3D clothing model generation application 118 may also provide a plurality of 2D clothing templates corresponding to different poses of clothing types. Therefore, a suitable 2D clothing template may be selected based on the type of clothing and the pose of the model image for landmark point recognition and resolution.

[0126] After this process, the landmark recognition and model garment parsing module 128 stores the landmark points and the parsed parts to the entry "Garment 1".

[0127] At process 912, upon obtaining the landmark points and parsed portions of the garment, the scheduler loads the garment representation module 132 and executes the module. When executed, the garment representation module 132 selects a 3D garment template according to the type of "Garment 1", fits the landmark points to the 3D garment template, and uses the fitted and calculated parameters to generate a 3D model or 3D representation of the garment. The garment representation module 132 stores the parameters and / or 3D garment model in "Garment 1" associated with the image of the garment.

[0128] At process 914, when the 3D clothing model is obtained, the scheduler loads the model representation module 134 and executes the module. When the model representation module 134 is executed, the human body model is selected and the 3D clothing model is registered to the human body model. In some embodiments, the model representation module 134 performs registration by aligning the landmark points of the 3D clothing model with the corresponding joints of the human body model. In some embodiments, the human body model and the 3D clothing model are represented by numerical parameters, and the alignment of the landmark points and the joints can be completed using mathematical calculations. After alignment, the model representation module 134 then generates a 3D clothing model based on these parameters, in which the clothing is worn on the human body model represented in 3D.

[0129] In some embodiments, the scheduler may also instruct the texture mapping module 136 to map the texture of the garment onto the 3D garment model. Because the garment is parsed into different parts, the texture mapping module 136 may characterize each part by the corresponding material and color of each part. The texture mapping module 136 then maps the material and color to the corresponding part in the 3D garment model to obtain a 3D garment model mapped with texture. In some embodiments, the texture mapping module 136 maps the texture to each 3D garment model, including the 3D garment model generated in the above step 914, the intermediate 3D garment model during the refinement in the following step 916, and the final 3D garment model refined in the following step 916, wherein the texture information may be used to assist the refinement process. In other embodiments, the texture mapping module 136 may only map the texture to the refined final 3D garment model.

[0130] At process 916, the scheduler loads the reprojection module 138 to the memory 114 and executes the module. The reprojection module 138, when executed, iteratively refines the 3D garment model to obtain a refined final 3D garment model.

[0131] When the image of the garment related to the reprojection is a model image, the pose of the model in the model image is estimated by the executed pose estimation module 130 before the reprojection. Specifically, after classifying the image as a model image and before the step of reprojection, the scheduler loads the pose estimation module 130 into the memory 114 and executes the module. The input of the pose estimation module 130 is a model image including a garment worn on the model, and the output is a skeleton of the fashion model estimated from the input image, which may be as follows: Figure 6 In some embodiments, the pose estimation module 130 performs pose estimation on the image whenever the image is determined to be a model image. In other embodiments, the pose estimation module 130 may perform pose estimation after the model representation module 134 obtains the 3D clothing model.

[0132] When the pose of the model in the 3D clothing model and the estimated model image is available, the reprojection module 138 is executed to adjust the pose of the 3D clothing model to the pose estimated by the pose estimation module 130 according to the model image, generate a 2D projection of the 3D clothing model with the adjusted pose, compare the generated 2D projection with the original model image to obtain a difference, and use the difference as feedback to improve the 3D clothing model. The improvement includes at least one of: adjusting the human body model parameters via the model representation module 134, adjusting the pose estimation parameters via the pose estimation module 130, adjusting the landmark recognition and model clothing resolution parameters via the landmark recognition and model clothing resolution module 128, and adjusting the clothing construction parameters via the clothing representation module 132 and the model representation module 134. After adjusting the parameters, the pose estimation module 130 estimates a new pose, the landmark recognition and model clothing resolution module 128 re-identifies the landmarks and resolves the mask into sub-masks, the clothing representation module 132 generates a new 3D clothing representation, and the model representation module 134 generates a new 3D clothing model based on the new 3D clothing representation. Then, the reprojection module 138 performs a 2D projection of the 3D clothing model using the new pose, and compares the new 2D projection with the original model image. The above process can be iteratively performed for a predetermined number of cycles (e.g., 3-5 times) or iteratively performed until the difference between the current 2D projection and the original model image is less than a predetermined threshold. In some embodiments, the reprojection includes: adjusting clothing representation parameters; generating an updated 3D clothing representation; generating an updated 3D clothing model (by aligning to a human body model); and reprojecting the 3D clothing model again to obtain a 2D projection image for comparison.

[0133] In some embodiments, when the image of the garment includes only the garment image and no model image, the pose estimation step is not required, but the following reprojection can still be completed for one or more garment images. In some embodiments, when there is no model image and only the garment image exists, the reprojection module will project the 3D garment model into the 2D garment space accordingly using the estimated garment landmarks and parsing results. Using the new landmarks and new parsing results, as well as the new 3D garment model, a similar iterative process can be performed to improve the 3D garment model results. In some embodiments, the reprojection process includes: adjusting garment representation parameters; generating an updated 3D garment representation; generating an updated 3D garment model with texture; and reprojecting the 3D garment model again to obtain a 2D projection image for comparison.

[0134] In some embodiments, when a garment has multiple images, all or most of the images are processed by reprojecting and comparing accordingly, and the results from the multiple images can be averaged or weighted to obtain an accurate 3D garment model. In some embodiments, when the multiple images include at least one model image and at least one garment image, the reprojection and comparison results about the at least one model image can be set with a greater weight when adjusting the corresponding parameters.

[0135] In some embodiments, for processing of the original image (model image or garment image), parameter adjustment after each reprojection may include changing the parameters of the 3D garment model so that the current reprojected 2D image better matches the original image than the previous intermediate reprojected 2D image.

[0136] In some embodiments, the reprojection and comparison results can be used to select a better intermediate result from the previous module. For example, clothing landmark recognition can generate the best estimate result - a landmark set according to the optimization target, however, it is not a result, it can generate more than one landmark set (such as the first three best estimates). The one with the smallest reprojection error can be determined as the final one for final clothing model generation. For example, for processing using the original image (model image or clothing image), the parameter adjustment after each reprojection can include changing the parameters of at least one of the landmark recognition and clothing parsing module 126, the landmark recognition and model clothing parsing module 128, the pose estimation module 130, the model representation module 134, and the texture mapping module 136. In some examples, different sets of parameters can be used for at least one of the above modules 126-136, each set corresponding to a 2D reprojection, and a parameter set with the best match between the 2D reprojection image and the original image is maintained for the module. In one example, the sleeve length is adjusted using different parameters, and the sleeve length is maintained when the 2D reprojection image corresponding to the sleeve length best matches the original image. Then, the shirt length is adjusted using different parameters, and the one shirt length is kept when the 2D reprojected image corresponding to the one shirt length best matches the original image. By adjusting these important parameters related to, for example, sleeve length and shirt length, a refined 3D clothing model can be obtained.

[0137] Fig.10 1000 according to some embodiments of the present disclosure. Fig.10 As shown, for the target garment, garment information and garment images are retrieved. Garment recognition extracts the type and attributes of the target garment. Image classification classifies each of the images into a garment image or a model image. The garment image contains only the target garment, which can be a front view, a side view, or a back view of the target garment; and the model image contains the target garment worn on a model, and the model can have a specific pose. The pose of the model image can be estimated by pose estimation.

[0138] When the image is a clothing image, clothing image segmentation gives a mask for the target clothing. A clothing template is provided based on the type of the target clothing, and the clothing template contains landmark points and parsed parts. The clothing template is deformably registered to the mask of the target clothing image to obtain the landmark points and parsed parts of the target clothing.

[0139] When the image is a model image, the garment image segmentation gives a mask for the target garment, where the mask also indicates the pose of the model. A garment template (with or without pose) is provided based on the type of the target garment, and the garment template contains landmark points and resolved parts. The garment template is deformably registered to the mask of the target garment image to obtain the landmark points and resolved parts of the target garment.

[0140] When landmarks and parsed parts from one or more images are available, this information is used for garment representation. A 3D garment template is provided based on the type of garment and garment representation is performed by fitting landmark points to the landmark points of the 3D garment template. The garment representation is actually a set of parameters that can be used to generate a 3D representation of the target garment.

[0141] A 3D human body model is provided, and a 3D representation of a target garment is registered to the 3D human body model. The registration is performed by aligning landmark points of the garment in the 3D representation with joints of the 3D human body model so that the target 3D garment model is "worn" on the human body.

[0142] The generated 3D clothing model of the clothing may not be the optimal result. In order to improve the model, reprojection is performed. Specifically, for reprojection for the model image, the generated 3D clothing model is set to the pose estimated from the model image and then projected to 2D. Then, the 2D projection is compared with the model image, and the difference is used as feedback to adjust the parameters for pose estimation, landmark recognition and model clothing parsing, clothing representation, and parameters for adjusting the human body model. The reprojection process is iteratively performed a predetermined number of times or until the difference between the 2D projection and the original model image is less than a predetermined threshold. Therefore, a refined final 3D clothing model is obtained. For reprojection for the clothing image, no pose estimation is required, and refinement is performed by the following operations: projecting the 3D clothing model to 2D corresponding to the layout of the clothing image (e.g., front, back left or right view); comparing the 2D projection image with the original clothing image; using the comparison result to adjust at least one of the parameters of landmark recognition, parsing, clothing representation and clothing registration; and performing the steps of landmark marker recognition, parsing, clothing representation and clothing registration again to obtain an updated 3D clothing model. The reprojection process is iteratively performed a predetermined number of times or until the difference between the 2D projection and the original model image is less than a predetermined threshold. Thus, a refined final 3D clothing model is obtained.

[0143] Fig.111100 according to some embodiments of the present disclosure. In an embodiment, the 3D garment model generation application 118 includes a scheduler for coordinating the process and storing its retrieved information and results to the garment data. Instead of data flow from one module to another, all data / results are stored in the garment data, and the stored garment data can be accessed by all modules for performing their respective functions. Fig.11 As shown, the executed scheduler retrieves or instructs certain modules to retrieve target garment information, target images, and garment templates, and stores this information in garment data. The scheduler also loads various functions for processing certain garment data in a predetermined time sequence. These functions may include retrieving inputs (target information, target images, and / or results from other modules) from garment data, and storing their results in garment data.

[0144] like Fig.11 As shown, when the processor 112 executes the 3D clothing model generation application 118, the scheduler is activated. For the target clothing, the scheduler retrieves clothing information and clothing images and stores them in clothing data. For processing efficiency, the clothing data is preferably present in the memory 114, and can be in the form of a table for storing information related to clothing. After obtaining the target clothing information, the scheduler instructs the clothing recognition module 120 to perform a clothing recognition process to extract the type and attributes of the target clothing, and store the clothing type and attributes in the data. At this time, the scheduler can also retrieve the configuration of the clothing corresponding to the clothing type and store the configuration of the clothing to the clothing data. The configuration of the clothing may include a mask of the clothing, a landmark of the clothing for the mask, a block or part of the resolved mask, a 2D reference texture of the clothing for the resolved block, a 3D template of the clothing, and one or more 3D human models. The scheduler also instructs the image type classification module 122 to perform image classification on each image in the image of the target clothing, and stores the result of whether the image is a clothing image or a model image in the clothing data. In some embodiments, the input for image classification may include not only the image, but also the clothing recognition of the image. In other words, the garment recognition results can be used to assist the image classification process to determine whether the image contains only the target garment, the garment worn on the model, or not the target garment at all.

[0145] When the image is a clothing image containing only clothing, the scheduler instructs the clothing segmentation module 124 to perform clothing image segmentation. A mask is obtained for the target clothing, and a 2D clothing template including landmark points and parsed parts is deformably registered to the mask of the target clothing image to obtain landmark points and parsed parts of the target clothing. The landmark points and parsed parts of the target clothing are also stored in the clothing data.

[0146] When the image is a model image, the scheduler instructs the clothing segmentation module 124 to perform clothing image segmentation. The process gives a mask for the target clothing, wherein the mask also indicates the pose of the model. The corresponding clothing template including landmark points and parsed parts is retrieved from the clothing data and deformably registered to the mask of the model image to obtain the landmark points and parsed parts of the target clothing. In some embodiments, the scheduler also instructs the pose estimation module 130 to estimate the pose of the model in the model image. In some embodiments, clothing segmentation of the model image and pose estimation of the model in the model image can be performed together. The segmentation results and pose of the model are stored in the clothing data.

[0147] When landmarks and parsed parts from one or more images are available, the scheduler uses this information and the 3D garment template for garment representation. Specifically, garment representation is performed by fitting landmark points to the landmark points of the 3D garment template, and the result is stored in the garment data. The garment representation is actually a set of parameters that can be used to generate a 3D representation of the target garment.

[0148] The scheduler instructs the model representation module 134 to register the 3D representation of the target garment to the 3D human body model. The registration is performed by aligning the landmark points of the garment in the 3D representation with the joints of the 3D human body model so that the target 3D garment model is "worn" on the human body. After the registration, the 3D representation worn on the human body can be stored in the garment data and is referred to as a preliminary 3D garment model because the garment model may not be optimal.

[0149] To improve the preliminary garment model, the scheduler schedules a reprojection process. Specifically, the generated preliminary 3D garment model is set to a pose estimated from the model image and then projected to 2D. The 2D projection is then compared to the model image (from which the pose is estimated), and the difference is used as feedback to adjust parameters for pose estimation, landmark recognition and model garment parsing, garment representation, and for adjusting parameters for the human body model. The reprojection process is iteratively performed a predetermined number of times or until the difference between the 2D projection and the original model image is less than a predetermined threshold.

[0150] In some embodiments, a texture is rendered for each of the preliminary 3D garment model and the intermediate 3D garment model during the reprojection cycle, and the textures of these models are used to facilitate the refinement of the 3D garment model. In some embodiments, the final 3D garment model is rendered with the texture, which is beneficial to the user's viewing.

[0151] In some embodiments, the 2D projection includes a projection of the 3D garment model for an image having only the garment, and the difference is calculated in a similar manner as described above to refine the 3D garment model.

[0152] In another aspect, the present disclosure relates to a non-transitory computer-readable medium storing computer executable code. When the code is executed at a processor of a computing device, the method 900 described above may be performed. In some embodiments, the non-transitory computer-readable medium may include, but is not limited to, any physical or virtual storage medium. In some embodiments, the non-transitory computer-readable medium may be implemented as Figure 1 The computing device 110 is shown with a storage device 116 .

[0153] In summary, certain embodiments of the present disclosure provide an automatic, cost-effective method for generating a 3D garment model of a garment based on a description and a 2D image. In addition, a reprojection method is used to adjust the 3D garment model by projecting the 3D garment model of the garment into 2D, comparing the 2D projection with the original 2D image, and using the difference to adjust the parameters of the model.

[0154] The foregoing description of exemplary embodiments of the present disclosure has been presented for purposes of illustration only, and is not intended to be exhaustive or to limit the disclosure to the precise form disclosed.Many modifications and variations are possible in light of the above teachings.

[0155] The embodiments are selected and described to explain the principles of the present disclosure and their practical applications, so that other persons skilled in the art can utilize the present disclosure and the embodiments and various modifications suitable for the intended specific use. Without departing from the spirit and scope of the present disclosure, alternative embodiments will become apparent to those skilled in the art to which the present disclosure belongs. Therefore, the scope of the present disclosure is limited by the appended claims, rather than by the exemplary embodiments described above and described herein.

[0156] References:

[0157] 1.Bin Zhou et al.,Garment Modeling from a Single Image,Pacific Graphics,2013,vol 32,No.7.

[0158] 2.Shan Yang et al., Detailed Garment Recovery from a Single-ViewImage, 2016, ArXiv:1608.01250v4.

[0159] 3.Moon-Hwan Jeong et al.,Garment capture from a photograph,ComputerAnimation and Virtual Worlds,2015,vol 26,issue 3-4.

[0160] 4.Knowledge Base Population(KBP),https: / / nlp.stanford.edu / projects / kbp / 。

[0161] 5.Ziwei Liu et al.,Fashion Landmark Detection in the Wild,2016,ArXiv:1608.03049v1。

[0162] 6.https: / / www.marvelousdesigner.com / product / overview。

[0163] 7.http: / / www.cs.cmu.edu / ~vramakri / poseMachines.html。

[0164] 8.Vivek Kwatra et al.,Graphcut Textures:Image and Video SynthesisUsing Graph Cuts,ACM Transactions on Graphics(TOG),2003,vol 22,issue 3。

[0165] 9.Shaoqing Ren et al.,Faster R-CNN:Towards Real-Time Object Detectionwith Region Proposal Networks,2016,arXiv:1506.01497v3。

[0166] 10.Jituo Li et al.,Fitting 3D garment models onto individual humanmodels,Computers&Graphics,2010,vol 34,issue 6,pp 742-755。

[0167] 11.Remi Brouet et al.,Design preserving garment transfer,ACMTransactions on Graphics,2012。

Claims

1. A method for generating a three-dimensional 3D clothing model, the method comprising: providing, by a computing device, a description and at least one image of the garment; identifying, by the computing device, the type and attributes of the garment based on the description; classifying the at least one image by the computing device to obtain a category of the at least one image as a garment image including the garment or a model image including the garment worn on a first model; Segmenting the at least one image based on the type of the clothing and the category of the image by the computing device to obtain a mask of the clothing; determining, by the computing device, landmarks of the garment in the at least one image using the mask and parsing the garment in the at least one image into a plurality of parts; fitting, by the computing device, the landmarks to a 3D garment template corresponding to the type of the garment to obtain a 3D garment representation; registering the 3D garment representation to a second model by the computing device to obtain a preliminary 3D garment model worn on the second model, wherein the second model is a 3D human body model; projecting, by the computing device, the preliminary 3D clothing model worn by the second model onto at least one 2D projection image; as well as comparing, by the computing device, the at least one 2D projection image with a corresponding one of the at least one image to refine the preliminary 3D garment model to obtain the 3D garment model, The method further comprises: After classifying the at least one image as a model image, estimating a pose of the first model in the model image; as well as Before the projecting step, the second model is arranged in the estimated pose of the first model in the model image.

2. The method according to claim 1, wherein: The types of clothing are categories of clothing including sweaters, shirts, skirts, T-shirts, and pants, and the attributes of clothing include color, material, neckline, and gender of clothing.

3. The method according to claim 2, wherein: The step of identifying the type and attributes of the garment is performed by retrieving the type and attributes from a product database or by extracting the type and attributes from a shopping web page for the garment.

4. The method according to claim 1, wherein: The step of classifying the at least one image is performed by a neural network comprising cascaded convolutional blocks for extracting semantic features from the at least one image, and each of the convolutional blocks comprises a 2D convolutional layer, a maximum pooling layer, and a batch normalization layer.

5. The method according to claim 1, wherein: The step of segmenting the at least one image includes segmenting the at least one image using graph cuts when the at least one image is classified as the clothing image.

6. The method according to claim 1, wherein: The step of segmenting the at least one image includes: when the at least one image is classified as the model image, using a faster regional convolutional neural network Faster R-CNN to locate a bounding box of a clothing area surrounding the at least one image, and using a fully convolutional neural network FCN to segment the at least one image.

7. The method according to claim 1, wherein: The step of determining the landmarks is performed using a neural network.

8. The method according to claim 1, wherein: The steps of determining the landmarks and resolving the garment include deforming a 2D garment template with respect to the mask, wherein the 2D garment template is retrieved based on the type of the garment and includes landmark points and a resolution of the 2D garment template.

9. The method according to claim 1, further comprising: A texture is rendered on the 3D garment model.

10. A method for generating a three-dimensional 3D clothing model, the method comprising: providing, by a computing device, at least one image of a garment; generating, by the computing device, a 3D garment representation based on the at least one image; registering the 3D garment representation to the 3D model by the computing device to obtain a preliminary 3D garment model worn on the 3D model; projecting the preliminary 3D clothing model worn on the 3D model onto at least one 2D projection image; as well as comparing the at least one 2D projection image with a corresponding one of the at least one image to refine the preliminary 3D garment model to obtain the 3D garment model, The method further comprises: classifying the at least one image of the garment into a garment image including the garment and a model image including the garment worn on a first model, and When the at least one image is a model image, estimating a pose of the first model in the model image to obtain an estimated pose; and setting a human body model in said estimated pose, Wherein, the first model is a 2D model and the 3D model is a 3D human body model.

11. The method according to claim 10, further comprising: Before generating the 3D garment representation: determining landmarks of the garment in the at least one image, and fitting the landmarks to a garment template to obtain the 3D garment representation.

12. The method of claim 10, further comprising identifying a type and attributes of the garment based on the description of the garment.

13. The method according to claim 10, wherein: The steps of generating, registering, projecting and comparing are iteratively performed until a difference between the at least one 2D projection image and a corresponding one of the at least one image of the garment is less than a predetermined value.

14. A system for generating a 3D clothing model, the system comprising a computing device, the computing device comprising a processor and a storage device storing computer executable code, wherein: The computer executable code, when executed at the processor, is configured to perform the method according to claim 10 .

15. A non-transitory computer readable medium storing computer executable code, wherein: When the computer executable code is executed at a processor of a computing device, it is configured to: providing at least one image of a garment; generating a 3D garment representation based on the at least one image; registering the 3D garment representation to the 3D model to obtain a preliminary 3D garment model worn on the 3D model; projecting the preliminary 3D clothing model worn on the 3D model onto at least one 2D projection image; as well as comparing the at least one 2D projection image with a corresponding one of the at least one image to refine the preliminary 3D garment model to obtain a 3D garment model, wherein the computer executable code is further configured to: classify the at least one image of the garment into a garment image including the garment and a model image including the garment worn on a first model, Wherein, the computer executable code is further configured to, when the at least one image is a model image: estimating a pose of the first model in the model image to obtain an estimated pose; and setting the 3D model at the estimated pose, Wherein, the first model is a 2D model and the 3D model is a 3D human body model.