3D garment generation from 2D graffiti images

Abstract

Systems and methods for generating a two-dimensional (2D) garment from a two-dimensional (2D) graffiti image drawn by a user are described. The system includes a conditional 2D generator, a conditional 3D generator, and two intermediate media (including a dimensionally coupled color density pair and a planar point cloud) that bridge the gap between the dimensions. Given a graffiti image, a 2D generator synthesizes a dimensionally coupled color density pair including an RGB projection and a density map from a front view and a rear view of the graffiti image. A density-aware sampling algorithm transforms the 2D-dimensional coupling color density pair into a 3D planar point cloud representation, wherein depth information is ignored. The 3D generator predicts depth information from the planar point cloud. By utilizing the 2D generative model to bridge the dimension gaps in a non-parametric manner, dynamic changes in each garment due to deformations caused by the wearer's posture and irregular wrinkles and wrinkles can be bypassed.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 957,049, filed on September 30, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] Examples described herein generally relate to generating three-dimensional (3D) garments using a digital framework, and more particularly to methods and systems for generating 3D garments from two-dimensional (2D) graffiti images drawn by a user. Background Art

[0004] Clothing is an integral part of human life, influencing our culture and daily activities in various ways. In virtual and augmented reality applications, clothing is becoming a fundamental component for various tasks related to digital content creation, such as wearable human digitization, virtual try-on, fashion design, etc. At the same time, clothing prototyping remains a process that is severely hampered by the need for tedious manual labor and specialized skills.

[0005] 3D clothing generation is particularly challenging because clothing of different categories and styles not only differs in static shape and appearance, but also in the dynamics of each garment due to deformation caused by the wearer's pose. In addition, common high-frequency geometric details such as irregular wrinkles and folds are inherently difficult to model. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In the accompanying drawings, which are not necessarily drawn to scale, like numbers may describe similar components in different views. Some non-limiting examples are shown in the images of the accompanying drawings, in which:

[0007] Figure 1 is an illustration depicting the transformation of a user-provided 2D scribble into a high-fidelity 3D garment, wherein an input scribble image alone (or on a mannequin) is transformed into a point cloud corresponding to a 3D realization of the garment accessible from any view;

[0008] Figure 2A is a diagram depicting the pipeline framework in a sample configuration, where two intermediate media are used: a planar point cloud obtained by ignoring the depth dimension, and dimensionally coupled color-density pairs obtained by rasterizing and sampling the real point cloud;

[0009] Figure 2B A sample configuration is shown Figure 2A Flowchart of the operation process of the pipeline framework;

[0010] Figure 3is a diagram depicting generated graffiti samples with different levels of detail with different target numbers of segments for clustering;

[0011] Figure 4 is an illustration depicting the process of creating a planar point cloud from a graffiti image in a sample configuration;

[0012] Figure 5 is an illustration depicting the input doodle (left) converted into 3D clothing using the point cloud-based approach in a sample configuration;

[0013] Figure 6 is a table with quantitative comparisons of geometry (EMD, CD), geometry and color (EMD-c, CD-c), and consistency with input scribbles (LPIPS, ScrRecon) using point clouds containing 8K (top) and 16K (bottom) points;

[0014] Fig. 7A and 7B is a diagram depicting two examples where an input doodle (left) is converted into a volumetric 3D garment (center) using the disclosed method, followed by an output of a mesh-based approach (right);

[0015] Figure 8 is a diagram depicting the input graffiti and its corresponding point cloud (real data), followed by the point clouds obtained using uniform sampling and density-aware sampling methods in the sample configuration;

[0016] Fig. 9 is an illustration depicting multi-view renderings of different 3D garments containing 32,768 points, their input graffiti, and color density pairs;

[0017] Fig.10 is an illustration depicting the result generated from a customized graffiti image, wherein the generated color point cloud has high quality, not only in terms of shape, but also in terms of color (left) and pattern (right) designed by the user, which is consistent with the user input;

[0018] Fig.11 is an illustration depicting an application where a user draws on an image of an arbitrary pose and the corresponding 3D garment is automatically generated from the doodle;

[0019] Fig.12 is an illustration depicting synthetic samples using different numbers of target segments during processing, where the data preparation pipeline simulates different levels of scribble detail;

[0020] Fig.13is an illustration depicting a sample in which a designer is asked to provide a design for a garment, where despite a large distribution gap between the doodles provided by the designer and those used in training, the model described in this paper still produces reasonable output;

[0021] Fig.14A is an illustration depicting the generation of 3D clothing from doodles including real commercial images;

[0022] Fig. 14B is an illustration depicting a “failure case” due to domain gap, where the model interprets the heart shape as a graffiti pattern and tries to cover the garment with this pattern;

[0023] Fig.15 is a diagram depicting some observed failure cases, where the scribbles include patterns that were not learned in the training set, the scribbles fall into some distribution outside the training range and are far from real-world cases, or the input scribbles happen to be similar to some training data and the model over-interprets the input;

[0024] Fig.16 is a block diagram of a machine in which instructions (e.g., software, programs, applications, applet, applications, or other executable code) for causing the machine to perform any one or more of the methodologies discussed herein may be executed; and

[0025] Fig.17 is a block diagram illustrating a software architecture in which the examples described herein may be implemented. DETAILED DESCRIPTION

[0026] While there are various types of 3D representations for clothing, each of which is beneficial for a specific set of clothing design and synthesis tasks, point clouds can serve as an effective representation for general clothing modeling, providing the required topological flexibility and detailed expression. However, when it comes to synthesizing 3D clothing from 2D user scribbles, additional challenges arise compared to general clothing generation. First, scribble-to-clothing generation requires a domain transformation from rough, abstract 2D scribbles to detailed, realistic 3D shapes. This domain transformation has been widely studied in many 2D image-to-image translation tasks and 3D translation works, such as point cloud completion and novel view synthesis. However, the 2D-to-3D translation problem has not been relevantly explored due to ambiguities caused by dimensionality differences, lack of training data, and limited capabilities of 3D generative networks. Second, unlike the case of 2D images, for unordered representations such as 3D point clouds or meshes, the lack of a common reference frame and, therefore, a lack of efficient convolution operators makes the joint modeling of shape and color a non-trivial task in 3D synthesis applications.

[0027] Systems and methods for addressing these and other limitations are described to enable generation of 3D garments from two-dimensional (2D) graffiti images drawn by a user. The system includes a conditional 2D generator, a conditional 3D generator, and two intermediate media (including dimensionally coupled color-density pairs and a planar point cloud) that bridge the gap between dimensions. Given a graffiti image, the 2D generator synthesizes dimensionally coupled color-density pairs including RGB projections and density maps from the front and back views of the graffiti image. A density-aware sampling algorithm transforms the 2D dimensionally coupled color-density pairs into a 3D planar point cloud representation, where depth information is ignored. The 3D generator predicts depth information from the planar point cloud. By utilizing a 2D generative model to bridge the dimensionality gap in a non-parametric manner, the dynamic changes of each garment due to deformations caused by the wearer's posture and irregular wrinkles and folds can be bypassed.

[0028] Designing clothing is a complex process that is mainly performed by large fashion houses. A digital framework is needed whereby users (not just users of large fashion houses) can generate realistic 3D clothing from simple colored scribbles. As described in this paper, a "doodle" is an abstraction of a real object to be reproduced, and can include, for example, a 2D hand-drawn or computer-generated image prepared by a user. Those skilled in the art will appreciate that the conditional generation process from doodles poses substantial challenges. First, 3D clothing geometry is difficult to model due to its non-rigidity and topological variability, which is further coupled with arbitrary texture and color patterns. Another challenge is to map the abstract 2D doodles into the corresponding detailed 3D implementation. In addition, at the current state of the art, there is neither an effective model available for this task nor high-quality paired data linking the 2D doodles and the corresponding 3D implementation. To meet these needs, this paper describes a modular pipeline framework that effectively decomposes the learning task into two internal dimensional transformation tasks in 2D and 3D. The decomposition is achieved through two proposed intermediate media and bridged using an efficient density-aware sampling algorithm.

[0029] In order to generate relevant and diverse training data, this paper describes a superpixel-based doodle synthesis method. To evaluate the results and establish ablations, reasonable metrics for the problem and the selected patterns are also explored. As is evident from the following description, the resulting framework unlocks downstream applications such as 3D clothing design, where high-resolution concepts can be generated from low-resolution input doodles.

[0030] Despite the challenges of direct 2D graffiti to 3D clothing conversion, the inventors have recognized that in-domain synthesis of 2D and 3D subjects has been better explored and has produced typical applications with high-fidelity results. In view of this, a graffiti-conditioned clothing generation pipeline is described, which employs domain-specific generators for 2D graffiti and 3D point clouds. Two strategies are implemented to bridge these domains. First, a conditional 2D generator is designed so that it returns dimensionally coupled images including front color-density pairs and back color-density pairs, which can be conveniently transformed into planar point clouds using a density-aware parameter algorithm. Second, a conditional 3D diffusion-based generator is designed to lift planar point clouds to 3D, i.e., recover the depth dimension, which is robust and easy to train.

[0031] The present disclosure provides methods for jointly generating the shape and color of 3D clothing conditioned on simple user-provided scribbles, which is achieved through a modular and efficient pipeline for generating 3D points from 2D scribbles. In addition to domain-specific 2D and 3D generators, two novel intermediate representations are provided, as well as non-parametric algorithms that bridge the gap between 2D and 3D. Reasonable metrics are explored and strong baselines are extensively evaluated to quantitatively show that the described methods outperform conventional methods. The resulting framework is further quantitatively shown to produce results that are high-resolution and consistent with the input scribbles, and are therefore suitable for real-world applications such as clothing design and fitting.

[0032] In a sample configuration, the system and method described herein enable a user to generate a three-dimensional (3D) garment image from a two-dimensional (2D) garment image drawn by the user. The system includes a conditional 2D generator that receives a 2D garment image drawn by the user and converts the 2D garment image into a color density pair, a density-aware sampling algorithm that performs density-aware sampling of the color density pair to generate a planar point cloud, and a conditional 3D generator that converts the planar point cloud into an output 3D point cloud to be displayed as a 3D garment image. In a sample configuration, a 2D garment image may include a 2D hand-drawn or computer-generated abstract of a garment that defines the basic color pattern and shape of the target garment. The resulting 3D garment image can be displayed on a digital representation of a user or avatar as a virtual try-on of the 3D garment image.

[0033] Now refer to Figure 1-17 A detailed description of a method for generating 3D clothing from 2D user doodles is described. Although this specification provides a detailed description of possible implementations, it should be noted that these details are merely exemplary and in no way limit the scope of the inventive subject matter.

[0034] By way of example, Figure 1The transformation of a user-provided 2D doodle 100 into a high-fidelity 3D garment is shown, where the input doodle image 100 alone (or on a human body model 120) is transformed into a point cloud 110 corresponding to a 3D realization of the garment, which can be accessed from any view using the techniques described herein. As shown, the user-provided doodle image 100 can be drawn alone or drawn on a human body model 120 and rendered as a separate point cloud or rendered on a human body model 120.

[0035] As used herein, "graffiti" (e.g., Figure 1 The doodles in 100) usually consist of several broad strokes that define the basic color patterns and shapes of the target garment, and several finer strokes that contain finer details. c ∈R H×W×3 Represented by, where H and W are their height and width, a point cloud for the target garment containing information about the position and color of the garment is obtained (e.g. Figure 1 The point cloud 110 in is represented by P G ∈R N×(3+3) , where each point p i ∈P G , i = 1...N, each point cloud contains 3 channels. The task is cross-domain (i.e., abstract to real) and cross-dimensional (i.e., 2D to 3D). Cross-domain translation has been relatively well explored and generally produces more impressive results than methods for cross-dimensional generative tasks, such as image-to-image translation in 2D and shape completion in 3D. However, as discussed below, performing domain translation across dimensions using a single model is currently challenging.

[0036] Based on the above observations, a two-stage digital framework 200 has been designed in which the generation step is performed in-domain. Figure 2A The digital framework 200 shown in FIG. 2 can be used to generate Figure 1The invention relates to a method for generating graffiti images, and includes a conditional 2D generator 220 and a conditional 3D generator 250, as well as two intermediate media, dimensionally coupled color density pairs 210 and a planar point cloud 230, to bridge the gap between dimensions. Given a graffiti image, the 2D generator 220 synthesizes dimensionally coupled color density pairs 210, including RGB projections and density maps from the front view and the back view. The density-aware sampling algorithm 260 transforms the 2D dimensionally coupled color density pairs 210 into a 3D planar point cloud representation 230, in which the depth information is ignored. Then, the 3D generator 250 is used to predict the depth information from the planar point cloud 230. Existing 3D generation works cannot handle clothing because clothing of different categories and styles not only differs in static shape and appearance, but also varies dynamically for each piece of clothing due to deformation caused by the wearer's posture. In addition, common high-frequency geometric details (such as irregular folds and wrinkles) are inherently difficult to model. These difficulties are circumvented by utilizing a 2D generative model to bridge the dimensionality gap in a non-parametric manner.

[0037] like Figure 2A As shown, in the first stage, the dimensionally coupled color density pair B = {{I,D} f ,{I,D} r}210 is generated, which includes the front color density pair {I, D} f and the rear color density pair {I,D} r , where I∈R H×W×3 is a color image, and D∈R H×W is a scalar image describing the local density of each pixel. These are generated using the conditional 2D generator G s 220, the generator generates dimensionally coupled color density pairs B according to the input scribble (Ic): Ic→B. In order to bridge the 2D domain and the 3D domain, a parameter-free density-aware sampling method is provided, which transforms the color density pairs B 210 from the first stage into a planar point cloud (P flat ) 230, in which point positions and colors are sampled from the corresponding pair B 210. The second stage uses the conditional 3D generator G p 250 performs draping, or lifting the planar point cloud 230 to 3D (output point cloud 240), the generator from P flat Generate 3D image P G :P flat →P G Both generation stages 220 and 250 may be implemented using a diffusion-based generator.

[0038] exist Figure 2AIn the pipeline digital framework 200 of , two intermediate media are used: a planar point cloud 230 obtained by ignoring the depth dimension, and a dimensionally coupled color density pair 210 obtained by rasterizing and sampling the real point cloud. In order to bridge the cross-dimensional gap between the planar point cloud 230 and the dimensionally coupled color density pair 210, a non-parametric density-aware sampling algorithm 260 is implemented. Conditional 2D generator G s 220 processes the conversion from the graffiti image 100 to the color density pair 210, and the conditional 3D generator G p 250 performs depth estimation from the planar point cloud 230 .

[0039] Select the diffusion model as the conditional 2D generator G in the sample configuration s 220 and conditional 3D generator G p 250. However, the diffusion model is only a design choice, as the diffusion model currently demonstrates superior quality in both image and point cloud generation. It should be understood that the pipeline digital framework 200 is independent of the design choice and can benefit from any future improvements in 2D and 3D generation.

[0040] In the sample configuration, for the 2D generator G s 220, the code can be based on the Pytorch version of SR3. For the conditional background, the dimension R 3×H×W The original image is replaced by the dimension R (3+3+1+1)×H×W The dimension of the coupled color density pairs is 210. The resolution of these images is 128. The model was trained for 1,000,000 iterations using a batch size of 16. The training was performed using a linear schedule with 2000 time steps and (start, end) = (10 -6 , 10 -2 ) diffusion model.

[0041] On the other hand, for the 3D generator G p250, the code can be based on the official version of the system described by Bhatnagar et al. in "Combining implicit function learning and parametric models for 3D human reconstruction" in the 2020 European Conference on Computer Vision, the contents of which are incorporated herein by reference. Bhatnagar et al. use a point-voxel convolutional neural network (CNN) as a point cloud feature extractor. Bhatnagar et al. further combine detailed implicit functions with parametric modeling to reconstruct a 3D model of a person that remains controllable even in the presence of clothing. Given a sparse 3D point cloud of a wearer, an implicit part network (IPNet) is used to jointly predict the wearer's external 3D surface, the internal body surface, and semantic correspondences to a parametric body model (SMPL). These correspondences are used to fit a body model to the inner surface, which is then non-rigidly deformed (under the model of a parametric body plus displacement) to the outer surface to capture details of clothing, face, and hair. In the sample configuration described in this paper, noise can be added to the depth dimension during the forward (diffusion) process. In the reverse (denoising) process, the depth can be initialized with a Gaussian distribution and the depth can be gradually updated. The model is trained for 100,000 iterations using a batch size of 16. A linear schedule with 1000 time steps and (start, end) = (10 -4 ,2×10 -2 ) diffusion model.

[0042] Denoising diffusion probabilistic models (DDPMs) are described by Ho et al., “Denoising diffusion probabilistic models,” Neural Information Processing Systems, 2020, which is incorporated herein by reference. DDPMs are a class of generative models that use a formula to transform from a simple distribution (e.g., a Gaussian distribution) to a target data distribution as a denoising process. To learn the denoising process, a diffusion process gradually adds noise to the data. Both the diffusion (forward) and denoising (reverse) processes can be modeled as Markov chains. In both processes, each step can be modeled as a Gaussian transformation. From a high-level perspective, the noisy data x at each time step t is t can be viewed as a weighted combination of the original data x0 and noise ∈, where the weights are obtained from a predefined sequence of biases. Therefore, the model aims to learn the t-1 to x t Through some reparameterization, the learning objective can be further simplified to a simple mean square error between the real noise and the predicted noise ||∈θ (x t ,t)-∈|| 2 Then, the prediction noise ∈ θ (x t ,t) can be used to calculate the average value μ in the denoising process θ (x t ,t).

[0043] When the source and target domains are of inner dimensions, the conversion task is simplified. The key to decoupling the conversion task into two separate generation tasks in 2D and 3D is to use two proposed intermediate media (i.e., dimensionally coupled color density pairs 210 and planar point clouds 230) and non-parametric algorithms 260 to bridge these media. Color density pairs B 210, as 2D media, are intended to provide sufficient information for the following lifting process. As described above, these color density pairs include front color density pairs and back color density pairs. On the other hand, the planar point cloud P flat 230 is a simplified representation of a point cloud without depth information, or equivalently, a simplified representation at constant depth.

[0044] Color density pairs and planar point clouds facilitate simple cross-dimensional transformations. Non-parametric algorithms, density-aware sampling 260, approximate planar point clouds P given color density pairs B 210 flat 230. Specifically, for each pixel position (i, j), given the color density pair B 210, the corresponding center coordinate x from the pixel coordinate (i, j) is obtained. ij ,y ij and the estimated number of points n to be added for the front (f) and rear (r) input 2D images f ij ,n r ij Then, n f ij +n r ij The point is attached to the target plane point cloud 230, where the first two coordinates are used as the center coordinates perturbed by noise, the depth dimension is 0, and the color is v f ij and v r ij That is, n f ij The coordinates of the point are (N(x ij ,σ 2 ),N(y ij ,σ 2 ),0), color is v f ij , and n r ij The coordinates of the point are (N(xij ,σ 2 ),N(y ij ,σ 2 ),0) and the color is v f ij The noise perturbation allows a non-deterministic mapping to obtain the edge points of the object.

[0045] Sample pseudo code for implementing density-aware sampling 260 is shown in Algorithm 1 below. For hyperparameters, the minimum and maximum number of density points sampled at each location are set to 5 and 30 respectively, and the standard deviation σ 2 Set to 0.001, and the point cloud coordinate range x min 、x max ,y min ,y max Set to -0.2, 0.25, -0.33, 0.12. Based on the distribution of the dataset, the point cloud coordinate range is set empirically.

[0046] Algorithm 1: Density-aware sampling 260

[0047] Input: image shape is W×H color density pairs {(I f ,D f ),(I r ,D r )}

[0048] Parameter: Minimum number of points n min , the maximum number of points n max , standard deviation σ 2

[0049] Point cloud coordinate range x min 、x max ,y min ,y max

[0050] Output: Planar point cloud p flat ={(p i coord ,p i color )} N i＝1 ∈R N×(1+3+3)

[0051]

[0052]

[0053] Therefore, if Figure 2A As shown, the 3D clothing generation from the 2D user doodle 100 includes a conditional 2D generator G s 220. Conditional 3D Generator Gp 250 and density-aware sampling 260 for cross-dimensional connections. The formulas for both conditional transformations are straightforward with the help of color-density pairs 210 and planar point clouds 230. For 2D transformation, we start with the graffiti image I c 100 synthetic color density pairs B 210. For 3D conversion, from the planar point cloud P flat 230 performs depth prediction. During inference, density-aware sampling 260 converts the generated B 210 into an approximate P flat 230. Both of these transformations can be formulated using the diffusion model format as:

[0054] For G s ,p θ (B t-1 |B t ,I c )=N(S t-1 |μ θ (B t ,t,I c ),σ t 2 I),

[0055] For G p ,p θ (P t-1 |P t ,P flat )=N(P t-1 |μ θ (P t ,t,P flat ),σ t 2 I) (1)

[0056] And G s The objective function is ||∈ θ (S t ,t,I c )-∈|| 2 , G p The objective function is ||∈ θ (P t ,t,P flat )-∈|| 2 .

[0057] Figure 2B is shown in the sample configuration Figure 2A Flowchart 270 of the operational flow of the pipelined digital framework 200 .

[0058] As shown, input doodle 100 is received at 272. Input doodle 100 can be a scan of a hand-drawn image prepared by a user, or a hand-drawn image drawn on a tablet or other input device. Input doodle 100 can also be a computer-generated image prepared by a user using a conventional graphic design program.

[0059] At 274, the received input graffiti 100 is generated by the conditional 2D generator G s 220 is converted into a color-density pair B 210. As described above, given the graffiti image 100, the 2D generator 220 synthesizes a dimensionally coupled color-density pair B 210 including an RGB projection and a density map from the front view (f) and the back view (r).

[0060] The color density pair B 210 is then sampled at 276 by density-aware sampling 260. For example, the density-aware sampling at 276 may be performed by an algorithm such as Algorithm 1 described above. According to such an algorithm, for a given color density pair B 210, for each pixel position (i, j), the corresponding center coordinate x from the pixel coordinate (i, j) is obtained. ij ,y ij and the estimated number of points n to be added for the front (f) and rear (r) input 2D images f ij ,n r ij Then, n f ij +n r ij The point is appended to the target plane point cloud 230, where the first two coordinates are used as the center coordinates perturbed by noise, the depth dimension is 0, and the color is v f ij and v r ij That is, n f ij Point utilization (N (x ij ,σ 2 ),N(y ij ,σ 2 ),0) is inserted, the color is v f ij , n r ij Point using coordinates (N(x ij ,σ 2 ),N(y ij ,σ 2 ),0) is inserted, the color is v r ij The noise perturbation allows a non-deterministic mapping to obtain the edge points of the object. The result is a planar point cloud P as described above. flat230.

[0061] At 278, the plane point cloud P flat 230 by conditional 3D generator G p 250 is converted into an output 3D point cloud 240. The 3D generator may include a diffusion model that predicts depth information from a planar point cloud. p 250 A point voxel convolutional neural network (CNN) can be used as a point cloud feature extractor. In the forward (diffusion) process, noise can be added to the depth dimension. In the reverse (denoising) process, the depth can be initialized with a Gaussian distribution and the depth can be gradually updated.

[0062] At 280 , the resulting 3D point cloud is output for display on a display.

[0063] Example

[0064] Unlike classical image-to-image translation, in the described setting, there are no pairs of scribble images corresponding to the associated color point clouds. It should be noted that there are multiple ways in which a garment can be sketched, depending on the granularity and complexity of the user input. Thus, downstream applications allow users to create 3D garments by specifying their features with broad and thin strokes, representing global and local properties of the garment. In this sense, scribbles are conceptually similar to superpixels, where pixels are grouped based on their perceptual features.

[0065] In the sample configuration, to generate the training dataset, the point cloud was first rendered into a 2D image using the Pytorch3D rasterizer, and the graffiti was extracted using the simple non-iterative clustering (SNIC) algorithm described by Achanta et al. in "Superpixels and polygons using simple non-iterative clustering" in the 2017 IEEE Conference on Computer Vision and Pattern Recognition, which is incorporated herein by reference. The SNIC algorithm is a non-iterative version of the simple linear iterative clustering (SLIC) algorithm that uses the SNIC segmentation as a starting point to perform polygon segmentation of the image. In the sample configuration, for the kth superpixel S k , the color is designated as C k =1 / |S k |(Σ ci∈Sk ci ), where c i YesS k The color of the i-th pixel is contained in |S k | indicates S kFor each image, K = {5, 10, 15, 25, 50} superpixels are extracted to simulate graffiti with different levels of detail and granularity. Figure 3 A sample of generated graffiti is shown, with different levels of detail ranging from coarse ( 310 ), medium ( 320 ) to fine ( 330 ), with different target numbers of segments for clustering. An input training dataset is generated from an input point cloud 300 .

[0066] To generate the front and back color density pairs B 210 as a training dataset, the point cloud is first spatially divided into subgroups (in the dimension without depth), the central depth is measured in the subgroup, and the point cloud is divided into the front part and the back part accordingly. To obtain the color image, an orthogonal camera is used to render the point cloud. To generate the density image, the front and back point clouds are projected onto a plane and their 2D histograms are calculated. The resulting image size is 512, and 128×128 bins are used in practice. Figure 4 The process of creating a planar point cloud from a graffiti image is shown. As shown, a single training example may include a graffiti image 210, a color density pair 210, and a target planar point cloud generated from an input point cloud 410 generated by an orthogonal camera 420: c ,B,P G}400.

[0067] In the sample configuration, the DeepFashion3D dataset is used as the dataset because it is the largest existing 3D clothing dataset. It provides point clouds of 2078 garments (e.g., coats, pants, dresses, skirts, etc.) in 10 different categories. For each garment, 5 scribbles of different granularity are generated, resulting in 10,390 inputs. The dataset is divided into 90% training samples and 10% testing samples.

[0068] To establish a baseline, three current point cloud generation methods and a grid-based method were modified for experimental comparison. To evaluate the design choices, three sets of comparisons were considered. First, single-stage method schemes were evaluated. To this end, current point cloud generation methods such as PointFlow, Shape GF (Shape Gradient Field), and Point Voxel Diffusion (PVD) were modified so that they take graffiti images as input and output color point clouds. The modified methods are denoted as s-PointFlow, s-Shape-GF, and s-PVD, respectively. For s-PointFlow, the official code of PointFlow was modified to create the original unconditional point conduction normal form (CNF) G on the graffiti image using an image encoder. θConditioning. For s-Shape-GF, the official code of Shape-GF is modified whereby the encoder and decoder in the variational autoencoder (VAE) are conditioned on the scribble image with the image encoder. For s-PVD, the official code of PVD is modified to append the features extracted from the scribble image to the features of each point of all point voxel CNNs used in the denoising model.

[0069] Second, different implementations of the two-stage approach are evaluated. In the first approach, conditional PVD is used to predict shapes (i.e., coordinates only) from the scribbles, and then another conditional PVD is used to colorize the point cloud again based on the scribbles. This approach is referred to as 2-stage-PVD. In the second approach, a 2D generator G is used s Generate dimensional coupling pairs, and then use PVD to generate G-based s The final colored point cloud is output. This method is denoted as G s +img-PVD. Finally, the results are compared with a recent mesh-based method (sketch2model).

[0070] The method was further modified to additionally predict the color of the vertices, denoted as c-sketch2model. For quantitative comparison, the points on the generated mesh were uniformly sampled to form a point cloud. In c-sketch2model, the official code of sketch2model was modified by providing the model with grayscale sketches (for contour matching) and colored scribbles as input. Then, in addition to vertex and surface predictions, the model was also asked to predict the color of each vertex. An additional L1 loss was used in color prediction.

[0071] There are many methods to evaluate the quality of the predicted geometry. Similarly, various existing perceptual metrics can be used to evaluate texture or color quality. However, for the current task, color and geometry need to be evaluated jointly. To facilitate future work on conditional 3D clothing synthesis, the method can be evaluated under many metrics, evaluating both geometry and color, as well as the consistency of the graffiti. In such an approach, the color point cloud P is represented as (P p , P c ), where P p ∈R N×3 represents the position, and P c ∈R N×3 The common earth moving distance (EMD) and chamfer distance (CD) are used to calculate the distance between points in two point clouds P1 and P2 as follows:

[0072] CD=minΣ p1∈P1 ||p1 p-φ1(p1) p ||2 2 +minΣ p2∈P2 ||p2 p –φ2(p2) p ||2 2 ,φ1(p1)∈P2,φ2(p2)∈P1

[0073] EMD = min Σ p1∈P1 || p1 p - φ(p1) p ||2 2 , φ : P1 → P2 is a bijection. (2)

[0074] The distribution function φ is obtained from the distance measurement. φ is used to measure the color distance, which is expressed as CD-c and EMD-c as follows:

[0075] CD-c=Σ p1∈P1 ||p1 c -φ1(p1) c ||2 2 +Σ p2∈P2 ||p2 p –φ2(p2) c ||2 2 ,EMD-c=Σ p1∈P1 ||p1 c -φ(p1) p ||2 2 (3)

[0076] The learned perceptual patch similarity (LPIPS) metric can also be used to measure the perceptual distance between the input graffiti image and the projected image of the generated color point cloud. The Pytorch3D rasterizer can be used to project the image from the point cloud. All reported numbers can be averaged from 5 trials. In order to measure the correspondence between the generated 3D clothing and the input graffiti, a graffiti-clothing consistency score (ScrRecon) can be introduced. The measurement can be performed by performing the above process to obtain the graffiti from the generated 3D clothing and compare them using the above perceptual distance. Therefore, the measurement can be formulated as ScrRecon = LPIPS (I c ,Scribble(Rasterize(P))).

[0077] To generate color point clouds from graffiti, the method described with reference to FIG2 is compared with a modified one-stage method including PointFlow, Shape-GF, and PVD. Figure 5An input doodle 100 is shown converted into a 3D garment 500 using a point cloud-based approach (left), where the synthesized garment contains 16,384 points. Figure 5 The qualitative samples in show that the one-stage methods (s-PointFlow, s-Shape-GF, and s-PVD) cannot jointly model shape and color well. s-PointFlow and s-Shape-GF predict noisy point clouds without clear clothing outlines. In addition, these methods can only roughly reproduce the average color of the graffiti, not to mention the subtle details such as Figure 6 The numerical scores in , where the method described in this paper is called "neural customization". EMD, CD and their color counterparts (EMD-c and CD-c) are much worse than the published methods. Although s-PVD produces sharp shapes, they are usually inconsistent with the input and the color distribution is not consistent. This is in Figure 6 This shows that it is still difficult for current 3D architectures to handle cross-dimensional and cross-domain transformations simultaneously. Specifically, Figure 6 Point clouds containing 8K (top) and 16K (bottom) points are used to provide quantitative comparisons of geometry (EMD, CD), geometry and color (EMD-c, CD-c), and consistency with input scribbles (LPIPS, ScrRecon). The mean and standard deviation of 5 trials are shown.

[0078] According to qualitative and quantitative evaluation, the two-stage method (2stage-PVD and G s +img-PVD) shows more promising results in shape and color modeling. The 2stage-PVD method improves generation quality due to the decoupled shape and color stages. However, since the model still needs to deal with difficult cross-domain transformations in 3D, it often fails to translate correctly ( Figure 5 On the other hand, although still inferior in terms of scores, G s The shape quality of +img-PVD is closest to the published methods. However, it can only vaguely capture the color distribution in the graffiti.

[0079] Figure 5 The results suggest that, at least given the capabilities of current 3D generators, it is desirable to rely on 2D generators to handle cross-dimensional and cross-domain translation tasks.

[0080] A comparison is also made with mesh-based methods to demonstrate the advantages of adopting point clouds as 3D representations. Existing sketch-to-mesh methods generate meshes using sketches, showing results on samples with simple topology and limited variations (e.g. ShapeNet). The results of the described methods are compared with a modified version of the shape2model method that also predicts vertex colors. Figure 6 , Fig. 7A and Figure 7B As shown, the performance of current mesh-based methods is substantially lower than any point cloud-based methods. Fig. 7A and 7B Two examples are shown. In each example, the input graffiti (left) 100 is converted into a volumetric 3D garment (center) using the disclosed method, and then output by a mesh-based method (right). It should be understood that without the correct 3D structure, the model can only predict a uniform average color.

[0081] The goal of density-aware sampling 260 is to align the distribution of the planar point cloud 230 sampled from the true point cloud and the planar point cloud 230 sampled from the auxiliary image set. Figure 8 Quantitative and qualitative results are depicted, showing the impact of density-aware sampling 260. Figure 8 The upper left corner of shows the input graffiti 100 and its corresponding point cloud (real data) 800, followed by the point clouds obtained using uniform sampling 810 and the density-aware sampling method 820 described herein. Figure 8 The upper right corner of shows the distribution of the plane point cloud 830 obtained from density-aware sampling 260, which is closer to the true distribution. Figure 8 The bottom part of shows the results without density-aware sampling, when uniform sampling is used, and the results of the full model. Figure 8 It is shown that the density distribution of the planar point cloud 230 sampled with density-aware sampling 260 is very similar to the real data. It should be understood that with only uniform sampling, the difference between the inference and training distributions may lead to a slight performance degradation.

[0082] In experiments, the baseline method fails to synthesize high-density results ( Figure 5 16K point cloud is shown). The disclosed method can generate a high multi-density garment containing 32,768 points, such as Fig. 9 As shown, Fig. 9 Shown by G s The resulting multi-view renderings 900 of various 3D garments contain 32,768 points, their input scribbles 910, and color density pairs 920. At this resolution, texture and color patterns, folds and wrinkles are conveyed in greater detail (e.g., Fig. 9 and Figure 5 for comparison).

[0083] Those skilled in the art will appreciate that the methods described herein are readily applicable to a number of real-world applications. Fig.10 , the generated results are presented from some customized graffiti images 1010 and 1020. The generated color point cloud 1000 is of high quality, not only in shape, but also in color (left) and pattern (right) designed by the user, which are consistent with the user input. Fig.10 It also shows that the model described in this paper does more than just memorize and overfit the dataset.

[0084] In addition, the user can simply draw a graphic of any posture on the human body model 1100, and the corresponding 3D clothing 1110 can be automatically generated according to the graffiti 1120, such as Fig.11 It should be appreciated that such an application can potentially simplify and speed up the fashion design pipeline and unleash the user's innate creativity.

[0085] Fig.12 More synthetic samples 1200 are shown using different numbers of target segments during processing. Figure 2A The data preparation pipeline can simulate different levels of graffiti details from the input image 1210.

[0086] To demonstrate some real-world applications, designers were asked to provide clothing designs. Despite the large distribution gap between the doodles provided by the designers and those used in training, the model described in this paper still generates reasonable outputs 1300 from the input designs 1310, such as Fig.13 The model described herein is also shown to be useful in a clothing design process, where the designer can stop at any time and visualize the current design.

[0087] The model described in this paper is also applied to real commercial images. Although the distribution gap is even larger, the model described in this paper still synthesizes good results, such as Fig.14A , which shows the generation of a 3D garment 1400 from graffiti 1410 including a real commercial image 1420 . Fig. 14B It further presents interesting “failure scenarios” due to domain gaps. Fig. 14B , the model interprets the heart shape 1430 as a graffiti pattern and attempts to cover the garment with the pattern, as shown at 1440. In the sample configuration, the training data can be used to train the model to not place the pattern on the back of the garment because the image is only a front-facing image.

[0088] Those skilled in the art will appreciate that the limitations of the methods described herein lie primarily in the bounded generality of the 2D image generator. Fig.15As shown, since the generator is trained on less than 2000 images, which is a relatively small set of data considering the current standards for training generative models, some of the observed failure cases may occur. First, if the scribble 1500 includes patterns that were not learned in the training set, sometimes the model may generate unlikely after images 1510 and 1520 whose patterns are similar to the before images (top left and top right samples). Second, if the scribble 1500 falls into some distribution outside the training range or is too far from reality, the predicted density maps 1510, 1530 may be noisy or even contain artifacts (top left and bottom left). Finally, since the training set is small, sometimes if the input scribble 1500 happens to be similar to some training data, the model may over-interpret the input, as shown at 1540 (bottom right). However, these problems are not relevant to the designed pipeline and can be solved with more training samples.

[0089] Thus, the techniques described herein bridge the gap between the use of 2D representations and the use of 3D representations for novice clothing designers. In order to enable the training of such models with appropriate paired data, suitable methods are provided for extracting abstract but plausible "doodles" from point clouds. Through the proposed intermediate medium and non-parametric 2D to 3D lifting techniques, the disclosed methods allow highly plausible clothing digitization results to be obtained from very simple user inputs, despite their complex and variable topology, non-rigid structure, and lack of accurate ground truth data corresponding to the inputs. The point-based representation avoids the problems of large computational cost and memory usage that arise at high resolutions with volumetric representations, while avoiding the need to address the need to accommodate topological changes that arise in mesh-based methods. The described methods also enable realistic 3D rendering of the generated garments from minimal user input and are easily used in some real-world applications where 3D garment images are displayed on a digital representation of a user or avatar as a flexible virtual try-on of the 3D garment images generated from the user's sketches.

[0090] Fig.161 is a diagram of a machine 1600 in which instructions 1610 (e.g., software, programs, applications, applet programs, application programs, or other executable code) may be executed to cause the machine 1600 to perform any one or more of the methods discussed herein. For example, the instructions 1610 may cause the machine 1600 to perform any one or more of the methods described herein. The instructions 1610 convert a general, unprogrammed machine 1600 into a specific machine 1600 that is programmed to perform the functions described and illustrated in the manner described. The machine 1600 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1600 may operate as a server machine or a client machine in a server-client network environment or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1600 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a network device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1610, sequentially or otherwise, that specify actions to be taken by the machine 1600. In addition, while only a single machine 1600 is shown, the term "machine" should also be construed to include a collection of machines that individually or jointly execute instructions 1610 to perform any one or more of the methodologies discussed herein. For example, the machine 1600 may include Figure 1 In some examples, the machine 1600 may also include a client and server system, where some operations of a particular method or algorithm are performed on the server side and some operations of a particular method or algorithm are performed on the client side.

[0091] Machine 1600 may include processor 1604, memory 1606, and input / output I / O components 1602, which may be configured to communicate with each other via bus 1640. In an example, processor 1604 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 1608 that executes instructions 1610 and processor 1612. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Fig.16Multiple processors 1604 are shown, but the machine 1600 may include a single processor having a single core, a single processor having multiple cores (eg, a multi-core processor), multiple processors having a single core, multiple processors having multiple cores, or any combination thereof.

[0092] The memory 1606 includes a main memory 1614, a static memory 1616, and a storage unit 1618, all of which are accessible by the processor 1604 via a bus 1640. The main memory 1606, the static memory 1616, and the storage unit 1618 store instructions 1610 for any one or more of the methodologies or functions described herein. During execution of the instructions by the machine 1600, the instructions 1610 may also reside, completely or partially, within the main memory 1614, within the static memory 1616, within the machine-readable medium 1620 within the storage unit 1618, within at least one of the processors 1604 (e.g., within a cache memory of the processor), or within any suitable combination thereof.

[0093] I / O components 1602 may include a variety of components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 1602 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that I / O components 1602 may include Fig.16 1602. In various examples, the I / O components 1602 may include user output components 1626 and user input components 1628. The user output components 1626 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistive mechanisms), other signal generators, etc. The user input components 1628 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other directional instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide location and force of touch or touch gestures), audio input components (e.g., microphones), and the like.

[0094] In further examples, the I / O component 1602 may include a biometric component 1630, a motion component 1632, an environment component 1634, or a position component 1636, as well as various other components. For example, the biometric component 1630 includes components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying individuals (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), and the like. The motion component 1632 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, and a rotation sensor component (e.g., a gyroscope).

[0095] Environmental components 1634 include, for example, one or more cameras (with still image / photo and video capabilities), lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors for detecting hazardous gas concentrations to ensure safety or to measure pollutants in the atmosphere), or other components that can provide identification, measurements, or signals corresponding to the surrounding physical environment.

[0096] Position component 1636 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), and the like.

[0097] The communication can be implemented using a variety of technologies. The I / O components 1602 also include a communication component 1638, which is operable to couple the machine 1600 to the network 1622 or the device 1624 via a corresponding coupling or connection. For example, the communication component 1638 may include a network interface component or another suitable device that interfaces with the network 1622. In further examples, the communication component 1638 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Parts (e.g. Low power consumption), Components and other communication components to provide communication via other modes. Device 1624 can be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0098] In addition, the communication component 1638 can detect the identifier or include components that can be used to detect the identifier. For example, the communication component 1638 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting a one-dimensional barcode (such as a universal product code (UPC) barcode), a multi-dimensional barcode (such as a Quick Response (QR) code, Aztec code, Data Matrix, Data Glyph, MaxiCode, PDF417, Hypercode, UCC RSS-2D barcode, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying a tagged audio signal). In addition, various information can be derived via the communication component 1638, such as a location via Internet Protocol (IP) geolocation, a location via Internet Protocol (IP), ... The location of signal triangulation, the location of an NFC beacon signal via detection that can indicate a specific location, etc.

[0099] Various memories (e.g., main memory 1614, static memory 1616, and memory of processor 1604) and storage unit 1618 may store one or more sets of instructions and data structures (e.g., software) embodying or used by any one or more of the methods or functions described herein. When executed by processor 1604, these instructions (e.g., instructions 1610) cause various operations to implement the disclosed examples.

[0100] The instructions 1610 may be sent or received over the network 1622 using a transmission medium via a network interface device (e.g., a network interface component included in the communication component 1638) and using any of several well-known transmission protocols (e.g., the Hypertext Transfer Protocol (HTTP)). Similarly, the instructions 1610 may be sent or received via a coupling (e.g., a peer-to-peer coupling) to the device 1624 using a transmission medium.

[0101] Fig.17 1700 is a block diagram showing a software architecture 1704 that can be installed on any one or more of the devices described herein. The software architecture 1704 is composed of hardware (such as a machine 1702 (see Fig.16 )) support, the machine 1702 includes a processor 1720, a memory 1726, and an I / O component 1738. In this example, the software architecture 1704 can be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 1704 includes layers such as an operating system 1712, a library 1710, a framework 1708, and an application 1706. In operation, the application 1706 calls an API call 1750 through the software stack and receives a message 1752 in response to the API call 1750.

[0102] The operating system 1712 manages hardware resources and provides public services. The operating system 1712 includes, for example, a kernel 1714, services 1716, and drivers 1722. The kernel 1714 acts as an abstraction layer between the hardware and other software layers. For example, the kernel 1714 provides memory management, processor management (e.g., scheduling), component management, network and security settings, and other functions. Services 1716 can provide other public services to other software layers. Drivers 1722 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1722 may include display drivers, camera drivers, or Low power drivers, Flash memory drivers, serial communication drivers (such as USB drivers), Drivers, audio drivers, power management drivers, etc.

[0103] The library 1710 provides a common low-level infrastructure used by the application 1706. The library 1710 may include a system library 1718 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 1710 may include an API library 1724, such as a media library (e.g., a library for supporting the presentation and operation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for rendering in two dimensions (2D) and three dimensions (3D) in graphical content on a display), a database library (e.g., SQLite for providing various relational database functions), a network library (e.g., WebKit providing network browsing functions), etc. The library 1710 may also include a variety of other libraries 1716 to provide many other APIs to the application 1706.

[0104] The framework 1708 provides a common high-level infrastructure used by the applications 1706. For example, the framework 1708 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. The framework 1708 may provide a wide range of other APIs that can be used by the applications 1706, some of which may be limited to specific operating systems or platforms.

[0105] In an example, applications 1706 may include a home application 1736, a contacts application 1730, a browser application 1732, a book reader application 1734, a location application 1742, a media application 1744, a messaging application 1746, a game application 1748, and various other applications (such as third-party applications 1740). Applications 1706 are programs that execute functions defined in the program. Various programming languages ​​can be used to generate one or more applications 1706, which are structured in various ways, such as object-oriented programming languages ​​(such as Objective-C, Java, or C++) or procedural programming languages ​​(such as C or assembly language). In a specific example, third-party applications 1740 (e.g., those used by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software development kit (SDK) applications can be developed on mobile operating systems such as IOS TM ANDROID TM , Phone or other mobile operating system). In this example, third-party applications 1740 can call API calls 1750 provided by the operating system 1712 to facilitate the functions described herein.

[0106] "Carrier signal" means any intangible medium capable of storing, encoding or carrying instructions for execution by a machine, and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device.

[0107] "Client Device" means any machine that connects to a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, desktop computer, laptop computer, portable digital assistant (PDA), smartphone, tablet computer, ultrabook, netbook, notebook computer, multiprocessor system, microprocessor-based or programmable consumer electronic device, game console, set-top box, or any other communications device that a user can use to access a network.

[0108] "Communications Network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Plain Old Telephone Service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless or cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interconnection for microwave access (WiMAX), long term evolution (LTE) standard, other standards defined by various standards development organizations, other remote protocols, or other data transmission technologies.

[0109] "Component" refers to a device, physical entity, or logic with boundaries defined by function or subroutine calls, branch points, APIs, or other technologies that provide partitioning or modularization of specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processes. Components can be encapsulated functional hardware units designed for use with other components, and are also part of a program that generally performs specific functions in related functions. Components can constitute software components (e.g., code contained on a machine-readable medium) or hardware components. "Hardware components" are tangible units that can perform operations and can be configured or arranged in a certain physical manner. In various examples, one or more computer systems (e.g., stand-alone computer systems, client computer systems, or server computer systems) or one or more hardware components of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or an application portion) as hardware components that perform certain operations described herein. Hardware components can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include a dedicated circuit or logic that is permanently configured to perform certain operations. A hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). The hardware components may also include programmable logic or circuits that are temporarily configured by software to perform certain operations. For example, the hardware components may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware components become a specific machine (or a specific component of a machine) that is specifically customized to perform the configuration function, and are no longer a general-purpose processor. It should be understood that the decision to mechanically implement the hardware components in a dedicated and permanently configured circuit or in a temporarily configured circuit (e.g., configured by software) may be driven by cost and time factors. Therefore, the phrase "hardware components" (or "hardware-implemented components") should be understood to include tangible entities, i.e., entities that are physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate or perform certain operations described herein in a certain manner. Considering the example that the hardware components are temporarily configured (e.g., programmed), each of the hardware components does not need to be configured or instantiated at any one point in time. For example, in the case where the hardware components include a general-purpose processor that is configured as a special-purpose processor by software, the general-purpose processor can be configured as a correspondingly different special-purpose processor (e.g., including different hardware components) at different times. The software configures one or more specific processors accordingly, for example, to constitute a specific hardware component at one point in time and to constitute different hardware components at different points in time. Hardware components can provide information to other hardware components and can also receive information from other hardware components. Therefore, the described hardware components can be considered to be communicatively coupled. In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission between or among two or more hardware components (e.g., through appropriate circuits and buses).In an example where multiple hardware components are configured or instantiated at different times, for example, communication between such hardware components can be achieved by storing and retrieving information in a memory structure that multiple hardware components can access. For example, a hardware component can perform an operation and store the output of the operation in a memory device coupled to it in communication. Then, later, another hardware component can access the memory device to retrieve and process the stored output. The hardware component can also initiate communication with an input or output device and can operate on a resource (e.g., a collection of information). The various operations of the example methods described herein can be performed at least in part by one or more processors, which are temporarily (e.g., by software) configured or permanently configured to perform related operations. Whether temporarily or permanently configured, such a processor can constitute a processor-implemented component for performing one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the method described herein can be implemented at least in part by a processor, wherein a specific one or more processors are examples of hardware. For example, at least some operations of the method can be performed by one or more processors or processor-implemented components. In addition, one or more processors may also be used in a "cloud computing" environment or as a "software as a service" (SaaS) as a performance to support related operations. For example, at least some operations may be performed by a group of computers (as an example of a machine including a processor), which may be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The performance of certain operations may be distributed among processors, not only residing in a single machine, but also deployed on multiple machines. In some examples, the processor or a component implemented by the processor may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other examples, the processor or a component implemented by the processor may be distributed in multiple geographic locations.

[0110] "Computer-readable storage media" refers to both machine storage media and transmission media. Therefore, these terms include storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" refer to the same thing and can be used interchangeably in this disclosure.

[0111] “Machine storage media” refers to a single or multiple storage devices and media (e.g., centralized or distributed databases and associated caches and servers) that store executable instructions, routines, and data. Thus, the term shall include, but is not limited to, solid-state memory, optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include nonvolatile memory, including, by way of example, semiconductor memory devices (e.g., Electrically Erasable Programmable Read-Only Memory (EPROM)), Electrically Erasable Programmable Read-Only Memory (EEPROM), FPGAs, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage media,” “computer storage media,” and “device storage media” expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term “signal media.”

[0112] “Non-transitory computer-readable storage medium” refers to tangible media that is capable of storing, encoding, or carrying instructions for execution by a machine.

[0113] "Signal medium" refers to any intangible medium that is capable of storing, encoding, or carrying instructions to be executed by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" shall include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" have the same meaning and may be used interchangeably in this disclosure.

Claims

1. A method for generating a three-dimensional (3D) garment image from a two-dimensional (2D) garment image drawn by a user, the method comprising: receiving the 2D clothing image drawn by the user; The 2D clothing image is converted into a color density pair by a conditional 2D generator; performing density-aware sampling on the color-density pairs to generate a planar point cloud; Converting the planar point cloud into an output 3D point cloud by a conditional 3D generator; and The output 3D point cloud is sent to be displayed as the 3D garment image.

2. The method according to claim 1, wherein: The received 2D garment image includes a 2D hand-drawn or computer-generated abstraction of the garment that defines the basic color pattern and shape of the target garment.

3. The method according to claim 2, wherein: Received 2D clothing image I c by I c ∈R H×W×3 Represented by, where H, W are its height and width, and for the target garment P G The output 3D point cloud contains information about the position and color of the target garment and is denoted as P G ∈R N×(3+3) , where each point p i ∈P G , i=1...N, where each 3D point cloud contains 3 channels.

4. The method according to claim 1, wherein: The received 2D clothing image is drawn by the user on a human body model, and the 3D clothing image is displayed on the human body model. 5 . The method of claim 1 , wherein converting the 2D garment image into color-density pairs comprises generating RGB projections and density maps from front and back views of the received 2D garment image.

6. The method according to claim 5, wherein: Performing the density-aware sampling includes converting the color-density pairs into the planar point cloud by sampling point positions and colors from corresponding pairs of the color-density pairs.

7. The method according to claim 6, wherein: Performing the density-aware sampling includes ignoring depth during generation of the planar point cloud.

8. The method of claim 1, wherein performing density-aware sampling comprises using a non-parametric algorithm, wherein the non-parametric algorithm obtains the corresponding center coordinate x from the pixel coordinate (i, j) for each pixel position (i, j). ij ,y ij , and the estimated number of points to be added for the front (f) 2D garment image and the rear (r) 2D garment image n r ij ,Will The points are appended to the target plane point cloud to approximate the plane point cloud P for a given color density pair B at each pixel location (i, j) flat , where the first two coordinates are the center coordinates affected by the noise, the depth dimension is 0, and the color is and v r ij ,thus Point using coordinates (N(x ij ,σ 2 ),N(y ij ,σ 2 ),0) is inserted and the color is And n r ij Point using coordinates (N(x ij ,σ 2 ),N(y ij ,σ 2 ),0) is inserted and the color is v r ij , where σ 2 Includes standard deviation.

9. The method according to claim 1, wherein: The conditional 3D generator includes a diffusion-based generator, and converting the planar point cloud into the output 3D point cloud includes adding noise to the depth dimension in a forward diffusion process, and initializing the depth with a Gaussian distribution and gradually updating the depth in a reverse denoising process, wherein the forward diffusion process and the reverse denoising process are modeled as Markov chains, and each step in each process is modeled as a Gaussian transformation.

10. The method according to claim 1, wherein: Displaying the output 3D point cloud as the 3D garment image includes displaying the 3D garment image on a digital representation of a user or an avatar as a virtual try-on of the 3D garment image.

11. A system for generating a three-dimensional (3D) garment image from a two-dimensional (2D) garment image drawn by a user, comprising: a conditional 2D generator that receives the 2D clothing image drawn by the user and converts the 2D clothing image into a color-density pair; a density-aware sampling algorithm that performs density-aware sampling on the color-density pairs to generate a planar point cloud; and A conditional 3D generator converts the planar point cloud into an output 3D point cloud to be displayed as the 3D garment image.

12. The system of claim 11, wherein the 2D garment image comprises a 2D hand-drawn or computer-generated abstraction of the garment defining a basic color pattern and shape of the target garment.

13. The system according to claim 12, wherein: Received 2D clothing image I c byI c ∈R H×W×3 Represented by, where H, W are its height and width, and for the target garment P G The output 3D point cloud contains information about the position and color of the target garment and is denoted as P G ∈R N×(3+3) , where each point p i ∈P G , i=1...N, where each 3D point cloud contains 3 channels.

14. The system according to claim 11, wherein: The conditional 2D generator generates the color-density pairs as RGB projections and density maps generated from front and back views of the received 2D garment image.

15. The system of claim 14, wherein: The density-aware sampling algorithm converts the color-density pairs into the planar point cloud by sampling point locations and colors from corresponding pairs of the color-density pairs.

16. The system of claim 15, wherein the density-aware sampling algorithm ignores depth during generation of the planar point cloud.

17. The system of claim 11, wherein: The density-aware sampling algorithm includes a non-parametric algorithm that obtains the corresponding center coordinate x from the pixel coordinate (i, j) for each pixel position (i, j). ij ,y ij , and the estimated number of points to be added for the front (f) 2D garment image and the rear (r) 2D garment image n r ij ,Will The points are appended to the target plane point cloud to approximate the plane point cloud P for a given color density pair B at each pixel location (i, j) flat , where the first two coordinates are the center coordinates affected by the noise, the depth dimension is 0, and the color is and v r ij ,thus Point using coordinates (N(x ij ,σ 2 ),N(y ij ,σ 2 ),0) is inserted and the color is And n r ij Point using coordinates (N(x ij ,σ 2 ),N(y ij ,σ 2 ),0) is inserted and the color is v r ij , where σ 2 Includes standard deviation.

18. The system of claim 11, wherein the conditional 3D generator comprises a diffusion-based generator that converts the planar point cloud into the output 3D point cloud by adding noise to the depth dimension in a forward diffusion process and initializing the depth with a Gaussian distribution and gradually updating the depth in a backward denoising process, wherein The forward diffusion process and the reverse denoising process are modeled as Markov chains, and each step in each process is modeled as a Gaussian transition.

19. The system of claim 11, further comprising a display that displays the output 3D point cloud as the 3D garment image on a digital representation of a user or an avatar for virtual try-on of the 3D garment image.

20. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to generate a three-dimensional (3D) garment image from a two-dimensional (2D) garment image drawn by a user by performing operations comprising: receiving the 2D clothing image drawn by the user; The 2D clothing image is converted into a color density pair by a conditional 2D generator; the color density pair is density-aware sampled to generate a planar point cloud; Converting the planar point cloud into an output 3D point cloud by a conditional 3D generator; and The output 3D point cloud is sent to be displayed as the 3D garment image.

Images