Virtual garment fitting using machine learning

Abstract

Methods and apparati for fitting a two dimensional digitized garment (2) onto a two dimensional digitized model (1). A method embodiment of the present invention comprises the steps of digitally removing (11) a portion of original clothing from a model (1) corresponding to an area where a garment (2) is to be fitted on the model (1); warping (12) a digitized representation of the garment (2) to correspond to the size and shape of the model (1); combining (13) the warped representation (5) of the garment (2) with the representation of the partially de-clothed model 6; and optionally increasing (14) the apparent resolution and image quality of the resulting digital image (4).

Description

VIRTUAL GARMENT FITTING USING MACHINE LEARNINGInventor: Minh VoRelated Application

[0001] This patent application claims the priority benefit of commonly owned U.S. provisional patent application serial number 63 / 540,886 filed September 27, 2023 entitled “Virtual Garment Fitting Using Machine Learning”, which provisional patent application is hereby incorporated by reference in its entirety into the present patent application.Technical Field

[0002] The present invention pertains to the field of fitting a two dimensional digitized garment onto a two dimensional digitized model.Backaroud Art

[0003] Various prior art techniques have been attempted, with varying degrees of success and lack of success, to fit garments onto models virtually, i.e., using computers to simulate the garments and the models.Disclosure of Invention

[0004] Given a source 2D (two dimensional) digital representation of a model 1 and a source 2D digital representation of at least one side of a garment 2, it is an object of the present invention to warp the garment 2 and fit the warped digitized garment 5 onto the model 1 in such a way that the resulting garmented model 4 is photorealistic. The model 1 is usually a human model, but model 1 can also be a non-human animalor an inanimate object. The side of the source garment 2 is normally the front of the garment 2, but in some embodiments, both the front and the back of the garment 2 are taken into account.

[0005] As illustrated in Figure 1, a method embodiment of the present invention comprises the steps of digitally removing 11 a portion of the original clothing from the model 1 corresponding to an area where the garment 2 is to be fitted; warping 12 the digitized representation of the garment 2 to correspond to the shape of the model 1 ; combining 13 the warped representation 5 of the garment 2 with the digitized representation of the partially de-clothed model 6; and optionally increasing 14 the apparent resolution and image quality of the resulting digital image 4. Step 11 can be performed before step 12, or step 12 can be performed before step 11 , or steps 11 and 12 can be performed simultaneously.Brief Description of the Drawings

[0006] These and other more detailed and specific objects and features of the present invention are more fully disclosed in the following specification, reference being had to the accompanying drawings, in which:

[0007] FIG. 1 is a flow diagram of a method embodiment of the present invention.

[0008] FIG. 2 is an illustration of the use of control points 3 to enhance the performance of tryon neural network 7 of the present invention in a training phase.

[0009] FIG. 3 is an illustration of the result of using tryon neural network 7 in a runtime (tryon) phase of the present invention.

[0010] FIG. 4 is an illustration of an alternative embodiment of the present invention in which the resolution of the resulting garmented model image 4 is improved, and artifacts are removed.

[0011] FIG. 5 is a series of illustrations 5A through 5E illustrating a styled declothing embodiment of the present invention.

[0012] FIG. 6 illustrates an embodiment of the present invention in which estimated digitized garments 62 are used in a technique of semi-supervised learning (training) from paired and unpaired data.Detailed Description of Preferred Embodiments

[0013] The following detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show illustrations in accordance with example embodiments. These example embodiments are described in enough detail to enable those skilled in the art to practice the present subject matter. The embodiments can be combined, other embodiments can be utilized, or structural, logical, and implementation changes can be made without departing from the scope of the present invention. The following detailed description is, therefore, not to be taken in a limiting sense.

[0014] In this document, the terms "a" or "an" are used, as is common in patent documents, to include one or more than one. The term "or" is used to refer to a nonexclusive "or," such that "A or B" includes "A but not B," "B but not A," and "A and B," unless otherwise indicated. Furthermore, all publications, patents, and patent documents referred to in this document are incorporated by reference herein in their entirety, as though individually incorporated by reference. In the event of inconsistent usages between this document and those documents so incorporated by reference, the usage in the incorporated reference(s) should be considered supplementary to that of this document; for irreconcilable inconsistencies, the usage in this document controls.

[0015] Figure 2 illustrates items 1 , 2 that are used as inputs in a training phase for a tryon neural network 7 that is at the heart of the present invention. Neural network 7 is adapted to employ techniques of machine learning (ML). Such networks 7 are first trained in a training phase (Figure 2) before they are used for “real” (run-time; tryon) data (Figure 3). In the training phase of the present invention, many input pairs of digitized garmented models 1 and garments 2 are fed to network 7, to teach network 7 how garments 2 should be warped when real (run-time; tryon) data is subsequentlyfed to network 7. The input pairs in the training phase of network 7 comprise many (typically thousands) of digitized flat (2D) sample garments 2, and four to five times as many digitized sample garmented models 1. The models 1 are garmented with the garments 2 from the training set. For purposes of the present invention, it doesn’t matter how the input garments 2 and the input models 1 have been digitized; any conventional digitization technique(s) can be used.

[0016] The set of input sample (training) garments 2 typically comprises different garments 2 of different sizes. The set of input sample (training) garmented models 1 typically comprises a combination of the same model 1 in different poses, and different models 2 having different sizes and / or shapes. The input sample models 1 and input sample garments 2 are fed to network 7 in pairs. When these inputs 1, 2 are fed to network 7, a plurality of control points 3(1) are digitally superimposed on the input digitized models 1 , and a corresponding plurality of control points 3(2) are digitally superimposed on the input digitized garments 2. There is a one-to-one correspondence between each model control point 3(1 ) and its corresponding garment control point 3(2). This correspondence helps to teach network 7 how a garment 2 should be warped to fit a corresponding model 1 .

[0017] The control points 3 are typically affixed onto the models 1 and garments 2 by a control point neural network (not illustrated) that is other than neural network 7. Control points 3 are provided to model 1 and garment 2 as a 2-dimensional location (e.g., in x-y coordinates) and a N-dimensional feature descriptor (vector). The user of the system runs a similarity matching algorithm on the N-dimensional feature descriptors to match between control points 3 in model 1 and garment 2.

[0018] By training network 7 in this manner, network 7 learns how to warp garments 2 based upon just 2D (two dimensional) input data 1 , 2, without having to need 3D (three dimensional) input data, and thus sidesteps the need for 3D processing entirely. This gives the present invention a great advantage over prior art techniques that use 3D processing in garment fitting, as it reduces the complexity of the processing required, while attaining superior results in terms of resolution and other quality metrics, as seen in the resulting output images 4.

[0019] After neural network 7 is trained in the training phase, an optional testing phase can be employed to improve the performance characteristics of network 7. Inthis testing phase, roughly 10% of the input sample models 1 and input sample garments 2 are held back from (i.e., not used in) the training phase. Rather than being fed as inputs to neural network 7 in the training phase, the held-back input pairs 1, 2 are fed as inputs to network 7 in the testing phase. Breaking up the set of input sample pairs 1 , 2 in this manner improves the accuracy of network 7 for the times when network 7 will later be used in the run-time (tryon) phase, compared with a scenario in which all of the inputs 1 , 2 are fed to network 7 in a single training phase. In the testing phase, it is important that the input models 1 and input garments 2 are different than the models 1 and garments 2 that were used as inputs to network 7 in the training phase.

[0020] Figure 3 corresponds to step 13 of Figure 1 , and illustrates the run-time (tryon) phase of neural network 7, in which real data are fed as inputs to network 7. A digitized model 1 , illustrated in Figure 3 as a photograph of a male human model, and a digitized version of a garment 2 that is desired to be fitted onto model 1 are inputs to neural network 7. In some embodiments, a DensePose image 9 is a third input to network 7. As is known to practitioners in this field of technology, a DensePose image 9 presents the pose of an image (in this case, the input model image 1 ) as a set of segments, and can improve the performance of network 7. In the run-time phase, network 7 has knowledge of the mappings between the control points 3(1 ) on the model 1 and the control points 3(2) on the garment 2 learned during the training phase (and in any optional testing phase) of network 7. This knowledge of the control point 3 correspondences facilitates and improves network 7’s determination of the desired warping that will convert input digitized garment 2 into a warped digitized garment 5 prior to fitting the warped digitized garment 5 onto model 1 to produce the resulting digitized garmented model 4. The fitting of warped digitized garment 5 onto digitized model 1 can be performed by neural network 7, a different neural network, or a conventional (Turing) computer.

[0021] Figure 4 corresponds to step 14 of Figure 1 , and illustrates a technique for improving the resolution and quality of the output image 4 that is produced by neural network 7. As shown on the left hand side of Figure 4, user uploaded model image 1 is partially de-clothed by a dedicated de-clothing neural network 53 to produce a partially de-clothed model image 6 (or 52 if the procedure illustrated in Figure 5 is employed). The portion of model 1 that is de-clothed corresponds to the area wheregarment 2 will be fitted onto model 1. Partially de-clothed model mage 6, 52 and garment image 2 are fed as inputs to neural network 7 (also identified as Neural net 1 in Fig. 4). The output of network 7 is model image 4 garmented with garment 2. At this juncture, image 4 may have undesirably low resolution, and / or may contain unwanted artifacts. The resolution of image 4 can be improved and artifacts removed by employing the technique shown on the right hand side of Figure 4. In this technique, a set of garment exemplars 41 are fed as inputs to neural network 42 (also identified as Neural net 2 in Fig. 4), along with image 4 being fed as an input to network 42, to produce an enhanced output image 43 of the garmented model 1 . Enhanced model image 43 has a higher resolution and fewer artifacts than model image 4.

[0022] Network 42 can be the same network as network 7, or network 42 can be a different neural network. When the networks 42, 7 are different, the image resolution improvement techniques described above in conjunction with network 42 can also be used by neural network 7. These same techniques can also be advantageously employed when de-clothing neural network 53 is not used.

[0023] The set of exemplars 41 typically comprises a more detailed view of garment 2 than the garment image 2 that was fed as an input to network 7, and / or images of different models 1 wearing garment 2, and / or images of a single model 1 wearing garment 2 in different poses. Good results 43 can often be obtained by using just three to five exemplars 41 .

[0024] Figure 5 corresponds to step 11 in Figure 1 , and consists of a set of images illustrating a styled declothing embodiment of the present invention. Fig. 5A shows the digitized garment image 2 that has been selected by the user of the present invention to be fitted onto model 1 . Fig. 5B and Fig. 5C show digitized image 4 (if the Figure 4 embodiment has not been used) or image 43 (if the Figure 4 embodiment has been used) of the model 1 wearing garment 2, in two different poses, which are referred to in Fig. 5 as Model image 1 and Model image 2. Model image 1 is the styled image, i.e., the image that exhibits the desired style. Model image 2 is the image to be de-clothed in the style of Model image 1. Fig. 5D shows a naive declothed model image 51 such as might be produced by the prior art. Note that in Fig. 5D, not just the desired jacket 2 has been removed from the model 4, 43, but the shirt underlying jacket 2 has been undesirably removed from model 4, 43.

[0025] Fig. 5E shows the styled declothed model image 52 that is produced by the illustrated embodiment of the present invention. Image 52 is produced by dedicated de-clothing neural network 53 (as shown in Figures 4 and 6). The inputs to network 53 are the (in this case, two) model images 4, 43. In some cases, better results can be obtained by having more than two model images 4, 43 fed as inputs to de-clothing neural network 53.

[0026] The embodiment illustrated in Figure 5 is particularly useful for situations in which garment 2 is worn as a layer of clothing on the model image 4, 43, and / or in situations where the garment 2 does not completely cover a portion of the model image 4, 43. The garment 2 that is illustrated in Figure 5 is a jacket covering the top portion of the body of a human model 1 . The naive declothing (Fig. 5D) simply erases every pixel at the top of the model image 4, 43. This is problematic for layered images, especially during the training phase of neural network 53, because the output of network 53 is supervised in its training phase with the ground truth layered image, and network 53 is thus forced to imagine the inner layer of the outfit and synthesize how the outer layer looks on top of the inner layer. The present invention solves this knotty technical problem by giving network 53 information as to how the garment 2 is actually worn on the model image 4, 43.

[0027] The embodiment illustrated in Figure 5 can be used in situations in which the models 4, 43 are wearing more than two garments 2. In this scenario, de-clothing neural network 53 is trained to remove, and does remove, at least one garment 2 from the model 4, 43, but does not remove all the garments 2 from model 4, 43.

[0028] De-clothing neural network 53 is typically trained as follows. The starting inputs are the styled image (Model image 1) and the model image to be de-clothed (Model image 2) in the style of Model image 1. The training of network 53 is done in three steps. First, the human body pose and the per-pixel semantic labels (cloth, skin, hair, etc.) of each of Model image 1 and Model image 2 are extracted by a dense pose neural network and a semantic label neural network, respectively. Second, a style encoder neural network is trained to extract the wearing style from the pose and semantics of Model image 1 using an auto-encoder method. Third, a style predictor neural network is trained on a masked-out region of the semantics of Model image 2 to predict the unmasked version of Model image 2, using supervised learning (i.e., theoutput is known). The masked-out region corresponds to the area to be de-clothed from Model image 2.

[0029] Figure 6 illustrates an embodiment of the present invention in which paired data and an estimated digitized flat garment 62 are used to enhance the performance of neural network 7. This embodiment can be referred to as “semi-supervised learning (training) from paired and unpaired data”. In this embodiment, the “paired data” are the image of the garment 2 and the image of the model 1 wearing that garment 2, as shown on the top half of Fig. 6. The “unpaired data” are images of the same model 1 wearing the same garment 2 but in different poses i, j, as shown on the bottom half of Fig. 6.

[0030] The Fig. 6 embodiment introduces the concept of an estimated digitized flat garment 62, which is produced by a dedicated garment estimator neural network 61. Network 61 is supervised in a training step shown in the upper left of Fig. 6. The training is supervised in that the output of network 61 , estimated flat garment 62, is known. The input to network 61 is a set 63 of images of selected garment 2. These images 63 comprise images of garment 2 in different positions, and / or images of the garment 2 on a single model 4, 43 in different poses, and / or images of the garment 2 on different models 4, 43. Adequate results can be obtained with as few as three or four input images 63.

[0031] In a separate training step, shown in the upper right of Fig. 6, tryon neural network 7 (see Fig. 4) is also subjected to supervised training. In this step, the inputs to network 7 are the partially de-clothed model images 6, 52 (which are obtained by subjecting a set of digitized garmented model images 1 to de-clothing neural network 53 as in Fig. 4 and / or Fig. 5) and a set of ground truth digitized flat garments 2. The output of tryon neural network 7 is a set of estimated tryon output images 60 that have been supervised with ground truth model images 1 and ground truth garment images 2.

[0032] The bottom half of Fig. 6 shows the present invention's technique of training the neural networks 61 and 7 using unpaired data consisting of different frames (i and j) of digitized garmented model image 1. Frame i is used to produce an estimated flat garment 62 using garment estimator neural network 61 . Frame j is fed to de-clothing neural network 53 to produce partially de-clothed model images 6, 52. Images 6, 52,62 are fed as inputs to tryon neural network 7, which produces the set of tryon output images 4.

[0033] In a preferred embodiment of the present invention, supervised training of networks 61 and 7 as shown in the top half of Figure 6 is performed first, to initialize the learning of garment estimator neural network 61 and tryon neural network 7. After these initial trainings have been incorporated in networks 61 and 7, networks 61 and 7 are trained further, with a combination of paired and unpaired data in the same training batch. In a typical training batch, there are N input images, where roughly N / 2 input images comprise paired data and roughly N / 2 input images comprise unpaired data. Because this final training step consists of both supervised training using paired data and self-supervised training using unpaired data, this training technique can be referred to as “semi-supervised learning” or “semi-supervised training”.

[0034] The above description is included to illustrate the operation of preferred embodiments, and is not meant to limit the scope of the invention. The scope of the invention is to be limited only by the following claims. From the above discussion, many variations will be apparent to one skilled in the art that would yet be encompassed by the spirit and scope of the present invention.

[0035] What is claimed is:

Claims

AMENDED CLAIMS received by the International Bureau on 31 July 2024 (31.07.2024)

1. A method for fitting a digitized representation of a garment onto a digitized representation of a model in a photorealistic manner using exactly two neural networks, said method comprising the steps of: digitally superimposing a plurality of control points on the digitized representation of the garment, and digitally superimposing a corresponding plurality of control points on the digitized representation of the model, by a control point neural network, wherein there is a one-to-one correspondence between each garment control point and a corresponding model control point; and feeding the digitized representation of the garment including control points and the digitized representation of the model including control points as inputs to a primary neural network; wherein the primary neural network outputs a digitized representation of the model garmented with the digitized representation of the garment.

2. The method of claim 1 where the digitized representation of the garment and the digitized representation of the model are each two dimensional.

3. The method of claim 1 where the model is a human.

4. The method of claim 1 where the model is an inanimate object.

5. The method of claim 1 where a DensePose image of the model is fed as a third input to the primary neural network.

6. The method of claim 1 where the primary neural network is trained in a single training phase in which: a plurality of digitized garments and a plurality of digitized garmented models are fed as inputs to the primary neural network; the plurality of garment control points are digitally superimposed on the digitized garments by the control point neural network; and the corresponding plurality of model control points are digitally superimposed on the digitized garmented models by the control point neural network.16AMENDED SHEET (ARTICLE 19)

7. The method of claim 6 wherein: the plurality of digitized garments comprise different garments of different sizes; and the plurality of digitized garmented models comprise a combination of the same model in different poses and different models having different sizes and / or shapes.

8. The method of claim 6 where the digitized garments and the digitized garmented models are fed as inputs to the primary neural network in pairs.

9. The method of claim 6 where, subsequent to the training phase, a testing phase is performed in which a subset of the digitized garments and a subset of the digitized garmented models are held back from the training phase, and the held back digitized garments and held back digitized garmented models are fed as inputs to the primary neural network during the testing phase.

10. A neural network configured to fit a digitized representation of a garment onto a digitized representation of a model, said neural network comprising: a first input port adapted to receive the digitized representation of the garment including a first plurality of control points superimposed on the digitized garment; and a second input port adapted to receive the digitized representation of the model including a second plurality of control points superimposed on the digitized model; wherein: there is a one-to-one correspondence between the first plurality of control points and the second plurality of control points.

11. A method for fitting a digitized representation of a garment onto a digitized representation of a model in an enhanced photorealistic manner, said method comprising the steps of: feeding the digitized representation of the model to a declothing neural network to produce a partially declothed digitized model; feeding the partially declothed digitized model and the digitized representation of the garment to a second neural network to produce a raw digitized model garmented with the digitized garment; and17AMENDED SHEET (ARTICLE 19)feeding the raw digitized model garmented with the digitized garment and a set of garment exemplars to a third neural network to produce an enhanced digitized model garmented with the digitized garment.

12. The method of claim 11 where the third neural network is different than the second neural network.

13. The method of claim 11 where the third neural network is the same as the second neural network.

14. The method of claim 11 where the set of exemplars comprises: a more detailed image of the garment than the image of the garment that was fed as an input to the second neural network; images of different models wearing the garment; and images of a single model wearing the garment in different poses.

15. Apparatus for fitting a digitized representation of a garment onto a digitized representation of a model, said apparatus comprising: a declothing neural network configured to accept as an input the digitized representation of the model, said declothing neural network configured to produce as an output a partially de-clothed digitized model; a second neural network configured to accept as inputs the partially de-clothed model and the digitized representation of the garment, said second neural network configured to produce as an output a first digitized representation of the model clothed with the digitized garment; and a third neural network configured to accept as inputs the first digitized representation of the model clothed with the digitized garment and a set of digitized garment exemplars, said third neural network configured to produce as an output a second digitized representation of the model clothed with the digitized garment.

16. A styled declothing method for partially declothing a digitized multi-garmented model, said method comprising the steps of: feeding a plurality of images of the digitized multi-garmented model to a declothing neural network, where each image18AMENDED SHEET (ARTICLE 19)illustrates the model wearing at least two garments in different poses; and invoking the declothing neural network to output a digitized garmented model in which at least one garment has been removed from the model.

17. The method of claim 16 where at least two garments are worn as layers of clothing on each of the model images.

18. The method of claim 16 where no garment completely covers a model image.

19. A neural network configured to partially declothe a digitized representation of a multi-garmented model, said neural network comprising: an input port adapted to receive a plurality of inputs comprising the digitized multi-garmented model in different poses; and an output port adapted to produce a digitized representation of the model in which the model has been de-clothed in an area corresponding to at least one of the garments.

20. A method for training a tryon neural network to virtually fit a digitized garment onto a digitized model, said method comprising the steps of: feeding an image of a digitized model in a first pose to a garment estimator neural network to produce a digitized estimated flat garment; feeding an image of the digitized model in a second pose to a declothing neural network to produce a declothed model image; and feeding the digitized estimated flat garment and the declothed model image to the tryon neural network to produce a tryon output image.

21. The method of claim 20 where the garment estimator neural network has been previously trained in a supervised training step in which a set of images of the garment are fed as inputs to the garment estimator neural network to produce a digitized estimated flat garment supervised with the ground truth input garment.

22. The method of claim 21 where the set of images of the garment comprises at least one of: images of the garment in different positions;19AMENDED SHEET (ARTICLE 19)images of the garment on different poses of a single model; and images of the garment on different models.

23. The method of claim 20 where the tryon neural network has been previously trained in a supervised training step in which a set of de-clothed model images and a set of ground truth flat garments are fed as inputs to the tryon neural network, thereby producing a set of estimated tryon output images that have been supervised with ground truth images of the models and flat garments.

24. The method of claim 20 where: the garment estimator neural network and the tryon neural network have been previously subjected to supervised training; and the garment estimator neural network and the tryon neural network are subsequently trained with a combination of paired and unpaired data in the same training batch.

25. The method of claim 24 where roughly half of the images in the training batch comprise paired data, and roughly half of the images in the training batch comprise unpaired data.

26. Apparatus for training a tryon neural network designed to virtually fit a digitized garment onto a digitized model, said apparatus comprising: a garment estimator neural network configured to accept as an input an image of a digitized model in a first pose, and configured to produce an output comprising a digitized estimated flat garment; a declothing neural network configured to accept as an input an input of the digitized model in a second pose, and configured to produce an output comprising a declothed model image; and a tryon neural network configured to accept as inputs the digitized estimated flat garment and the declothed model image, and configured to produce as an output a set of tryon digitized images comprising the model wearing the garment.

27. The method of claim 9 wherein the digitized garments and digitized garmented models fed as inputs to the primary neural network during the testing phase are different than the20AMENDED SHEET (ARTICLE 19)digitized garments and digitized garmented models fed to the primary neural network during the training phase.

28. The method of claim 1 further comprising the step of the primary neural network fitting a warped digitized garment onto the digitized representation of the model.

29. The method of claim 1 wherein the digitized representation of the garment is two dimensional with a front and a back, and both the front and the back are taken into account in the fitting of the digitized representation of the garment onto the digitized representation of the model.

30. The method of claim 6 wherein, during the training phase, the number of digitized garmented models is between four and five times the number of digitized garments.21AMENDED SHEET (ARTICLE 19)

Images