Method and system for automatically generating large training data sets from 3D models for training deep learning networks

By augmenting 3D models with diverse data to generate extensive 2D image datasets, the method addresses the challenge of training deep learning networks with limited data, enabling accurate 3D measurement extraction from uncontrolled environments.

JP7579546B2Active Publication Date: 2024-11-08BODYGRAM INC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2021534791
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-07-19
Filing Date
2019-12-17
Publication Date
2024-11-08
Estimated Expiration
2039-12-17

AI Technical Summary

Technical Problem

Existing methods for training deep learning networks to extract 3D measurements from 2D images require large amounts of annotated data, which is difficult to obtain due to the need for human intervention and controlled environments, limiting the scalability and efficiency of the training process.

Method used

A method and system for generating large datasets by augmenting 3D models with various types of data, such as skin color, facial features, clothing, and lighting conditions, and projecting them onto 2D images to create a vast array of training examples, effectively amplifying small datasets into millions of training data points.

Benefits of technology

This approach significantly expands the training data available for deep learning networks, allowing for accurate measurement extraction from 2D photographs taken in real-world conditions without requiring controlled environments, thus enhancing the training efficiency and reducing the need for manual annotation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007579546000002
    Figure 0007579546000002
  • Figure 0007579546000003
    Figure 0007579546000003
  • Figure 0007579546000004
    Figure 0007579546000004
Patent Text Reader

Abstract

A system and method are disclosed for generating a large dataset for training a deep learning network (DLN) to extract 3D measurements from 2D images captured using a mobile device camera. The method includes receiving a 3D model of a 3D object, extracting spatial features from the 3D model, generating a first type of augmentation data for the 3D model, such as, but not limited to, skin color, facial contours, hairstyle, virtual clothing, and / or lighting conditions, augmenting the 3D model with the first type of augmentation data to generate an augmented 3D model, generating at least one 2D image from the augmented 3D model by projecting the augmented 3D model onto at least one surface, and generating a training dataset for training the deep learning network (DLN) for spatial feature extraction by aggregating the spatial features and the at least one 2D image.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] Embodiments of the present invention are in the field of automated 3D object measurement, and in particular, relate to generating datasets for training deep learning algorithms for the measurement of 3D objects using photographs taken with a mobile device. [Background technology]

[0002] The description in the background of the invention is provided to aid in the understanding of the invention and its applications and uses and may not constitute prior art.

[0003] Several approaches have been attempted to extract 3D measurements from images of 3D objects, including using special 3D cameras as well as 2D videos or 2D photographs, followed by 2D-to-3D reconstruction techniques to estimate the 3D measurements.

[0004] A promising new technique described in U.S. Patent Application No. 10,321,728, entitled "Systems and methods for full body measurements extraction," filed on November 19, 2018 as U.S. Patent Application No. 16 / 195,802, and issued on June 18, 2019, utilizes deep learning networks to extract measurements from 2D photographs taken with a single mobile device camera. This technique promises to produce highly accurate measurements while minimizing user friction. However, this approach may require a large amount of training data, including segmented and annotated frontal and profile images and corresponding ground truth data, to train the deep learning network. In practice, it may be difficult to acquire the large training datasets required to train deep learning networks. Images must be acquired from willing volunteers, segmented, annotated by human annotators, and matched to ground truth data of actual measurements collected.

[0005] It would therefore be an advancement in the state of the art to provide a process that may be used to effectively amplify small data sets for training deep learning networks to extract measurements. In this way, small amounts of collected data can be effectively amplified to produce large data sets that may be used to train deep learning algorithms to accurately measure.

[0006] It is against this background that the present invention was developed. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] U.S. Patent No. 10,321,728 [Non-patent literature]

[0008] [Non-Patent Document 1] “Image Inpainting for Irregular Holes Using Partial Convolutions” by Guilin Liu et al. [Non-Patent Document 2] “AutoHair:Fully Automatic Hair Modeling from A Single Image” by Menglei Chai et al. [Non-Patent Document 3] Hengshuang Zhao et al., "Pyramid Scene Parsing Network," CVPR 2017, December 4, 2016. [Non-Patent Document 4] "Random Forests" by Leo Braiman, Machine Learning, 45, 5-32, 2001, Kluwer Academic Publishers, Netherlands. Summary of the Invention [Means for solving the problem]

[0009] The present invention relates to a method and system for generating large datasets for training deep learning networks to extract 3D measurements from 2D images, e.g., from a mobile device camera. In summary, according to one embodiment, multiple datasets may be generated from a single 3D base mesh model from an existing database or a human body scanner, adding augmented data (skin, face, hair, light, multiple virtual clothes, multiple virtual backgrounds, etc.). As a result, thousands or even millions of training datasets may be generated from only a single 3D model (e.g., a body scan of one person). In general, according to one aspect, the present invention is a virtual space and human augmentation system for deep learning (DL) and machine learning (ML) dataset generation.

[0010] More specifically, in various embodiments, the invention provides a computer-implemented method for generating a training dataset for training a deep learning network (DLN) for spatial feature extraction from a two-dimensional (2D) image of a three-dimensional (3D) object, the computer-implemented method being executable by a hardware processor, the method including: receiving a 3D model of the 3D object; extracting spatial features from the 3D model; generating a first type of augmentation data for the 3D model; augmenting the 3D model with the first type of augmentation data to generate an augmented 3D model; generating at least one 2D image from the augmented 3D model by projecting the augmented 3D model onto at least one surface; and generating the training dataset for training the deep learning network (DLN) for spatial feature extraction by aggregating the spatial features and the at least one 2D image.

[0011] In one embodiment, the method further includes receiving a second type of augmentation data for the at least one 2D image, augmenting the at least one 2D image with the second type of augmentation data to generate a plurality of augmented 2D images, each of which includes a projection of the augmented 3D model and an example of the second type of augmentation data, and generating a training dataset for training a deep learning network (DLN) for spatial feature extraction by aggregating the spatial features and the plurality of augmented 2D images.

[0012] In one embodiment, the 3D model is a 3D human body model. In one embodiment, the spatial features are one-dimensional (1D) body measurements. In one embodiment, the spatial features are 2D body segments.

[0013] In one embodiment, the first type of augmentation data is selected from the group consisting of skin color, face shape, hairstyle, and virtual clothing.

[0014] In one embodiment, the first type of augmented data is lighting conditions.

[0015] In one embodiment, the second type of augmented data is a background image representing various backgrounds. In one embodiment, the method further includes performing segmentation on the augmented 2D image utilizing a plurality of data points from the 3D model to extract one or more features associated with the 3D model from the background. In one embodiment, the method further includes performing line annotation on the augmented 2D image utilizing a plurality of data points from the 3D model to generate annotation lines on each spatial feature corresponding to the spatial feature measurements.

[0016] In one embodiment, the method further includes generating ground truth feature measurements from the spatial features using a plurality of data points from the 3D model.

[0017] In one embodiment, the second type of augmented data is white noise.

[0018] In one embodiment, the second type of augmentation data is perspective distortion data.

[0019] In one embodiment, the method further includes generating a second training dataset from the second 3D model to train the deep learning network for spatial feature extraction, the second training dataset comprising a second plurality of spatial features, a second plurality of augmented 3D models, a second plurality of 2D images, and a second plurality of augmented 2D images.

[0020] In one embodiment, the method further includes training the deep learning network for spatial feature extraction using the second training data set.

[0021] In one embodiment, the at least one 2D image is selected from the group consisting of a front view image of the 3D model, a side view image of the 3D model, and an image projected from the 3D model at an angle of approximately 45 degrees relative to the front view image.

[0022] In one embodiment, the 3D model is a 3D body model, and the method further includes training a sizing machine learning module that is trained on ground truth data that includes 1D body measurements extracted from the 3D body model.

[0023] In one embodiment, the sizing machine learning module is trained on ground truth data that includes 1D body measurements extracted from a 3D body model and one or more user parameters associated with the 3D body model.

[0024] In one embodiment, the user parameters are selected from the group consisting of height, weight, gender, age, and demographic information associated with the user.

[0025] In one embodiment, the sizing machine learning module includes a random forest algorithm.

[0026] In one embodiment, the 3D body model includes at least one 3D body model of a nude user or a semi-nude user.

[0027] In one embodiment, the deep learning network includes a convolutional neural network (CNN).

[0028] In one embodiment, the deep learning network further includes a pyramid pooling module.

[0029] In one embodiment, the at least one 3D model is received from a 3D scanner, hi some embodiments, the 3D human model is received from a 3D body scanner of the human body.

[0030] In one embodiment, the method further comprises training a deep learning network for body measurement determination using the training dataset. In one embodiment, the method further comprises providing a trained deep learning network for body measurement determination.

[0031] In one embodiment, the method further includes post-processing the training dataset prior to providing the training dataset for training the deep learning network.

[0032] In various embodiments, a computer program product is disclosed that may be used to generate a training data set for training a deep learning network for spatial feature extraction from two-dimensional (2D) images of three-dimensional (3D) objects and may include a computer-readable storage medium having program instructions, i.e., program code, embodied therein, the program instructions being executable by a processor to cause the processor to perform the steps described above.

[0033] In various embodiments, a system is described that includes a memory that stores computer-executable components and a hardware processor that is operatively coupled to the memory and that executes the computer-executable components stored in the memory, where the computer-executable components may include components that are communicatively coupled to the processor to perform the steps described above.

[0034] In another embodiment, the invention is a non-transitory computer readable storage medium storing executable instructions that, when executed by a processor, cause the processor to perform a process for generating spatial features, the instructions causing the processor to perform the steps described above.

[0035] In another embodiment, the present invention is a system for extracting spatial features using a 2D telephone camera, the system comprising: a user device having a 2D camera, a processor, a display, and a first memory; a server having a second memory and a data repository; an electrical communication link between the user device and the server; and a plurality of computer codes embodied in the first and second memories of the user device and the server, the plurality of computer codes, when executed, causing the server and the user device to perform a process including the steps described above.

[0036] In yet another embodiment, the invention is a computerized server comprising at least one processor, a memory, and a plurality of computer codes embodied on said memory, which, when executed, cause said processor to perform a process including the steps set forth above. Other aspects and embodiments of the invention include methods, processes, and algorithms including the steps described herein, and include processes and modes of operation of the systems and servers described herein.

[0037] Still other aspects and embodiments of the present invention will become apparent from the detailed description of the invention when considered in conjunction with the accompanying drawings.

[0038] Embodiments of the invention described herein are illustrated by way of example and not by way of limitation.Embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which: FIG. [Brief description of the drawings]

[0039] [Figure 1]FIG. 1 illustrates an example tree demonstrating the steps of generating a large dataset for training a deep learning network for body measurement determination, according to one embodiment of the present invention. [Diagram 2] FIG. 1 is a flow diagram of an example of generating a large data set for training a deep learning network for spatial feature determination of a 3D model, according to one embodiment of the present invention. [Diagram 3] FIG. 1 is a flow diagram of an example of generating a large dataset for training a deep learning network for spatial feature determination of 3D human models, according to an embodiment of the present invention. [Figure 4] FIG. 1 is a flow diagram of an example of generating a large dataset for training a deep learning network for spatial feature determination of multiple 3D models, according to one embodiment of the present invention. [Diagram 5] FIG. 1 is a flow diagram of an example of generating a large dataset for training a deep learning network for spatial feature determination of a 3D model by augmenting the 3D model with multiple types of augmentation data, according to one embodiment of the present invention. [Figure 6A] 1 is an example input image (front view) of an example 3D model of a human to generate a training dataset for training a deep learning network for measurement determination. [Figure 6B] 1 is an input image (side view) of an example of a 3D model of a human example for generating a training dataset to train a deep learning network for measurement determination. [Figure 7A] FIG. 1 is an example segmented and annotated image for an example process of segmenting and annotating front view images (which may be performed manually by a human operator or automatically using an underlying 3D model) to train a deep learning network for measurement determination. [Figure 7B] FIG. 1 is an example segmented and annotated image for an example process of segmenting and annotating side view images (which may be performed manually by a human operator or automatically using an underlying 3D model) to train a deep learning network for measurement determination. [Figure 8A]FIG. 1 is a flow diagram of an example of training a deep learning network for measurement determination, according to an embodiment of the present invention. [Figure 8B] FIG. 11 is a flow diagram of another detailed example of training a deep learning network for body measurement determination, according to an embodiment of the present invention. [Figure 9A] FIG. 13 shows an example test run on a sample 2D image (frontal view) of a sample 3D model utilizing a deep learning network (DLN) for body measurement determination after the DLN has been trained. [Figure 9B] FIG. 13 shows an example test run on a sample 2D image (side view) of a sample 3D model utilizing a deep learning network (DLN) for body measurement determination after the DLN has been trained. [Figure 9C] FIG. 9C illustrates the results of an example test run on the sample 2D images from FIGS. 9A and 9B. [Figure 10] FIG. 2 illustrates an example process for generating a first type of augmented data including skin tone. [Figure 11] FIG. 2 illustrates an example process for generating a first type of augmented data that includes facial augmentation. [Figure 12] 1 is a flowchart illustrating a process for generating a first type of augmented data that includes a facial augmentation. [Figure 13] FIG. 1 illustrates an example architecture showing a deep learning network for body measurement determination. [Figure 14] FIG. 1 illustrates an example flow diagram of a deep learning network for body measurement determination. [Figure 15] FIG. 13 is another example flow diagram illustrating a deep learning network for body measurement determination. [Figure 16] FIG. 13 is yet another example flow diagram illustrating a deep learning network for body measurement determination. [Figure 17] FIG. 2 is an exemplary hardware architecture diagram of a computing device for implementing one embodiment of the present invention. [Figure 18]FIG. 1 is an exemplary system architecture diagram for implementing an embodiment of the present invention in a client-server environment. [Figure 19] FIG. 1 is an exemplary diagram illustrating a use case of the present invention where a single camera of a mobile device is used to capture anthropometric measurements, showing a frontal view of a human wearing typical clothing standing in front of a standard background. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0040] overview The embodiments of the present invention are described in detail below with reference to the drawings provided.

[0041] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention can be practiced without such specific details. In other instances, structures, devices, activities, and methods are shown using schematics, examples, and / or flow diagrams to avoid obscuring the present invention. Although the following description contains many specifics for purposes of illustration, those skilled in the art will recognize that many variations and / or modifications to the proposed details are within the scope of the present invention. Similarly, although many of the features of the present invention are described in relation to or in conjunction with each other, those skilled in the art will recognize that many of these features can be provided independently of other features. Thus, this description of the present invention is set forth without any loss of generality to, and without imposing limitations on, the present invention.

[0042] Various types of approaches to generate or extract 1D measurements from 2D images of 3D objects have been attempted by others. All of these approaches generally require the user, i.e. the object, to have a particular pose, to stand in front of an empty background at a particular distance from the camera, and, in the case of human subjects, to wear tight-fitting clothing and / or to be partially nude. Such requirements for a controlled environment and significant user friction are undesirable.

[0043] The present invention solves the above problems by providing a system and method for accurately extracting 1D measurements, such as body measurements, from 2D photos of: 1) objects in any pose; 2) objects in front of any background; 3) photos taken at any distance; 4) objects with any type of covering, such as clothed humans; and 5) light in any position, angle, color, or shadow, so that anyone can easily take a photo of a real-world object or themselves and benefit from complete measurement extraction. Some embodiments of the present invention do not involve 3D reconstruction or 3D models, and do not require a dedicated hardware camera. Instead, advanced computer vision combined with deep learning techniques is used to generate accurate measurements from photos provided by a simple mobile device camera, no matter how the object is positioned or covered. In this disclosure, the term "2D phone camera" is used to refer to any conventional camera mounted on or connected to a computing device, such as a smartphone, tablet, laptop, or desktop.

[0044] One difficulty with implementing deep learning networks to determine measurements from 2D photographs is the enormous size of the training data sets required to train the deep learning network. In one embodiment, the present invention solves these problems by providing methods, apparatus, and systems that significantly expand the amount of training data available from small samples of training data. For example, in one embodiment, as described in more detail below, thousands of 3D scans can be utilized and amplified to generate millions (or more) of training data points. In general, the present invention can be utilized to expand the amount of training data available to train a deep learning network by hundreds or thousands of times, or more.

[0045] Generate a dataset to train a deep learning network on body measurements FIG. 1 shows an example tree 100 demonstrating steps for generating a large data set for training a deep learning network for measurement determination, according to one embodiment of the present invention. Initially, a 3D model 101 is received. In step 110, a first type of augmented data is generated with a plurality of (P1) options, and the 3D model 101 is augmented with each option to generate a plurality of (P1) first stage augmented 3D models 111-114. In step 120, which may be optional, another type of augmented data is generated with a plurality of (P2) options, and the plurality of (P1) first stage augmented 3D models 111-114 are again augmented with each option in the plurality of (P2) options to generate a plurality of (P1×P2) second stage augmented 3D models 121-126. At step 130, at least one 2D image is generated from each of the multiple (P1×P2) augmented 3D models by performing a projection of the 3D model to generate multiple (P1×P2) 2D images 131-132. At step 140, a second type of augmented data is generated comprising multiple (Q) options, and the multiple (P1×P2) 2D images 131-132 are augmented with each option in the multiple (Q) options to generate multiple (P1×P2×Q) augmented 2D images 141-144. In this way, from a single 3D model 101, multiple (P1×P2×Q) augmented 2D images may be generated and included as training data used in training the deep learning network. Various embodiments of the process of generating the example tree 100 are further described in FIGS. 2-5.

[0046] 2 illustrates an example flow diagram 200 for generating a large amount of data set for training a deep learning network for spatial feature determination of a 3D model, according to an embodiment of the present invention. A 3D model 202 is received. The 3D model 202 may be a variety of 3D objects, such as a human body, an animal (e.g., dog, cat), a piece of furniture (e.g., sofa, table), a vehicle (e.g., car, motorcycle), or a musical instrument (e.g., piano). For example, in one embodiment, the 3D human body model may be obtained from a 3D body scanner (not shown).

[0047] In the case of 3D human models, and according to some embodiments, 3D body scan data from the "SizeUSA" dataset, a commercially available sample of 3D body scans obtained on approximately 10,000 subjects (both male and female), may be used as the source of the 3D model. In other embodiments, 3D body scan data from the "CAESAR" dataset, another commercially available sample of 3D body scans obtained on approximately 4,000 subjects, which also includes ground truth data manually measured using human tailors, may be used. In yet other embodiments, institutions utilizing the present invention may capture and / or generate 3D models of any 3D object of interest itself.

[0048] In step 204, a first type of augmented data is generated with a plurality (P1) of options. An example of the first type of augmented data of a 3D human body model is described in Fig. 3. In step 206, the 3D model 202 is augmented with each option to generate in step 208 a plurality (P1) of first stage augmented 3D models 111-114.

[0049] In step 210, a plurality (P1) of 2D images 131-132 are generated from a plurality (P1) of first stage augmented 3D models. The generation may be performed, for example, by performing a projection of the 3D model onto a surface (front or side), for example using an orthographic projection. In some embodiments, both front and side images are taken.

[0050] In step 212, a second type of augmented data is generated having a plurality (Q) of options, and in step 214, the plurality (P1) of 2D images 131-132 are augmented with each option in the plurality (Q) of options to generate a plurality (P1×Q) of augmented 2D images 141-144.

[0051] In some embodiments, the second type of augmented data may be, for example, multiple (Q) background images. The background images represent a variety of possible backgrounds that may be observed by a subject in a real environment. The background images should represent different lighting, shadow, locality, and other background conditions that are typical for subjects taking on spatial features such as body measurements. The variety of backgrounds should be selected to optimize training of the deep learning network, particularly the segmentation deep learning network, in body segmentation as described below. The background images from the multiple (Q) background images received in step 212 are added to the front and side images from step 210 to generate 2×P2 images with a given background. In step 214, this process is repeated for the multiple (Q) background images, i.e., Q times, to generate a total of 2×P1×Q images.

[0052] In some embodiments, the second type of augmented data may be, for example, white noise or other types of random noise, which may be useful for rendering the deep learning network more robust to imperfect image capture and other environmental conditions. In other embodiments, the second type of augmented data may be, for example, perspective distortion data.

[0053] At step 218, spatial features are extracted from the 3D model to serve as "ground truth" data for training the annotation deep learning network and machine learning modules described below. In some examples, the spatial features are 1D measurements and corresponding annotation lines, such as shirt size, hat size, waist circumference, or chest circumference, of the human body model. The 1D measurements may be extracted from the 3D model by performing virtual measurements of the corresponding desired measurements along any given "virtual" annotation line in the 3D model, similar to how a human tailor measures a real 3D human body.

[0054] By combining the 2×P1×Q images from step 216 with the 1D measurements ("ground truth data") from step 218, it is possible to generate multiple (2×P1×Q) training data for training a deep learning network from only one 3D model, as shown in step 220. Finally, in step 222, the training data set is ready to be used in training a deep learning network.

[0055] 3 illustrates an example flow diagram 300 for generating a large dataset for training a deep learning network for spatial feature determination of a 3D model, according to one embodiment of the present invention. The steps are generally more specific examples of the steps of FIG. 2, e.g., 302 corresponds to 202, 304 corresponds to 204, 306 corresponds to 206, 308 corresponds to 208, 310 corresponds to 210, 312 corresponds to 212, 314 corresponds to 214, 316 corresponds to 216, 318 corresponds to 218, 320 corresponds to 220, and 322 corresponds to 222.

[0056] Examples of the first type of augmented data for the 3D human model include skin color, facial contours, hairstyle and hair color, virtual clothing, and lighting conditions. In one embodiment, in step 304, a plurality (U) of skin colors are generated that represent various possible skin colors that the subject may have. These skin colors may encompass the range of skin colors naturally present in human subjects, as well as colors created by other means, such as tanning and tattoos. In step 306, the plurality (U) of skin colors generated in step 304 are added to the 3D body model 302 to generate a plurality (U) of augmented 3D human models including various skin colors in step 308.

[0057] In another embodiment, in step 304, multiple (V) facial contours are generated that represent various possible facial contours that the subject may have. Such facial contours may include, for example, the shape of the eyes, nose, cheeks, chin, jaw line, mouth, and forehead. The facial contours are also independent of the body shape of the 3D body model 302. In step 306, the multiple (V) facial contours generated in step 304 are added to the 3D body model 302 to generate multiple (V) augmented 3D human models including various facial contours in step 308.

[0058] In another embodiment, in step 304, multiple (W) hairstyles and hair colors are generated that represent various hairstyles and hair colors that the subject may have. Such hairstyles may include hair of various lengths, hair with various levels of straightness or curlyness, various haircuts, and hairlines with various stages of baldness. Such hair colors may include colors that are naturally present in human subjects (e.g., white, blonde, red, brown, black), as well as colors that are created by other means such as dyeing (e.g., pink, blue, green, purple). In step 306, the multiple (W) hairstyles and hair colors generated in step 304 are added to the 3D body model 302 to generate multiple (W) augmented 3D human models including various hairstyles and hair colors in step 308.

[0059] In another embodiment, in step 304, a number (X) of virtual garments are generated representing various fitting garment possibilities that the subject may wear, including but not limited to tight fit, regular fit, loose fit, etc. In step 306, the number (X) of virtual garments generated in step 304 are added to the 3D body model 302 to generate a number (X) of augmented 3D human models including the various virtual garments in step 308.

[0060] In another embodiment, in step 304, multiple (Y) illuminants are generated representing various positions, angles, colors, and shadows that may be on the subject, including but not limited to natural sunlight, indoor lighting, moonlight, camera flash, street lights, etc. In step 306, the multiple (Y) illuminants generated in step 304 are added to the 3D body model 302 to generate multiple (X) augmented 3D human models including the various illuminants in step 308.

[0061] In some implementations, skin color, facial contour, hairstyle, and lighting augmentation are performed using image inpainting techniques. For example, a PyTorch implementation of partial convolution layers is described at https: / / github.com / NVIDIA / partialconv. In some implementations, skin color augmentation may be performed by adjusting the RGB curves of the face rendering based on skin color statistics. The statistics may follow various distributions, including, for example, a uniform distribution or a Gaussian distribution.

[0062] By combining the 2×P1×Q images (where P1 is either U, V, W, X, or Y) from step 316 with the 1D measurements ("ground truth data") from step 318, it is possible to generate multiple (2×P1×Q) training data for training a deep learning network from only one 3D body model, as shown in step 320. Finally, in step 322, the training data set is ready to be used in training a deep learning network.

[0063] 4 illustrates an example flow diagram 400 for generating a large dataset for training a deep learning network for spatial feature determination of multiple 3D models, according to an embodiment of the present invention. Flow diagram 400 illustrates an iteration or loop of flow diagram 200 of FIG. 2 for multiple (N) 3D models. Thus, the following steps correspond to each other: 402 corresponds to 202, 404 corresponds to 204, 406 corresponds to 206, 408 corresponds to 208, 410 corresponds to 210, 412 corresponds to 212, 414 corresponds to 214, 416 corresponds to 216, 418 corresponds to 218, 420 corresponds to 220, and 422 corresponds to 222.

[0064] In step 421, the creation of a training data set for a single 3D model as completed in step 420 is repeated multiple times (N times) for multiple (N) 3D models, resulting in P1×Q×N training data from only N 3D models (2×P1×Q×N if two images are generated from each augmented 3D model), as shown in output element 422. The training data 422 may then be used to train a deep learning network, as described below in connection with Figures 8A and 8B.

[0065] 5 shows an example flow diagram 500 of generating a large amount of data set for training a deep learning network for spatial feature determination of a 3D model by augmenting the 3D model with multiple types of augmentation data according to an embodiment of the present invention. Flow diagram 500 is similar to flow diagram 200 of FIG. 2, but the augmentation steps are repeated or looped, as described below. Thus, the following steps correspond to each other: 502 corresponds to 202, 504 corresponds to 204, 506 corresponds to 206, 508 corresponds to 208, 510 corresponds to 210, 512 corresponds to 212, 514 corresponds to 214, 516 corresponds to 216, 518 corresponds to 218, 520 corresponds to 220, and 522 corresponds to 222.

[0066] In step 504, a first type of augmented data is generated with a plurality of options (P1). In step 506, the 3D model 502 is extended with each option to generate a plurality of (P1) first stage augmented 3D models 111-114. In step 507, steps 504 and 506 are repeated for another type of augmented data with a plurality of options (P2), and the plurality of (P1) first stage augmented 3D models 111-114 are again extended with each option in the plurality of options (P2) to generate a plurality of (P1×P2) second stage augmented 3D models 121-126 in step 508. Step 507 is repeated any finite number of times (M) to generate a plurality of (P1×P2×P3×...×P M For example, steps 504-507 may be performed (M=5) on a 3D human body model having multiple (U) skin colors, multiple (V) facial contours, multiple (W) hairstyles, multiple (X) types of virtual clothing, and multiple (Y) lighting conditions to generate multiple (U×V×W×X×Y) augmented 3D body models having various combinations of those features.

[0067] In step 514, a second type of augmented data is generated with a plurality of (Q1) choices, and a plurality of (P1×P2×P3×...×P M ) 2D images 131-132 are expanded with each option in the multiple options (Q1) to create multiple (P1×P2×P3×…×P M ×Q1) extended 2D images 141-144 are generated. In step 515, steps 512 and 514 are repeated for another type of extended data with multiple (Q2) options, and the multiple (Q1) first stage extended 2D images are again extended with each option in the multiple (Q2) options to generate multiple (Q1×Q2) second stage extended 2D images in step 516. Step 515 is repeated any finite number of times (L) to generate multiple (P1×P2×P3×...×P M ×Q1×Q2×Q3×…×Q L For example, multiple (P1×P2×P3×...×P) augmented 2D images may be generated having multiple (Q1) background images, multiple (Q2) noise terms, and multiple (Q3) perspective distortion data terms. M) 2D images, perform steps 512 to 515 (L=3) to generate multiple (P1×P2×P3×…×P M 2D image (Q1×Q2×Q3) is generated.

[0068] It should be noted that the processes disclosed in Figures 2-5 may be combined. For example, the iteration of multiple (N) 3D models in step 421 of flow diagram 400 may be combined with the iteration step 507 (M times) for the first type of augmented data and the iteration step 515 (L times) for the second type of augmented data. Other combinations are within the scope of the invention.

[0069] In one embodiment, one or more user parameters associated with each subject corresponding to each 3D model may be received, not shown in FIGS. 2-5. In the case of a 3D human model, the one or more user parameters may include height, weight, sex, age, and / or additional demographic information associated with each subject of each 3D human model, and any subset thereof. For example, in one embodiment, the user parameter is the height of the user, and in another embodiment, the user parameter is the height and weight of the user. In one embodiment, these user parameters may be used to train a sizing machine learning module on these additional user parameters associated with each subject. In one embodiment, the sizing machine learning module is trained on ground truth data including 1D body measurements extracted from the 3D human model and one or more user parameters associated with the 3D human model.

[0070] Example of generating training data for deep learning networks To understand the benefits of the present invention, it is useful to compare the process described in Figures 2-5 for generating a training dataset with a typical process used to generate segmented, annotated front and side images for training a deep learning network. In that regard, Figures 6A and 7A show an example annotation process for annotating a front view photograph of a subject for training a deep learning network for body measurement determination, and Figures 6B and 7B show an example annotation process for annotating a side view photograph of a subject for training a deep learning network for body measurement determination, with Figures 6A and 6B being input images to the process (before annotation) and Figures 7A and 7B being output images of the process (after annotation). Although a particular user pose "A pose" is shown in Figures 6A, 6B, 7A, and 7B, it will be understood by those skilled in the art that any pose is within the scope of the present invention, such as A pose, hands on body pose, or any other pose. An optimal pose would be one in which the legs and arms away from the body are clearly visible. One advantage of the present invention is that a person can stand in front of any type of background and in almost any reasonable posture - the person does not need to stand in front of a blank background or make special arrangements to take a photograph.

[0071] Figures 6A, 6B, 7A, and 7B show example diagrams 610, 620, 710, and 720 of an annotator, or operator, manually segmenting one or more features of a human body beneath clothing from the background in order to train a segmentation DLN. In Figures 6A, 6B, 7A, and 7B, an annotator has manually annotated the body beneath clothing. Humans have a lot of experience seeing other humans and guessing their body shapes beneath clothing, and this data is used to train a segmentation DLN to automatically perform a similar operation on new photos of unknown humans.

[0072] 7A and 7B show example diagrams of the physical features of a human body segmented from the background after an annotator has successfully annotated all the physical features. In the example diagrams 710 and 720, the dashed lines represent the segmentation of the body parts and the solid lines represent the annotation lines for the body part measurements. In other embodiments, other prominent markings may be used to indicate the two types of segmentation and annotation. The annotations, for example, identify the base of the neck or top of the torso 711 and 721, the base of the upper arms 712 and 722, the waist 713 and 723, and the tops of the ankles 714 and 724.

[0073] Additionally, the user's height may also be annotated. In the example diagrams 710 and 720, the top and bottom of the user are marked by small circles just above the user's head and between the user's feet, respectively. This height information may be used to accurately scale the 2D images, for example to convert other measurements from pixels to real-world units (e.g., cm). This data is used to train a human segmentation DLN.

[0074] The process described with respect to Figures 2-5 can be used to automate the very tedious process of manually generating segmentation and annotation data, as described with respect to Figures 7A and 7B. Instead of a human annotator having to manually segment the body under the clothing from the background and manually add annotation lines corresponding to the measurement lines, the process of Figures 2-5 can be used to automatically generate segmentation and annotation data from a 3D model. The 3D model already contains data corresponding to the contour lines of the human under the virtual clothing, as well as data corresponding to the annotation of the measurement lines. This data is potentially stored in the 3D model and can be extracted from the 3D model and used to generate large amounts of training data, as described above with respect to Figures 2-5.

[0075] In addition, corresponding 1D measurements have also been extracted from the 3D model, so that this data can also be used as "ground truth" data corresponding to actual anthropometric measurements and used to train the sizing machine learning module, also described below.

[0076] Thus, using the principles taught in this disclosure, it is possible to avoid manual segmentation of the body under the clothes, as well as manual annotation of measurement lines on the human body. Furthermore, the present invention alleviates the need to obtain "ground truth" data from real human tailors. Together, the present invention significantly reduces the effort and removes many manual steps from the process of collecting and generating a training data set useful for training a deep learning network. In addition, the principles taught in this invention allow the amount of training data to be significantly expanded, i.e., increased. Large amounts of useful training data may be generated quickly and efficiently from only a relatively small set of 3D models, without the need for front and side photos, or for a real tailor to measure the sample subject.

[0077] While only front and side views are shown in Figures 6A, 6B, 7A, and 7B, one skilled in the art would recognize that any other orientation of view, such as 45 degree views, top views, etc., are within the scope of the present invention, depending on the type of body measurements desired. For example, head measurements used to manufacture custom hats would be best suited to a top-view photograph from the top of a person's head. Similarly, face measurements used to size eyeglasses, ophthalmic devices, etc. would be best suited to a frontal face-only photograph. Close-up photographs of the front and back of a person's hand could be used to size custom gloves, custom PPE for hands, custom nails, etc.

[0078] Training a deep learning network with the generated dataset The starting point of any machine learning method used, such as by a deep learning network, is a documented dataset that contains multiple instances of system inputs and correct outcomes (e.g., training data). This dataset can be used to train the machine learning system and evaluate and optimize the performance of the trained system using methods known in the art, including, but not limited to, standardized machine learning methods, such as parametric classification methods, non-parametric methods, decision tree learning, neural networks, methods that combine both inductive and analytical learning, and modeling strategies such as regression models. The quality of the output of a machine learning system depends on (a) the pattern parameterization, (b) the learning machine design, and (c) the quality of the training database. These components can be refined and optimized using various methods. For example, the database can be refined by adding a dataset for a new documented subject. The quality of the database can be improved, for example, by populating the database with cases where customization was performed by one or more experts. Thus, the database becomes better representative of the experts' knowledge. In one embodiment, the database includes data for matching examples that can assist in the evaluation of the trained system.

[0079] Thus, generating training data from 3D models taken from a variety of sources to train a deep learning network as described in this disclosure would significantly advance the state of the art in deep learning network training. The following description describes how, after a large training dataset is generated as described above, the generated dataset is used to train a deep learning network.

[0080] FIG. 8A shows an example flow diagram 800 of training a deep learning network for measurement determination, according to one embodiment of the present invention. Consider a generalized case where there are multiple (N) 3D models, multiple (M) categories of a first type of augmentation, multiple (L) categories of a second type of augmentation, and two images (front and side) are taken for each augmented 3D model. In step 802, multiple (2×P1×P2×P3×...×P M ×Q1×Q2×Q3×…×Q L ×N) training data is generated. In step 804, multiple (2×P1×P2×P3×...×P M ×Q1×Q2×Q3×…×Q L ×N) of training data is used to train a deep learning network for both segmentation and annotation, as well as to train a machine learning module for measurement extraction, as described in more detail below. Finally, in step 806, the trained segmentation DLNs, annotation DLNs, and machine learning module are output and utilized for body measurement determination, as described in more detail below.

[0081] In summary, the process described in Figures 2-5 results in a significantly increased training data set (2 × P1 × P2 × P3 × … × P) from only a small number (N) of 3D models. M ×Q1×Q2×Q3×…×Q L ×N) are generated. A modest number of P1, P2, P3, …, P M , Q1, Q2, Q3, …, Q L, and N, a very large training data set may be generated from a small number (N) of 3D models. For example, using the principles taught in Figures 2-5, it is possible to generate only two (P1) different skin colors, two (P2) different facial contours, two (P3) different hairstyles, ten (P4) different types of virtual clothing, two (P5) different lighting conditions, two images (front and side) of each augmented 3D model, 100 (Q1) different backgrounds, and two (Q2) different noise profiles to generate 640,000,000 training data points from only 10,000 3D models (2 x 2 x 2 x 10 x 2 x 100 x 2 x 10,000), which is a factor of 64,000, or an increase in the size of the training data set of around 4.5 times.

[0082] FIG. 8B shows another detailed example flow diagram 850 of training a deep learning network for body measurement determination according to an embodiment of the present invention. FIG. 8B shows an example flow diagram diagram 850 of training a segmentation DLN, annotation DLN, and sizing ML used to generate body measurements according to an exemplary embodiment of the present invention. At step 852, one or more photographs or generated augmented 2D images are received. In one embodiment, front and side view augmented 2D images are generated from the 3D model as described above, and the system utilizes the generated or "virtual" front and side view images throughout the remainder of the process described in FIG. 8B.

[0083] In another example, the photos may be obtained from a user device (e.g., a mobile phone, laptop, tablet, etc.). In another example, the photos may be obtained from a database (e.g., a social media database). In another example, the photos from the user include photos showing a front view and a side view of the user's entire body. As shown, in one example, the images may be taken at a specified distance (e.g., about 10 feet from the camera of the user device). In one example, an image of the user may be taken having a particular posture (e.g., arms in a given position, legs shoulder-width apart, back straight, "A posture", etc.). In another example, multiple images (e.g., front and side view photos) of a given position may be taken and an average image may be determined for each position. This may be done to increase accuracy. In another example, the user may be positioned in front of a particular type of background (e.g., having a neutral color, or a particular background image). In another example, the front and side view photos may be taken under similar lighting conditions (e.g., given brightness, shadows, etc.). In another example, the front and side view photographs may include images of the user wearing normally fitting clothing (e.g., not too loose or too tight). Alternatively and / or in addition, the front and side view photographs may include images of the user partially clothed (e.g., no shirt) or having a different type of fit (e.g., tight, loose, etc.), depending on the needs of the AI-based algorithms and associated processes.

[0084] In step 854, the segmented body parts are received to train the segmentation DLN. In one embodiment, as described above, the front and side view images are segmented using data from the 3D model, and the system utilizes the generated "virtual" segmented images throughout the remainder of the process described in FIG.

[0085] In another embodiment, an annotator may use human intuition to manually segment physical features such as body parts under clothing. In particular, body segmentation may be performed by a human to extract the human body, excluding clothing, from the background of the photograph. For example, a human annotator may be used to visually edit the photographs (e.g., trace and color code) and indicate which body parts correspond to which parts of the photograph to extract the human body, excluding clothing, from the background. In one embodiment, the photographs may include a human with hands in a 45 degree position ("A pose"), with different clothing in different environments. The body contours may be drawn with any suitable software platform, or a peripheral device (e.g., a smart pen) may be used to facilitate annotation. In another embodiment, a printout of the image may be used and manually segmented with a pen / pencil, and the segmented printout may be scanned and recognized by the system using one or more AI algorithms (e.g., computer vision-based algorithms).

[0086] In step 855, annotation lines for each body part are received to train the annotation DLN. In one embodiment, as described above, the front and side view images are annotated using data from the 3D model, and the system utilizes the generated or "virtual" annotated front and side view images throughout the remainder of the process described in FIG.

[0087] In another embodiment, a human annotator may use human intuition to draw estimated annotation (measurement) lines for each physical feature underneath the clothing. As shown, precise annotation lines may be manually drawn by a human annotator. The annotation lines may be drawn with any suitable software platform, or a peripheral device (e.g., a smart pen) may be used to facilitate annotation. In another embodiment, a printout of the image may be used and manually annotated with a pen / pencil, and the annotated printout may be scanned and recognized by the system using one or more AI algorithms (e.g., computer vision based algorithms).

[0088] At step 856, real human measurements for each physical feature may be received and serve as ground truth data. In one embodiment, the real human measurements are determined from 1D measurements taken from a 3D body model, as described above. In another embodiment, the real human measurements are determined from measurements collected by a human tailor. The real human measurements are used as "ground truth" data and may be used to train a machine learning module used by the system. For example, the real human measurements may be used to minimize an error or loss function (such as mean squared error, likelihood loss, logarithmic loss, hinge loss, etc.) associated with the machine learning module described below.

[0089] In one embodiment, the human measurements may be received from a user input (e.g., input on a user device such as a smartphone). In another embodiment, the human measurements may be received from a network (e.g., the Internet), such as through a website. For example, a tailor may upload one or more measurements to a website, and the system may receive the measurements. As will be shown, in another embodiment, the actual measurements may be used to train and / or improve the accuracy of an AI-based algorithmic result (e.g., a deep learning model), as described below.

[0090] At step 858, a segmentation DLN may be trained for body segmentation or physical feature extraction. In one embodiment, the segmentation DLN may be trained using the human body segmentation obtained at step 854. For example, the segmentation DLN may be represented using labeled data (e.g., images of a user and associated actual body segmentation), and an error function may be determined (e.g., from a loss function as described above) based on the results of the segmentation DLN and the actual body segmentation. The segmentation DLN may be trained to reduce the magnitude of the error function.

[0091] In another embodiment, the segmented DLN may be validated by an accuracy estimation technique such as a hold-out method, which may split the data (e.g., all images with corresponding segmentation and images without corresponding segmentation from which segmentation is extracted using the segmented DLN) in a training and test set (conventionally designated as training set 2 / 3 and test set 1 / 3), and may evaluate the performance of the segmented DLN model on the test set. In another embodiment, an N-fold cross-validation method may be used, in which the data is randomly split into k subsets, k-1 instances of the data are used to train the segmented DLN model, and the kth instance is used to test the predictive ability of the segmented DLN model. In addition to the hold-out and cross-validation methods, a bootstrap method may be used, in which n instances with replacement from the dataset are sampled to assess the accuracy of the segmented DLN model.

[0092] In step 860, one or more annotation DLNs for each physical feature may be trained, or a single annotation DLN may be trained for the whole body. For example, 16 annotation DLNs may be trained, each corresponding to 16 different body parts. In one embodiment, the annotation DLNs may be trained using the annotations obtained in step 856. For example, the annotation DLNs may be represented using labeled data (e.g., an image of a user with line annotations), and an error function may be determined (e.g., from a loss function as described above) based on the results of the annotation DLNs and the actual annotations. The annotation DLNs may be trained to reduce the magnitude of the error function.

[0093] In another embodiment, the annotation DLN may be specifically trained to draw annotation lines from a particular physical feature of a particular body part, such as, for example, an arm, a leg, or a neck. In another embodiment, the training of the annotation DLN for each physical feature may be performed sequentially (e.g., in a hierarchical fashion where a group of related physical features are trained one after the other) or in parallel. In another embodiment, different training data sets may be used for different annotation DLNs, with different annotation DLNs corresponding to different physical features or body parts. In one embodiment, there may be more or fewer than 16 DLNs for, for example, 16 body parts, depending on the computational resources. In another embodiment, the training of the annotation DLNs may be performed at least partially in the cloud, as described below.

[0094] Optionally, at step 860, one or more sizing ML modules for each physical feature may be trained, or alternatively, a single sizing ML module may be trained for the entire body. In one embodiment, the sizing ML module may be trained using the measurements obtained at step 856. For example, the sizing ML module may be represented using labeled data (e.g., annotation line lengths and associated actual measurement data) and an error function may be determined (e.g., from a loss function as described above) based on the results of the sizing ML module and the actual measurements. The sizing ML module may be trained to reduce the magnitude of the error function.

[0095] In another embodiment, the sizing ML modules may be trained specifically to extract measurements from a particular physical feature of a particular body part, such as, for example, an arm, a leg, or a neck. In another embodiment, the training of the sizing ML modules for each physical feature may be performed in series (e.g., in a hierarchical fashion where a group of related physical features are trained one after the other) or in parallel. In another embodiment, different training data sets may be used for different sizing ML modules, with different sizing ML modules corresponding to different physical features or body parts. In one embodiment, there may be more or fewer than 16 sizing ML modules for, for example, 16 body parts, depending on the computational resources. In another embodiment, the training of the sizing ML modules may be performed at least partially in the cloud, as described below.

[0096] At step 862, the trained segmentation DLN, annotation DLN, and optionally a sizing ML module may be output for use in determining body measurements. In particular, the segmentation DLN trained at step 858 is output for use in segmenting physical features. Similarly, one or more annotation DLNs trained at step 860 are output for annotation of physical features. Finally, optionally, the sizing ML module trained at step 860 is output for determining measurements of physical features.

[0097] Evaluating a Trained Deep Learning Network 9A and 9B show an example test run on sample 2D images (front view and side view, respectively) of a sample 3D model utilizing a deep learning network (DLN) for body measurement determination after the DLN has been trained. As mentioned above, FIGS. 6A and 6B are input images of an example annotation process for annotating front view and side view photos of a subject for training a deep learning network for body measurement determination. The same FIGS. 6A and 6B are considered as input images of an example test run on sample 2D images for body measurement determination. As mentioned above, a specific user pose "A pose" is shown in FIGS. 6A, 6B, 9A, and 9B, but it will be understood by those skilled in the art that any pose is within the scope of the present invention, such as A pose, hands on body pose, or any other pose. An optimal pose would be one in which the legs and arms away from the body are clearly visible.

[0098] After the test run is completed, the DLN outputs example diagrams 930 and 940 showing the front and side views, respectively, of a sample 3D model with annotation marks associated with body measurement determinations. In example diagrams 930 and 940, several such annotation marks are shown, including neck width 931 and 941, arm edges 932 and 942, wrist edges 933 and 943, torso edges 934 and 944, inseam peaks 935 and 945, and ankle peaks 936 and 946, all determined under clothing.

[0099] Similar to example diagrams 710 and 720, the user's height may also be annotated. In example diagrams 930 and 940, the top and bottom of the user are marked by small circles just above the user's head and between the user's feet, respectively. This height information may be used to accurately scale the 2D image, for example, to convert other measurements from pixels to real-world units (e.g., cm).

[0100] FIG. 9C illustrates the results of an example test run on the sample 2D images from FIGS. 9A and 9B. The results are displayed as an example table 950, where the first column indicates the body parts identified on the 3D model and the second column indicates the sizing results determined for those body parts. The body parts may include, for example, neck (circumference), shoulder (width), sleeve (length), wrist (circumference), and waist (circumference). In the example table 950, the body part "neck" in cell 951 is shown to be size "33.2227 [cm]" in cell 952. The body parts may also include garment measurements inferred from physical body parts, such as the outseam and inseam of pants that fit the particular 3D model. In some embodiments, the sizing results represent a best-spot estimate that reduces the mean squared error (MSE estimate). In some embodiments, the test run may also include a confidence interval for the sizing results (not shown). In test runs of the present invention, accuracy of less than one centimeter (<1 cm) has been achieved.

[0101] Example 1: Extension data of the first type First, why is the first type of augmentation data (e.g., face augmentation) necessary? In a typical dataset, for example, a typical 3D body dataset is generated from a 3D body model, and since many 3D body shape models are available, each rendered image has many body shapes. However, depending on a given base mesh, there are only a few, typically two or three, face patterns. A given base mesh is a unified mesh that wraps the 3D scanned body data. Therefore, in a virtual deep learning dataset generation, skin, face, hair, and other first type augmentations are necessary. We will consider these next.

[0102] Skin expansion Regarding skin augmentation, various implementations are possible according to the present invention. In one embodiment, the current generation system has skin color augmentation. Skin augmentation is implemented in one embodiment by skin augmentation shader code, as described in more detail below. The shader is a shading program of the 3D graphics program, and the shader has a random RGB curve that can be used to change the skin color from black to white and colors in between. Skin augmentation is done programmatically during the rendering of the 3D model. In one embodiment, a script modifies the shader RGB curve based on the statistics of the skin color, for example using a uniform, Gaussian, or other distribution.

[0103] For example, the script shown in Table 1 can be used to generate a skin color extension using a uniform distribution.

[0104] [Table 1]

[0105] FIG. 10 illustrates an example process 1000 for generating a first type of augmented data including skin tone. This process references steps 202-208 of FIG. 2, steps 302-308 of FIG. 3, steps 402-408 of FIG. 4, and steps 502-508 of FIG. 6. Initially, at step 1002, a 3D model is received. At step 1004, skin color statistics are received. The skin color statistics include information such as a range of skin colors (rendered as red, green, and blue RGB values), a quantization of the colors within the range, and the relative probability of each individual having a skin color within the range quantized by the quantization. In some embodiments, these skin color statistics correspond to the entire world population. In other embodiments, these skin color statistics correspond to the population of a particular country, region, or ethnicity. In other embodiments, these skin color statistics take into account tanning, tattoos, or other natural or artificial alterations to human skin color.

[0106] In step 1006, a skin color is generated from the skin color statistics by selecting a random skin color within the color range with an associated probability. In step 1008, the 3D model received in step 1002 is augmented with the generated skin color to generate an augmented 3D model having the selected skin color in step 1010.

[0107] In one embodiment, skin augmentation may be performed programmatically during the model rendering of skin using a shader or shading program of a 3D graphics program as described above. The shader may have a red-green-blue (RGB) curve based on skin color statistics that a script may programmatically adjust. For example, a color feature is first selected programmatically. When the script chooses to adjust the value of the color feature, a particular base color (cyan, red, green, blue) is programmatically selected by the script using a color chart with an RGB curve, and a particular color point is selected. The script can programmatically cycle through the color points to generate various skin tones or colors. The script can then programmatically apply the selected skin color to the surface of the model.

[0108] Facial expansion In one embodiment, face augmentation may be achieved using a partially convolutional deep learning network by 1) first rendering an image with the eyes, nose, and mouth masked, and 2) second applying the partially convolutional deep learning network to the masked image to obtain a face augmented image. Thus, it is possible to obtain a large number of realistically rendered humans with realistic faces from a limited number of 3D body models that contain only some facial variations.

[0109] 11 illustrates an example process 1100 for generating a first type of augmented data including facial augmentation. In particular, the facial augmentation may comprise the facial contours that a subject may have, such as the shape of the eyes, nose, cheeks, chin, jaw line, mouth, and forehead. The facial contours are also independent of the body shape of the 3D body model. In the process 1100, an input face mask 1101 is selected and features to be augmented, such as eyes 1110, are selected. An output face 1151 is generated, comprising an augmented face with the selected features, such as eyes 1160, added.

[0110] FIG. 12 shows a flow chart of a process 1200 for generating a first type of augmented data including facial augmentation. At step 1210, a 3D human model is received. At step 1220, a facial mask with features such as eyes 1221 and mouth 1222 is received. At step 1230, a partial convolution deep learning model is applied to combine the facial mask with the 3D human model. In some embodiments, the partial convolution deep learning model is implemented as described by Guilin Liu et al., “Image Inpainting for Irregular Holes Using Partial Convolutions,” available at https: / / arxiv.org / abs / 1804.07723 and https: / / github.com / NVIDIA / partialconv, the entire disclosures of which are incorporated by reference as if fully set forth herein. This approach utilizes partial convolution, where the convolutions are masked and renormalized as defined for only valid pixels. As part of the forward pass, a mechanism is provided to automatically generate an updated mask for the next layer. The resulting face is augmented onto a nude 3D body model as described herein.

[0111] Hair Extensions A first type of augmented data in another category, which can be used independently or partially dependent on the others, includes hair augmentation. Hair augmentation may include various hairstyles, hair colors, hair textures, and hair partially covering the face or body. In some embodiments, hair augmentation may be implemented by an automatic hair algorithm. In some embodiments, the automatic hair algorithm is realized as described by Menglei Chai et al., "AutoHair: Fully Automatic Hair Modeling from A Single Image," available at http: / / www.kunzhou.net / 2016 / autohair.pdf, the entire disclosure of which is incorporated by reference as if fully set forth herein. Initially, given a single image as input, the AutoHair pipeline generates a hair segmentation along with an instruction map. This information is then combined with a pre-computed 3D hair model exemplar to generate a final strand-level hair model that may be augmented to a nude 3D body model as described herein.

[0112] Light Expansion Light augmentation may include simulating various lighting conditions in which the 3D object is imaged or photographed. For example, the angle of the light source will affect shadows, the color of the light source will affect the color of the subject and surrounding objects, and low lighting conditions will affect the accuracy of determining the edges of physical features. In order to generate additional realistic datasets for training the deep learning network, all of these lighting conditions may be cycled through to generate a first type of augmented data in a manner similar to the process used above to augment skin color using a skin shader.

[0113] Augmented Virtual Clothing Other first type augmented data also include, but are not limited to, various virtual clothes that illustrate various clothes that a real person may be wearing when the photograph is taken, as described above. Figures 6A, 6B, 7A, 7B, 9A, and 9B show various virtual clothes models that are added to the sample 3D base mesh body model.

[0114] Other categories of first type augmented data are possible, either independently or partially dependent on each other. Other first type augmented data are within the scope of the present invention, and the present invention is not limited to the exemplary first type augmented data shown and described herein.

[0115] Example second type of extension data Background Extension As described above with reference to FIG. 2, in some embodiments, the second type of augmented data may be, for example, multiple (Q) background images. The background images represent a variety of possible backgrounds that may be observed by a subject in a real environment. The background images should represent other background conditions that are typical for subjects taking on different lighting, shadows, localization, and spatial features such as body measurements. The variety of backgrounds should be selected to optimize the training of the deep learning network, especially in a segmented deep learning network as described above.

[0116] Noise Expansion In other examples, the second type of augmented data may be, for example, white noise or other types of random noise. This may be useful to render the deep learning network more robust to imperfect image capture and other environmental conditions. The noise may be generated by a random number generator, for example according to a selected probability distribution, such as a Gaussian or uniform distribution. In other examples, the noise may be generated from measurements of noise in a real-life environment. For example, such noise may result from dust on a camera lens, smoke or fog in the environment when the picture is taken, or external light (e.g., sunlight reflected directly into the camera lens).

[0117] Perspective Distortion Enhancement In other embodiments, the second type of augmentation data may be, for example, perspective distortion data. When a 2D image of a 3D object or scene is taken from a fixed point, there is a warping or transformation of the object or scene that is significantly different from what the object would look like if the 2D image was taken from an infinite distance (i.e., a 2D projection of the 3D object) due to the relative scaling of near and far features. The warping or transformation involves distorting the relative size of features based on changes in distance and angle from a fixed point. The perspective distortion data includes information that changes the 2D projected image to what it would be if the image was taken from a particular point. Thus, the DLN is trained to compensate for the warping and transformation that occurs in real-life photographs by using the perspective distortion data to generate multiple 2D images that simulate photographs taken from various focal points.

[0118] Other second type extension data are within the scope of the present invention, and the present invention is not limited to the exemplary second type extension data shown and described herein.

[0119] Deep Learning Networks and Machine Learning for Body Measurements Once the deep learning network has been trained using the synthetic training data as described above, it can be used for accurate measurement determination from 2D photographs taken in natural environments. For example, in one embodiment, the trained deep learning network may be used for highly accurate body part measurements of 2D photographs taken of human subjects under typical conditions. The process of utilizing the trained deep learning network for accurate measurement determination is next described.

[0120] 13 illustrates an example architecture 1300 of a deep learning network for body measurement determination (using separate segmentation DLN, annotation DLN, and sizing ML modules) according to one embodiment of the present invention. In one embodiment, frontal and side images are received from a user at 1302.

[0121] The received front and side images are first pre-processed at 1304. In some embodiments, pre-processing of one or more images of the user, such as perspective correction, may be performed on the front and side photos, if necessary. For example, the system may use OpenCV, an open source machine vision library, and utilize head features in the front and side photos as well as the user's height as a reference for perspective correction. In this way, embodiments of the present disclosure may avoid determining inaccurate measurements unless there are abnormalities in the ratio of body lengths, such as torso length and leg length. Optionally, a perspective side photo showing where the camera is positioned relative to the person being photographed may result in even more accurate perspective correction by allowing the system to calculate the distance between the camera and the user. In some embodiments, the system may instead use gyroscope data provided by the user device (or a peripheral device connected to the user device, such as an attached computing device) to detect the perspective angle of the photo and perform perspective correction based on the perspective angle of the photo.

[0122] Various computer vision techniques may be utilized to further pre-process the image or images. Examples of additional pre-processing steps may include, but are not limited to, contrast correction, lighting correction, and other image processing techniques to improve the quality of the image or images before further processing.

[0123] After pre-processing, the pre-processed image is sent to a segmentation DLN at 1306 to generate a segmentation map that segments the body part from the background. The segmentation map is aggregated with the rest of the data at 1314. In parallel with the segmentation, in one embodiment, the pre-processed image is also sent to an annotation DLN at 1308 to generate annotation measurement lines. The annotation map is aggregated with the rest of the data at 1314. The annotation map, in one embodiment, is provided to a sizing machine learning (ML) module 1310 to generate a body feature measurement for each body feature that has been segmented and annotated by measuring each annotation line. The sizing results are aggregated with the rest of the data at 1314. The sizing results are output at 1312 to one or more external systems for various uses as described herein. Finally, the aggregated and structured data, (1) pre-processed front and side images, (2) segmentation map, (3) annotation map, and (4) sizing results, all collected at 1314, are stored in a database at 1316 for further DLN training.

[0124] FIG. 14 shows a diagram of an example flow 1400 for a body measurement determination operation, according to an example embodiment of the present disclosure. In some embodiments of the present invention, computer vision techniques and deep learning are applied to one front view photograph and one side view photograph of the user, in addition to the user's height and possibly other user parameters such as weight, gender, age, etc., to generate overall body measurements using one or more deep learning networks that are trained on the segmented and annotated training data, as generated according to the process of FIGS. 2-5. As more data is collected by the system, the accuracy of the body measurements is automatically improved. Optionally, and according to some other embodiments, strategies of perspective correction, human background subtraction, skeleton detection, and 3D model matching are used to utilize computer vision techniques to improve low-confidence body measurements from deep learning approaches.

[0125] The overall process 1400 begins at step 1402, where normalization data (one or more user parameters), such as the user's height, is obtained, generated, and / or measured to perform the normalization or conversion. In another embodiment, weight may also be used in conjunction with height. Both user parameters may be determined automatically (e.g., using computer vision algorithms or mined from one or more databases) or from the user (e.g., user input). In one embodiment, a body mass index (BMI) may be calculated from these user parameters. The BMI may be used to calibrate the body measurement extraction that uses both weight and height. The additional user parameters may include at least one of height, weight, sex, age, race, country of origin, activity, and / or other demographic information associated with the user, among others. The user's height is used to normalize, i.e., convert, the front view and / or side view photographs and provide a human reference for the photographs. Other user parameters, such as weight, BMI index, age, and sex, are used as additional inputs to the system to optimize the body sizing measurements. In one embodiment, other user parameters may also be obtained automatically from the user device, from one or more third party data sources, or from a server.

[0126] At step 1404, one or more user photos may be received, for example, at least one front view and / or side view photo of a given user may be received. In another embodiment, the photos may be obtained from a user device (e.g., a mobile phone, laptop, tablet, etc.). In another embodiment, the photos may be obtained from a database (e.g., a social media database). In another embodiment, the user photos include a photo showing a front view and a photo showing a side view of the user's entire body. In some embodiments, only one photo is utilized, such as a front view, and one photo is sufficient to perform accurate body measurement extraction. In still other embodiments, three or more photos are utilized, and in some embodiments, includes a front view photo, a side view photo, and a photo taken at a 45 degree angle. Other combinations of user photos are within the scope of the present invention, as will be recognized by those skilled in the art. In some embodiments, a user video may be received, such as a front view, a 90 degree view, a 180 degree view, or even a 360 degree view of the user. From the user video, one or more still images or photographic frames are extracted from the video, such as from the user's front view, side view, and / or 45 degree view, and used in the following process. Steps 1402 and 1404 may be performed in any order in various embodiments of the present invention, or the two steps may be realized in parallel.

[0127] In one embodiment, the system may automatically calculate body measurements (e.g., using one or more AI algorithms) using the photograph and normalized data, as further described below in connection with the following steps. In one embodiment, the user may also indicate whether the user is wearing tight, normal, or loose clothing for more accurate results.

[0128] In one embodiment, the images may be taken at a specified distance (e.g., about 10 feet from the user device camera). In one embodiment, an image of the user may be taken having a particular posture (e.g., arms in a given position, legs shoulder-width apart, spine upright, "A posture", etc.). In another embodiment, multiple images (e.g., front and side view photos) of a given position may be taken and an average image may be determined for each position. This may be done to increase accuracy. In another embodiment, the user may be positioned in front of a particular type of background (e.g., having a neutral color, or a predetermined background image). In some embodiments, the user may be positioned in front of any type of background. In one embodiment, the front and side view photos may be taken under similar lighting conditions (e.g., a given brightness, shadows, etc.). In another embodiment, the front and side view photos may include an image of the user wearing normally fitting clothing (e.g., not too loose or too tight). Alternatively, or in addition, the front and side view photographs may include images of the user partially clothed (e.g., shirtless) or having a different type of fit (e.g., tight, loose, etc.), depending on the needs of the AI-based algorithms and associated processes.

[0129] In step 1406, physical features such as human body parts (e.g., neck, arms, legs, etc.) may be extracted from the image using a first deep learning network (DLN) known as a segmentation DLN. In one embodiment, "deep learning" may refer to a class of machine learning algorithms that use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation, modeled after neural networks. In one embodiment, successive layers may use the output from the previous layer as input. In one embodiment, the "deep" in "deep learning" may refer to the number of layers that transform the data. An example of physical feature extraction, i.e., body segmentation, is described and illustrated with reference to FIGS. 6A, 6B, 7A, and 7B.

[0130] Prior to performing this segmentation step on data from real users, the system may first be trained, for example, on sample photos of humans posing in different clothing in different environments, or by utilizing the training data generation process described in connection with FIGS. 2-5. In some embodiments, any suitable deep learning architecture may be used, such as deep neural networks, deep belief networks, and / or recurrent neural networks. In other embodiments, the deep learning algorithm may learn in a supervised (e.g., classification) and / or unsupervised (e.g., pattern analysis) manner. Additionally, the deep learning algorithm may learn multiple levels of representations that correspond to different levels of abstraction of the information encoded in the image (e.g., body, body parts, etc.). In another embodiment, the image (e.g., front and side photos) may be represented as a matrix of pixels. For example, in one embodiment, a first representation layer of the deep learning network may abstract pixels and encode edges, a second layer may configure and encode the arrangement of edges, a third layer may encode noses and eyes, a fourth layer may recognize that the image includes a face, etc.

[0131] In one embodiment, the segmentation DLN algorithm may be trained using segmentation training data, as described with respect to FIG. 8B above. In some embodiments, the segmentation training data may include thousands of sample humans with manually segmented physical features or generated by utilizing the training data generation process described with respect to FIGS. 2-5. In some embodiments, the training data includes medical data, for example from CAT scans, MRI scans, etc. In some embodiments, the training data includes data from a previous tailor or 3D body measurements, including 3D body scans and "ground truth" data from a 3D body scanner. In some embodiments, the 3D body scans may be used to extract approximate front and / or side view photographs, if front and side view photographs are not explicitly available, as described with respect to FIGS. 2-5. In some embodiments, the ground truth data includes data measured by a human tailor, while in other embodiments, the ground truth data includes automatically extracted 1D body size measurements from a 3D body scan.

[0132] In step 1407, the annotation lines of each body part extracted in step 1406 may be rendered using one or more additional deep learning networks (DLNs), such as annotation DLNs. In one embodiment, there is a separate physical feature annotation DLN for each body part. In another embodiment, there is one physical feature annotation DLN for the whole body. An example of a physical feature annotation is described and shown with reference to Figures 7A-7B.

[0133] In one embodiment, the system may generate and extract physical feature measurements by using an AI-based algorithm such as annotation DLN, for example, by first drawing annotation lines from signals obtained from the physical features. Each annotation line may be different and may be drawn differently for each physical feature. For example, for upper arm width and circumference, the system may draw a line perpendicular to the skeletal line at the upper arm location, and for the chest, the system may instead connect two chest points. From the annotations of each physical feature, physical feature measurements may then be obtained by normalizing against the user's height received at 1402, as further described below.

[0134] Before performing this annotation step on data from real users, the system may first be trained, for example, on sample photographs of humans in different clothing and poses in different environments, or by utilizing the training data generation process described in connection with Figures 2-5. The segmentation and annotation DLN is described in further detail below in connection with Figure 15.

[0135] In step 1408, physical feature measurements may be estimated for each body part having annotation lines drawn in step 1407 using one or more machine learning algorithms, such as a sizing machine learning (ML) algorithm. In one embodiment, the sizing ML algorithm includes a random forest machine learning module. In one embodiment, there is a separate sizing ML module for each body part. In some embodiments, there is one sizing ML module for the whole body. In one embodiment, the system may use the height received in step 1402 as an input to determine the size of the physical features and normalize the sizing estimates. To do this, the annotation DLN draws a "full body" annotation line indicating the location of the subject's height, in one embodiment, a point representing the bottom of the subject's feet and another point representing the top of the subject's head. This "full body" annotation line is used to normalize the other annotation lines by the subject's known height provided in step 1402. In other words, the subject's height in the image is detected and used along with the known actual height to normalize all annotation line measurements. This process may be considered "height-based normalization," using the subject's known height as a standard measurement for normalization. Other normalization factors are possible and within the scope of the present invention, including but not limited to a book of known height, a standard sized piece of paper of known size, a standard issued credit card of known size, etc.

[0136] In another embodiment, additional user demographic data, such as, but not limited to, weight, BMI index, gender, age, and / or other demographic information associated with the user received in step 1402, is used as input to a sizing ML algorithm (such as a random forest), which is described in further detail in connection with FIG. 15.

[0137] The system may also use other algorithms, measures, and median values ​​for each physical feature measurement. The annotation DLN and sizing ML may be implemented as one sizing DLN that annotates and performs measurements for each physical feature, or may be implemented as two separate modules: an annotation DLN that annotates each physical feature, and a separate sizing ML module that performs measurements on the annotated physical features. Similarly, various alternative architectures are possible for implementing the segmentation DLN of step 1406, the annotation DLN of step 1407, and the sizing ML module of step 1408. For example, FIG. 13 corresponds to one possible architecture in which the segmentation DLN, annotation DLN, and sizing ML modules are separate modules. In contrast, an alternative architecture (not shown in FIG. 13) is where the segmentation DLN and annotation DLN are combined into a single annotation DLN (effectively performing both segmentation and annotation), followed by the sizing ML module. Finally, yet another alternative architecture (not shown in FIG. 13) is where the segmentation DLN, annotation DLN, and sizing ML modules are all combined into a single sizing DLN that effectively performs all the functions of segmentation, annotation, and sizing.

[0138] At step 1410, a confidence level for each physical characteristic measurement may be determined, obtained, or received from the sizing ML module from step 1408. In addition to outputting a predicted physical measurement for each physical characteristic, the sizing ML module also outputs a confidence level for each predicted physical characteristic measurement, which is then used to determine whether to utilize any other measures to improve the output, as described below. In another embodiment, the confidence level may be based on a confidence interval. In particular, a confidence interval may refer to a type of interval estimate calculated from statistics of observed data (e.g., frontal and lateral photographs encoding image data) that may contain the true value of an unknown population parameter (e.g., a body part measurement). An interval may have an associated confidence level that may quantify the confidence level that the parameter is within the interval. More precisely, the confidence level represents the frequency (i.e., proportion) of possible confidence intervals that contain the true value of the unknown population parameter. In other words, if a confidence interval is constructed using a given confidence level of a finite number of independent sample statistics, the proportion of those intervals that contain the true value of the parameter will be equal to the confidence level. In another embodiment, a confidence level may be assigned prior to inspecting the data (e.g., the image and measurements extracted therefrom). In one embodiment, a 95% confidence level is used. However, other confidence levels may be used, e.g., 90%, 99%, 99.5%, etc.

[0139] In various embodiments, the confidence interval and corresponding confidence level may be determined based on a determination of validity and / or optimality. In another embodiment, validity may refer to the confidence level of the confidence interval being accurate or well-approximated. In one embodiment, optimality may refer to the rules for constructing the confidence interval should utilize as much of the information in the dataset (images and extracted features and measurements) as possible.

[0140] At step 1412, it may be determined whether the confidence level is higher than a predetermined value. If the confidence level is determined to be higher than the predetermined value, the process may proceed to step 1414, where high confidence physical feature measurements may be output. If the confidence level is determined to be lower than the predetermined value, the process may proceed to step 1416 or step 1418. Steps 1416 and 1418 are illustrative of one or more optional fallback algorithms that predict or project estimated physical feature measurements for those physical features for which the deep learning approach has low confidence. Both the high confidence physical feature measurements from the deep learning approach (illustrated by dashed lines) and the projected physical feature measurements from the alternative fallback algorithms for the low confidence physical feature measurements are later combined into a complete set of high confidence physical feature measurements, as described below. As shown, in another embodiment, the confidence level may be specified prior to inspecting the data (e.g., the image and measurements extracted therefrom).

[0141] In particular, in steps 1416 and 1418, any other model (e.g., AI-based or computer vision-based model) may be applied. In step 1416, and according to one optional embodiment, a 3D human model matching algorithm may be applied. For example, the system may first utilize OpenCV and / or deep learning techniques to extract a human body from the background. The extracted human body is then matched to one or more known 3D human models to obtain physical feature measurements. Using this technique and a database of existing 3D body scans, e.g., a database of thousands of 3D body scans, the system may match the closest detected body to the points of the 3D body scan. Using the best matching 3D model, the system may then extract physical feature measurements from the 3D model.

[0142] Alternatively and / or in addition, other models may be applied at step 1418, such as a skeleton / joint position model. In one embodiment, skeleton / joint detection may be performed using OpenPose, an open source algorithm for pose detection (discussed further below). Using this technique to obtain skeleton and joint positions, the system may then use additional deep learning networks (DLNs) as needed to draw lines between appropriate points showing the intermediate positions of bones depicted on the user's photograph to indicate various major skeletal structures and to indicate where various body parts such as shoulders, neck, and arms are located. From this information, physical feature measurements may be obtained from the appropriate lines. For example, a line connecting the shoulders and wrists may be used to determine arm length.

[0143] In one embodiment, the 3D model algorithm and the skeleton / joint position model are combined as follows (although this is not explicitly shown in FIG. 14): Using a database of existing 3D body scans, e.g., a database of thousands of 3D body scans, the system may match the closest skeleton detections to the skeleton points of the 3D body scans to show points and lines indicating the positions of bones indicating various major skeletal structures, indicating where various body parts are located, such as shoulders, neck, and arms. Once the best matching 3D model has been matched, the system may extract physical feature measurements from the 3D model.

[0144] In either or both instances, a high confidence physical feature measurement may be projected (e.g., estimated) at step 1420, or at step 1422, or both. In particular, the estimation of the high confidence physical feature measurement may be performed using a process that is different from the first low confidence deep learning process (e.g., the one shown and described in connection with step 108 above).

[0145] One advantageous feature of this approach is that the high confidence physical feature measurements from step 1414 (illustrated as dashed lines) may be used as inputs to aid in the calibration of other models, e.g., the 3D human model algorithm in step 1416 and the skeleton / joint position model in step 1418. That is, the high confidence physical feature measurements from the deep learning approach obtained in step 1408 may be used to aid the other models, e.g., the 3D human model 116 and / or the skeleton / joint position model 1418. The other models (1416 and / or 1418) may then be used to obtain projected high confidence physical feature measurements for those physical feature measurements determined in step 1412 to have a confidence lower than a predetermined value. The projected high confidence physical feature measurements may later replace or supplement the low confidence physical feature measurements from the deep learning approach.

[0146] Further, in step 1424, the high confidence physical feature measurements determined in step 1420 and / or step 1422 may be used to determine high confidence physical feature measurements. In that way, using various models, both 3D human models and skeleton / joint position models, the accuracy of the physical feature measurements obtained in step 1414 may be further improved. Thus, the high confidence physical feature measurements may be aggregated and the high confidence physical feature measurements from step 1414 (e.g., deep learning approaches) may be combined with the projected high confidence physical feature measurements from steps 1420 and 1422 (e.g., other models).

[0147] In step 1426, the reliable physical characteristic measurements are aggregated into complete body measurements of the entire human body, which are then output for use. In particular, the body measurements may be output to a user device and / or corresponding server, for example, associated with a company that manufactures clothing based on the measurements. In one embodiment, the output may be in the form of a text message, an email, a text description on a mobile application or website, combinations thereof, and the like. The complete body measurements may then be used for any purpose, including, but not limited to, the generation of custom clothing. One skilled in the art will recognize that the output of the complete body measurements may be utilized for any purpose for which accurate and simple body measurements are useful, such as, but not limited to, fitness, health, shopping, and the like.

[0148] FIG. 15 shows another example flow diagram 1500 for determining body measurements using a deep learning network (DLN) and machine learning according to another embodiment of the present invention. In step 1551, input data 1552 comprising a frontal photograph, a side photograph, and user parameters (height, weight, age, gender, etc.) are received. In step 1553, one or more image processing steps are applied. First, any image pre-processing steps (perspective correction, human cropping, resizing, etc.) may be performed. Next, a deep learning network (DLN) 1554 is applied to the image to segment and annotate the body features. Then, a sizing machine learning module (ML) 1556 is applied to the annotated body features to determine body size measurements from the annotation lines and one or more user parameters. Finally, in step 1555, body size measurements (e.g., 16 standard body part sizes) are output, exemplarily shown as output data 1558. The output 1558 may include sizing results (a set of standard body size measurements such as neck, shoulders, sleeves, height, outseam, inseam, etc.) and may also include front and side photos annotated with annotation lines.

[0149] As shown in FIG. 15 and according to one embodiment, body part segmentation and annotation is performed using a deep learning network (DLN) 1554. In one embodiment, body part segmentation and annotation is performed using a convolutional neural network (CNN) in combination with a pyramid scene parsing network (PSPNet) for improved global and local context information. In PSPNet, the process may utilize global and local context information from different sized regions aggregated through a pyramid pooling module. As shown in FIG. 15, the input image is first passed through a convolutional neural network (CNN) to obtain a feature map that classifies or segments each pixel into a given body part and / or annotation line. Next, global and local context information is extracted from the feature map using a pyramid pooling module that aggregates information from the image at different size scales. Finally, the data is passed through a final convolutional layer to classify each pixel into a body part segment and / or annotation line.

[0150] In one embodiment, the PSPNet algorithm may be an implementation as described in "Pyramid Scene Parsing Network" by Hengshuang Zhao et al., CVPR 2017, December 4, 2016, available at arXiv:1612.01105. PSPNet is just one exemplary deep learning network algorithm within the scope of the present invention, and the present invention is not limited to the use of PSPNet. Other deep learning algorithms are also within the scope of the present invention. For example, in one embodiment of the present invention, a convolutional neural network (CNN) is utilized to extract body segments (segmentation), and a separate CNN is used to annotate each body segment (annotation).

[0151] Various exemplary machine learning algorithms 1556 for determining body measurements can predict measurements from one or more feature values ​​obtained from a deep learning network (DLN). In one embodiment, the sizing of the body part is determined using one exemplary machine learning algorithm, the Random Forest algorithm 1556. The Random Forest algorithm uses multiple decision tree predictors such that each decision tree depends on the values ​​of a random subset of the training data, thereby minimizing the possibility of overfitting to the training data set. In one embodiment, the Random Forest algorithm is an implementation as described in Leo Braiman, "Random Forests," Machine Learning, 45, 5-32, 2001, Kluwer Academic Publishers, Netherlands, available at doi.org / 10.1023 / A:1010933404324. Random Forest is just one exemplary machine learning algorithm within the scope of the present invention, and the present invention is not limited to the use of Random Forests. Other machine learning algorithms are within the scope of the present invention, including but not limited to nearest neighbors, decision trees, support vector machines (SVMs), AdaBoost, Bayesian networks, various neural networks including deep learning networks, evolutionary algorithms, etc. The inputs to the machine learning algorithms are feature values ​​(x) including body part circumference, height, and other user parameters obtained from a deep learning network, as shown in Figure 15. The output of the machine learning algorithm is a predicted value for the sizing measurement (y).

[0152] As shown, embodiments of the devices and systems (as well as their various components) described herein may employ artificial intelligence (AI) to facilitate automation of one or more of the mechanisms described herein (e.g., providing body extraction, body segmentation, measurement extraction, etc.). The components may employ various AI-based schemes to implement the various embodiments / examples disclosed herein. To provide or assist in the numerous decisions described herein (e.g., determine, ascertain, infer, calculate, predict, foresee, estimate, derive, anticipate, detect, compute), the components described herein may examine all or a subset of the data to which they are granted access, and may provide or determine reasons for the state of the system, environment, etc. from a set of observations captured via events and / or data. The decisions may be used to identify a particular context or action, or may generate a probability distribution for the state, for example. The decisions may be probabilistic, i.e., the calculation of a probability distribution for a state of interest is based on a consideration of data and events. The decisions may also refer to techniques used to compose top events from a set of events and / or data.

[0153] Such determination can result in the construction of new events or actions from a set of observed events and / or stored event data, whether the events are closely correlated in time, and whether the events and data are from one or multiple event and data sources. The components disclosed herein can employ a variety of classification (explicitly trained (e.g., via training data) and implicitly trained (e.g., via observed behavior, preferences, historical information, received exogenous information, etc.)) schemes and / or systems (e.g., support vector machines, neural networks, expert systems, Bayesian belief networks, fuzzy logic, data fusion engines, etc.) in connection with performing automated and / or determined actions associated with the claimed subject matter. Thus, the classification schemes and / or systems can be used to automatically learn and perform numerous functions, actions, and / or decisions.

[0154] A classifier may map an input attribute vector z=(z1,z2,z3,z4,...,zn) to the confidence of the class to which the input belongs by f(z)=confidence(class). Such classification may use probabilistic and / or statistical-based analysis (e.g., taking into account the usefulness and cost of the analysis) to determine the action to be automatically performed. Another example of a classifier that can be used is a support vector machine (SVM). SVM operates by finding a hypersurface in the space of possible inputs that attempts to split triggering criteria from non-triggering events. Intuitively, this makes the classification suitable for testing data that is near but not identical to the training data. Other directed and undirected model classification strategies can be used, for example, naive Bayes, Bayesian networks, decision trees, neural networks, fuzzy logic models, and / or probabilistic classification models that provide different independence patterns. Classification as used herein also encompasses statistical regression that is utilized to develop models of priority.

[0155] FIG. 16 shows another detailed flow diagram 1600 for body measurement determination using a deep learning network (DLN) and machine learning according to another embodiment of the present invention. Inputs to the body measurement process include a frontal photo 1661, a side photo 1662, height 1663, and other user parameters (weight, age, sex, etc.) 1664. The frontal photo 1661 is pre-processed in step 1665, and the side photo 1662 is pre-processed in step 1666. Examples of pre-processing steps, such as perspective correction, human cropping, and image resizing, are described above. In step 1667, the pre-processed frontal photo is used as input to DLN1 to extract annotation lines for the frontal photo 1661. In step 1668, the pre-processed side photo is used as input to DLN2 to similarly extract annotation lines for the side photo 1662. Annotation lines for each body part from the front view 1669 are output from DLN1, and annotation lines for each body part from the side view 1670 are output from DLN2. At step 1671, the two sets of annotation lines from the front photo 1661 and the side photo 1662 are utilized along with the height normalization criterion 1675 received from the height input 1663 to calculate the circumference of each body part. At step 1672, the circumference of each body part along with the height and other user parameters 1676 received from the inputs 1663 and 1664 are utilized in a machine learning algorithm such as random forest to calculate one or more body size measurements. At step 1673, the body size measurement results (length of each standard measurement) are output. Finally, the body measurement process ends at step 1674.

[0156] Hardware, Software, and Cloud Implementations of the Invention As will be discussed, data described throughout this disclosure (e.g., photographs, text descriptions, etc.) may include data stored in a cloud computing platform or in a database hosted therein. Although this disclosure includes a detailed description of cloud computing below, it should be understood that implementation of the teachings referred to herein is not limited to a cloud computing environment. Instead, embodiments of the present invention may be implemented in conjunction with any other type of computing environment now known or later developed.

[0157] Cloud computing can refer to a model of service delivery for convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort and interaction with the provider of the service. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0158] The characteristics may include one or more of the following: On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as needed without the need for a human to interact with the provider of the service. Broad Network Access: Capacity is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, and PDAs). Resource Pooling: Provider computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. There is location independence in the sense that consumers generally have no control or knowledge over the exact location of the resources provided, but the location can be specified at a higher level of abstraction (e.g., country, state, or data center). Rapid Scalability: Capacity can be rapidly elastically provisioned, potentially automatically, rapidly scaled out, rapidly released, and rapidly scaled in. To the consumer, the capacity available to provision often appears infinite, and can be purchased at any time in any quantity. Metered Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported to provide transparency to both providers and consumers of the services utilized.

[0159] In another embodiment, the service model may include one or more of the following: Software as a Service (SaaS): The capability offered to the consumer is to use a provider's application running on a cloud infrastructure. The application is accessible from a variety of client devices through a thin-client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0160] Platform as a Service (PaaS): The ability offered to consumers is to deploy consumer-created or acquired applications, written using provider-supported programming languages ​​and tools, on a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the configuration of the environment in which the applications are hosted.

[0161] Infrastructure as a Service (IaaS): The capability offered to a consumer is to provision processing, storage, network, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, deployed applications, and possibly limited control to select networking components (e.g., host firewalls).

[0162] Deployment models may include one or more of the following: Private Cloud: The cloud infrastructure is operated solely for the institution. It can be managed by the institution or a third party and can exist on-premise or off-premise.

[0163] Community Cloud: Cloud infrastructure is shared by multiple agencies and serves a particular community of shared interests (e.g., missions, security requirements, policies, and compliance considerations). It can be managed by the agency or a third party and can exist on-premise or off-premise.

[0164] Public Cloud: The cloud infrastructure is made available to the general public or large business groups and is owned by an institution that sells cloud services.

[0165] Hybrid Cloud: A cloud infrastructure is a blend of two or more clouds (private, community, or public) that remain their own entities but are linked together by standardized or proprietary technologies that allow data and application portability (e.g., cloud bursting for load balancing between clouds).

[0166] Cloud computing environments are service-oriented and focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0167] A cloud computing environment may include one or more cloud computing nodes, by which local computing devices used by cloud consumers, such as, for example, personal digital assistants (PDAs), cellular phones, desktop computers, laptop computers, and / or automobile computer systems, can communicate. The nodes can communicate with each other. They can be physical or virtual groups in one or more networks, such as private, community, public, or hybrid clouds as described above, or combinations thereof. This allows the cloud computing environment to provide infrastructure, platform, and / or software as a service without the cloud consumer having to maintain resources on the local computing device. The types of computing devices are intended to be merely exemplary, and it is understood that the computing nodes and cloud computing environment can communicate with any type of computerized device through any type of network and / or network addressable connection (e.g., using a web browser).

[0168] The present invention may be implemented using server-based hardware and software. FIG. 17 shows an example hardware architecture diagram 1700 of a server implementing one embodiment of the present invention. To avoid obscuring the present invention, many components of the system, such as network interfaces, are not shown. However, one skilled in the art would recognize that the system necessarily includes these components. The user device is hardware that includes at least one processor 1740 coupled to memory 1750. Processor may represent one or more processors (e.g., microprocessors), and memory may represent a random access memory (RAM) device that comprises the hardware's primary storage, as well as any secondary level of memory, such as cache memory, non-volatile or backup memory (e.g., programmable or flash memory), read-only memory, and the like. In addition, memory may be considered to include memory storage physically located anywhere in the hardware, such as any cache memory in the processor, as well as any storage capacity used as virtual memory, such as stored in a mass storage device.

[0169] The user device hardware also typically receives a number of inputs 1710 and outputs 1720 for communicating information externally. In terms of interfacing with a user, the hardware may include one or more user input devices (e.g., a keyboard, mouse, scanner, microphone, webcam, etc.) and a display (e.g., a liquid crystal display (LCD) panel). For additional storage, the hardware may also include one or more mass storage devices 1790, such as floppy or other removable disk drives, hard disk drives, direct access storage devices (DASD), optical drives (e.g., compact disk (CD) drives, digital versatile disk (DVD) drives, etc.), and / or tape drives, among others. Additionally, the hardware may include interfaces, one or more external SQL databases 1730, and one or more networks 1780 (e.g., a local area network (LAN), a wide area network (WAN), a wireless network, and / or the Internet, among others) so that it can communicate information with other computers coupled to the network. It should be appreciated that the hardware typically includes suitable analog and / or digital interfaces for communicating with each other.

[0170] The hardware operates under the control of an operating system 1770 and executes various computer software applications 1760, components, programs, code, libraries, objects, modules, etc., collectively indicated by reference numerals, to implement the methods, processes, and techniques described above.

[0171] The present invention may be implemented in a client-server environment. Figure 18 shows an example system architecture 1800 for implementing an embodiment of the present invention in a client-server environment. Client-side user devices 1810 may include smartphones 1812, laptops 1814, desktop PCs 1816, tablets 1818, or other devices. Such user devices 1810 access the services of a system server 1830 through some network connection 1820, such as the Internet.

[0172] In some embodiments of the present invention, the entire system can be realized and presented to end users and operators over the Internet in a so-called cloud implementation. No local installation of software or hardware is required, and end users and operators can directly access the system of the present invention over the Internet using a web browser or similar software on a client, which can be a desktop, laptop, mobile device, etc. This eliminates the need for custom software installation on the client side, increases the flexibility of service delivery (software as a service), and increases user satisfaction and ease of use. Various business models, revenue models, and delivery mechanisms for the present invention are contemplated and are all considered to be within the scope of the present invention.

[0173] In general, the methods performed to implement the embodiments of the present invention may be implemented as part of an operating system or dedicated application, a component, program, object, module, or sequence of instructions, referred to as a "computer program" or "computer code." A computer program generally comprises one or more sets of instructions in various memories and storage devices of a computer at various times, which, when read and executed by one or more processors of the computer, cause the computer to perform the operations necessary to execute the elements involved in the various aspects of the present invention. Furthermore, while the present invention has been described in the context of fully functioning computers and computer systems, those skilled in the art will recognize that various embodiments of the present invention can be distributed as program products in a variety of forms, and that the present invention applies equally regardless of the particular type of machine or computer readable medium used to actually effect the distribution. Examples of computer readable media include, but are not limited to, recordable type media, such as volatile and non-volatile memory devices, floppy and other removable disks, hard disk drives, optical disks (e.g., compact disk read only memory (CD-ROM), digital versatile disk (DVD), etc.), and digital and analog communication media.

[0174] Illustrative Uses of the Invention FIG. 19 is an exemplary diagram 1900 of a use case of the present invention in which a single camera of a mobile device 1902 is used to capture anthropometric measurements, showing a front view of a human wearing normal clothing standing in front of a normal background. In the exemplary diagram 1900, the mobile device 1902 has a display 1904 that displays a human form. In one embodiment, the torso of the human form is identified, for example, by a dotted line 1906, and the body segments are identified, for example, by a solid line 1908. The top of the head is identified, for example, by a first round dot 1910, and the bottom of the feet is identified, for example, by a second round dot 1912. The mobile device shown in FIG. 19 comprises at least one camera, a processor, a non-transitory storage medium, and a communication link to a server. In one embodiment, one or more pictures of the user's body are transmitted to the server, which performs the operations described herein. In one embodiment, the one or more pictures of the user's body are analyzed locally by the processor of the mobile device 1902. The performed operations return one or more body measurements, which may be stored on the server and presented to the user. Additionally, the body measurements may then be utilized for many purposes, including, but not limited to, presenting the user with sales on one or more custom clothing, custom eyeglasses, custom gloves, custom body suits, custom PPE, custom hats, custom diets, custom exercise, gyms, and training routines, etc. Without loss of generality, the body measurements may be output, transmitted, and / or utilized for any purpose for which body measurements are useful.

[0175] The invention has been successfully implemented, resulting in body measurements accurate to within 1 cm relative to a human tailor. The system can achieve accuracy comparable to a human tailor using only two photographs. The system does not require the use of any dedicated hardware sensors, does not require the user to stand in front of any special background, does not require special lighting, and can use photographs taken at any distance and with the user wearing any type of clothing. The result is a body measurement system that works on any mobile device, so that anyone can simply take a photo of themselves and take advantage of the automatic full body measurement extraction.

[0176] Those skilled in the art will appreciate that the use examples, structures, schematic diagrams, and flow diagrams may be implemented in other sequences or combinations while still retaining the inventive concept of the present invention without departing from the broader scope of the present invention. Every embodiment is unique, and methods / steps may be shortened or lengthened, overlapped with other operations, or continued after a postponement, delay, or time gap, so that every user is adapted to practice the method of the present invention.

[0177] Although the present invention has been described with reference to certain exemplary embodiments, it will be apparent that these embodiments can be modified and altered in various ways without departing from the broader scope of the present invention. The specification and drawings should therefore be interpreted in an illustrative and not a restrictive sense. It will also be apparent to those skilled in the art that the above-described embodiments are specific illustrations of a single broader invention that may have a broader scope than any of the individual descriptions taught. Many modifications may be made to the description without departing from the scope of the present invention.

Claims

1. 1. A computer-implemented method for generating a training dataset for training a deep learning network (DLN) for extracting spatial features from two-dimensional (2D) images of a three-dimensional (3D) object, comprising: The computer-implemented method is executable by a hardware processor, the method comprising: receiving a 3D model of the 3D object; extracting spatial features from the 3D model, the spatial features being one-dimensional (1D) body measurements; generating a first type of augmented data for the 3D model; augmenting the 3D model with the first type of augmentation data to generate an augmented 3D model, the first type of augmentation data being selected from the group consisting of skin color, facial contours, hairstyle, virtual clothing, and lighting conditions; generating at least one 2D image from the augmented 3D model by projecting the augmented 3D model onto at least one surface; and generating a training dataset for training the deep learning network (DLN) for spatial feature extraction by aggregating the spatial features and the at least one 2D image.

2. receiving a second type of augmented data for the at least one 2D image; and augmenting the at least one 2D image with the second type of augmentation data to generate a plurality of augmented 2D images, each of which includes the projection of the augmented 3D model and an instance of the second type of augmentation data; generating the training dataset for training the deep learning network (DLN) for spatial feature extraction by aggregating the spatial features and the plurality of augmented 2D images; The computer-implemented method of claim 1 , further comprising:

3. The computer-implemented method of claim 1 , wherein the 3D model is a 3D human body model.

4. The computer-implemented method of claim 2 , wherein the second type of augmented data is background images representing different backgrounds.

5. the 3D model is a 3D human body model, and the method further comprises: performing a segmentation on the augmented 2D image utilizing a plurality of data points from the 3D model to extract one or more features associated with the 3D model from the background; The computer-implemented method of claim 4 , further comprising:

6. the 3D model is a 3D human body model, and the method further comprises: performing line annotation on the augmented 2D image utilizing a plurality of data points from the 3D model to generate annotation lines on each spatial feature corresponding to the spatial feature measurements; The computer-implemented method of claim 4 , further comprising:

7. generating ground truth feature measurements from the spatial features utilizing a plurality of data points from the 3D model; The computer-implemented method of claim 4 , further comprising:

8. The computer-implemented method of claim 2 , wherein the second type of extended data is white noise.

9. The computer-implemented method of claim 2 , wherein the second type of enhanced data is perspective distortion data.

10. generating a second training data set from a second 3D model to train the deep learning network for spatial feature extraction; the second training data set comprises a second plurality of spatial features, a second plurality of augmented 3D models, and a second plurality of 2D images.

10. The computer-implemented method of claim 1.

11. Training the deep learning network for spatial feature extraction using the second training data set. The computer-implemented method of claim 10 further comprising:

12. 2. The computer-implemented method of claim 1 , wherein the at least one 2D image is selected from the group consisting of a front view image of the 3D model, a side view image of the 3D model, and an image projected from the 3D model at a 45 degree angle relative to the front view image.

13. the 3D model is a 3D body model, and the method further comprises: training a sizing machine learning module, the sizing machine learning module being trained on ground truth data including 1D body measurements extracted from the 3D body model; The computer-implemented method of claim 1 , further comprising:

14. 14. The computer-implemented method of claim 13, wherein the sizing machine learning module is trained on ground truth data comprising the 1D body measurements extracted from the 3D body model and one or more user parameters associated with the 3D body model.

15. 15. The computer-implemented method of claim 14, wherein the user parameters are selected from the group consisting of height, weight, gender, age, and demographic information associated with the user.

16. The computer-implemented method of claim 13 , wherein the sizing machine learning module comprises a random forest algorithm.

17. The computer-implemented method of claim 1 , wherein the 3D model includes at least one 3D body model of a nude user or a semi-nude user.

18. 2. The computer-implemented method of claim 1, wherein the deep learning network (DLN) comprises a convolutional neural network (CNN).

19. 20. The computer-implemented method of claim 18, wherein the deep learning network (DLN) further comprises a pyramid pooling module.

20. The computer-implemented method of claim 1 , wherein the 3D model is received from a 3D scanner.

21. 21. The computer-implemented method of claim 20, wherein the 3D model is received from a 3D body scanner of a human body.

22. training the deep learning network (DLN) for determining body measurements using the training dataset; The computer-implemented method of claim 1 , further comprising:

23. Providing the trained deep learning network (DLN) for measurement determination.

23. The computer-implemented method of claim 22, further comprising:

24. post-processing the training data set prior to providing the training data set to train the deep learning network (DLN); 23. The computer-implemented method of claim 22, further comprising:

25. 1. A computer program for generating a training data set for training a deep learning network to extract spatial features from two-dimensional (2D) images of a three-dimensional (3D) object, comprising: a non-transitory computer-readable storage medium having embodied program instructions executable by a processor, the program instructions comprising: receiving a 3D model of the 3D object; extracting spatial features from the 3D model, the spatial features being one-dimensional (1D) body measurements; generating a first type of augmentation data for the 3D model; augmenting the 3D model with the first type of augmentation data to generate an augmented 3D model, the first type of augmentation data being selected from the group consisting of skin color, facial contours, hairstyle, virtual clothing, and lighting conditions; generating at least one 2D image from the augmented 3D model by projecting the augmented 3D model onto at least one surface; A computer program product that generates a training data set for training the deep learning network (DLN) for spatial feature extraction by aggregating the spatial features and the at least one 2D image.

26. receiving a second type of augmented data for the at least one 2D image; augmenting the at least one 2D image with the second type of augmentation data to generate a plurality of augmented 2D images, each of which includes the projection of the augmented 3D model and an instance of the second type of augmentation data; generating the training dataset for training the deep learning network (DLN) for spatial feature extraction by aggregating the spatial features and the plurality of augmented 2D images; 26. The computer program product of claim 25, further comprising program instructions.

27. 26. The computer program product of claim 25, wherein the 3D model is a 3D human body model.

28. 1. A system for generating a training dataset for training a deep learning network to extract spatial features from two-dimensional (2D) images of a three-dimensional (3D) object, comprising: At least one processor; at least one non-transitory computer-readable storage medium having program instructions embodied therein; The program instructions are executable by the at least one processor, the at least one processor comprising: receiving a 3D model of the 3D object; extracting spatial features from the 3D model, the spatial features being one-dimensional (1D) body measurements; generating a first type of augmentation data for the 3D model; augmenting the 3D model with the first type of augmentation data to generate an augmented 3D model; generating at least one 2D image from the augmented 3D model by projecting the augmented 3D model onto at least one surface, wherein the first type of augmented data is selected from the group consisting of skin color, facial contours, hairstyle, virtual clothing, and lighting conditions; generating a training dataset for training the deep learning network (DLN) for spatial feature extraction by aggregating the spatial features and the at least one 2D image.

Citation Information

Patent Citations

  • US10,321,728