Extracting and using measurements from images

Abstract

A garment size appropriate for a user is determined by obtaining a reference image of a human wearing a corresponding garment and obtaining nominal size data for the garment worn in the reference image. Body dimensions of the human depicted in the reference image are derived from the reference image and / or from another data source and are compared with the nominal size data of the garment. Corrections based on variations between the body dimensions and the nominal size data are used to generate corrected size data for the garment. On receiving a request for the garment in conjunction with user body data uploaded by the user, for example in a body image, the corrected size data is looked up for the requested garment. A size recommendation for the requested garment is conveyed to the user based on a comparison between the uploaded user body data and the corrected size data.

Description

[0001] Extracting and using measurements from images

[0002] This invention relates to the challenges of extracting and using dimensional measurements derived from an image without the need for a reference for scale in the image. Such images could include bodies of humans or animals, and objects such as items of clothing.

[0003] In this specification, references to the body are not limited to the torso but include other parts of the body, such as the legs, feet, arms, hands, head and neck.

[0004] In some commercial applications of the invention, images can be paired. For example, images of human bodies and garments can be paired to determine garments that will be a correct fit for certain body sizes and body shapes. When shopping for garments online, this enables a user to upload a body image and to receive suggestions as to garments that will suit the size and shape of that body.

[0005] As providing separate dimensional measurements of a body may be inconvenient and inaccurate, it is desirable for those measurements to be derived from the image itself but this is challenging, especially if the image lacks a reliable point of reference to determine scale.

[0006] Inferring body measurements can be attempted by using a pre-trained 2D body segmentation network such as BodyPix (trade mark), an open-source machine learning model that provides for image segmentation. Specifically, the BodyPix model is trained to classify the pixels of an image into pixels that represent a body and pixels that represent background, and further to classify the pixels representing the body into any of various body parts. Thus, BodyPix can be used to extract body shape from an image. Thereafter, in principle, linear interpolation can be used to predict various dimensions of the determined body shape. However, error rates have been found to be unacceptably high in this approach, in some cases up to 50%. Accuracy of significantly better than 90% is likely to be necessary in commercial applications.

[0007] A similar approach is disclosed in US 10321728, which teaches systems and methods for extracting body measurements using a mobile device camera. After identifying body features associated with a human body in an image, body feature annotation is performed on the identified body features to generate an annotation line on each body feature. Body feature measurements are generated from the annotated body features using a machine-learning sizing module and body size measurements are generated by aggregating the body feature measurements for each body feature.

[0008] Other prior art documents in this field include CN 114419677, US 10410414, US 10706262, JP 2008224323 and WO 2007 / 087485.

[0009] Against this background, an aspect of the invention resides in a method of assessing an actual size of a garment. The method comprises: obtaining a reference image of a human wearing a garment of a corresponding type; obtaining nominal size data for the garment worn by the human in the reference image; determining body dimensions of the human depicted in the reference image; comparing the body dimensions with the nominal size data to determine variations between the body dimensions and the nominal size data; calculating correction values based on the variations; applying the correction values to the nominal size data to generate corrected size data for the garment; and storing the corrected size data in a database.

[0010] The body dimensions can be determined by applying a machine learning model to the reference image. The machine learning model can be trained using image data representing the body of the human depicted in the reference image. The image data can be obtained from the reference image or from another image of the human depicted in the reference image.

[0011] The body dimensions can instead, or additionally, be determined by looking up body dimension data for the human depicted in the reference image. In the latter case, a machine learning model can be trained using the body dimension data in conjunction with the image data.

[0012] Images of the human depicted in the reference image may be classified to distinguish desired images, preferably full-body front-facing images, from unwanted images before deriving the image data from the desired images. Once generated, a dataset of the desired images and the unwanted images can be assessed and corrected before using the corrected dataset to train a classification model that classifies reference images such as the reference image.

[0013] The reference image, the body dimension data, the image data and / or the nominal size data can be obtained from the same source, such as a single website or web page, or from different sources, such as different websites or web pages. The nominal size data may, for example, be obtained from a size chart relating to the garment.

[0014] The method may further comprise: receiving a request for a garment from a user; receiving user body data uploaded by the user, for example from a body image uploaded by the user; looking up the corrected size data for the requested garment in the database; and recommending a garment size for the requested garment to the user based on a comparison between the uploaded user body data and the corrected size data. The user body data could be obtained by applying the same machine learning model to the uploaded body image.

[0015] In an initial effort to improve measurement accuracy, we gathered a dataset of approximately five thousand images of male and female humans and body measurements of those humans, namely height, bust / chest, waist, hips and inseam. We investigated the possibility of direct measurement extraction using a residual neural network (ResNet) running a regression algorithm. We found that by using transfer learning on an eighteen-layer ResNet model previously trained on the ImageNet visual database, it was possible to infer measurements with up to 90% accuracy.

[0016] So, by building a high-resolution ResNet-based machine-learning architecture, we can ensure sufficient resolution to infer an accurate scale. Using that inferred scale, we can then accurately predict dimensional measurements of the subject, whether a human or animal body or an object such as a garment.

[0017] To improve accuracy further, we have taken the approach of improving both the dataset and the model. In this respect, accuracy of measurement depends upon having a large, accurate and consistent dataset. We found that the best approach was to assemble the improved dataset first and to use that as the basis for improving the machine-learning model architecture.

[0018] Thus, we moved from a single dataset to a dataset assembly pipeline that consistently scrapes, anonymises and feeds training data to the models. In this way, the dataset has been enlarged to stand currently at twenty-four thousand images of six thousand humans. The result is a self-training dataset assembly pipeline, allowing the models to be trained iteratively on a dataset of constantly increasing size. For every training iteration, we calculated the mean average error across all output dimensions using Adam optimisation. Moreover, the regression model architecture used in Al / machine learning was increased to fifty layers. In conjunction with this, greyscale pre-processing was used to reduce the overall training time. The regression model was then split into male and female versions and trained separately to predict body measurements with 99% accuracy.

[0019] An important commercial application of the invention lies in accurate sizing of garments to assess their suitability for certain body sizes and body shapes, whose dimensions can also be measured by the invention. In this respect, nominal garment sizes such as dress sizes or S / M / L / XL sizes are so variable as to be essentially meaningless. What is needed instead is a consistent, accurate, measurement-based system that can work across all garments.

[0020] We considered compiling a dataset of garment images and measurements to train a separate regression model for clothing but it was impractical to obtain the necessary volume of images. We therefore considered how we could improve on the accuracy of measurements provided by garment manufacturers and garment vendors.

[0021] When crawling an online store, a classifier model of the invention identifies a full-front image of a human wearing a garment of a nominal size identified on that store. A regression model of the invention is used to process the image, thereby to obtain measurements of the human’s body. This gives body measurements that fit the nominal size of the garment.

[0022] By determining the nominal size of the garment and the dimensions attributed to that garment in a size chart, we can, in effect, reverse-engineer the garment displayed on the human to produce accurate size charts. This is done by subtracting the human’s body measurements from the measurements in the size chart for the given size of garment, allowing us to calculate a size chart error for each measurement and hence a correction offset required to correct that error. For this purpose, the size charts may be standardized and converted from bracket sizing to point sizing as necessary. The chart data may be stored as vectors in conjunction with the error correction offset.

[0023] In a pairing process of the invention, a user can upload an image of themselves from which we can extract measurement data and cross-reference that data with a garment size chart which may be corrected as described above. This allows accurate determination not just of best fit, but also to know if a garment is unlikely to fit any part of the body. By comparing all available measurements of the body and the garment, we are more likely to predict the correct garment size. Sizing must be done on an object- by-object level as size differences are critical.

[0024] By measuring different sizes of a garment as vectors, and by measuring a body as a vector, we can predict the most accurate fit by finding the smallest Euclidean distance between the body vector and the garment vectors. This enables automatic garment sizing based on calculating dimensional distances between garment and body metrics. Thus, we can extract measurements from a single two-dimensional image with a high degree of accuracy without the need for a point of reference for scale in that image. We can also pair the subjects of such images to fit one subject to another, for example, human or animal bodies and garments.

[0025] The invention can employ a plug-in to accept images from a third-party website and an application programming interface (API) to convert those images to measurements. Measurements can be logged in a dashboard for a manufacturer or vendor of products to use as feedback for production. Moreover, integration with the manufacturer’s or vendor’s analytics can measure business impact.

[0026] The approach of the invention is far simpler than previous approaches, which require a minimum of two images and / or the use of LIDAR or photogrammetry. We have combined the use of a classification algorithm to ensure image quality with a regression algorithm to predict the measurements of image subjects. For example, we can use a two-stage machine-learning pipeline on both human bodies and garments, using a classifier to identify images that are most likely to yield the correct data and a regression model to extract the data from those images. We can scrape a large number of images using the classifier to filter for useful images from which measurements can be taken and we can train a machine learning model to classify images into those that are useful for training, such as full-front images, and those that are not.

[0027] Calculating a distance between points to ascertain best fit relies upon correct data. We use a combination of existing published data and data extracted from images to arrive at more accurate measurements than are currently held, or made available, by manufacturers or vendors. Aspects of the invention therefore contemplate reverse-engineering garments displayed on a human subject to produce accurate garment size charts; automatic sizing based on calculating dimensional distances between garment and body metrics; and / or a selftraining dataset assembly pipeline.

[0028] To summarise a commercial application of the invention, a garment size appropriate for a user may be determined by obtaining a reference image of a human wearing a corresponding garment and obtaining nominal size data for the garment worn in the reference image. Body dimensions of the human depicted in the reference image may be derived from the reference image and / or from another data source and may be compared with the nominal size data of the garment. Corrections based on variations between the body dimensions and the nominal size data can then be used to generate corrected size data for the garment.

[0029] On receiving a request for the garment in conjunction with user body data uploaded by the user, for example in a body image, the corrected size data can be looked up for the requested garment. A size recommendation for the requested garment can then be conveyed to the user based on a comparison between the uploaded user body data and the corrected size data.

[0030] In order that the invention may be more readily understood, reference will now be made, by way of example, to the accompanying drawings in which:

[0031] Figure 1 is a diagram representing a self-training pipeline for assembling a training dataset comprising body images and corresponding size measurement data;

[0032] Figure 2 is diagram representing a pipeline for assembling a garment size database;

[0033] Figure 3 is a diagram representing use of the garment size database in a pairing process to suggest garments that will fit a user’s body size; and

[0034] Figure 4 is a diagram that illustrates generation of a corrected size chart.

[0035] Figure 1 shows how to assemble a training dataset of human body size data and corresponding body images from selected websites or other sources containing human measurement and image data, referred to hereinafter simply as source sites. In Figure 1 , a pipeline for assembling a training dataset comprises a scraper 10 that in turn comprises a crawler 12. The crawler 12 interacts with a browser 14 via a browser driver 16 to extract human body size data 18 and corresponding body images 20 from selected source sites. The crawler 12 operates in accordance with page parsing rules 22 set within source site modules 24 configured to suit the respective selected source sites. The source site modules 24 apply size information parsing rules 26 to the body size data 18 to obtain measurement data 28 that is added to a regression model training dataset 30.

[0036] The body images 20 are processed by a classifier 32 in the scraper 10 that separates those images into desired images 34, in this example full-body front-facing images referred to hereinafter simply as full-front images, and unwanted images 36 that do not meet the desired criteria. Optionally, as shown, human dataset correction 36 can be used to check the classification of desired full-front images 34 and unwanted images 36 produced by the classifier 32 and to assemble a classification model training dataset 38 that is used to train the classifier 32. The desired full-front images 34 may then undergo face segmentation and blurring 42 to anonymise them before they are added to the regression model training dataset 30 in conjunction with the corresponding measurement data 28.

[0037] Figure 2 shows how to assemble a garment size database from garment size charts and corresponding garment images from websites of selected online stores. In Figure 2, a pipeline for assembling a garment size database comprises a scraper 44 that in turn comprises a crawler 46. The crawler 46 interacts with a browser 48 via a browser driver 50 to extract garment size charts 52 and corresponding garment images 54 from websites of selected online stores. The crawler 46 operates in accordance with page parsing rules 56 set within store modules 58 configured to suit the respective selected store websites. The store modules 58 apply size chart parsing rules 60 to the garment size charts 52 to obtain garment size data 62 that is added to a garment sizing dataset 64.

[0038] The garment images 54 are processed by a classifier 66 in the scraper 44 that extracts desired images, in this example full-front images 68, and applies them to a regression model 70 in conjunction with the garment size data 62 scraped from the corresponding garment size charts 52. Although not shown in Figure 2, the classifier 66 can be trained, for example by assembling a classification model training dataset using human dataset correction as in Figure 1 , to separate the garment images 54 into desired full-front images 68 and unwanted images that do not meet the desired criteria.

[0039] Figure 3 shows how the garment sizing dataset of Figure 2 can be used to recommend an appropriate garment size to a user in a pairing process of the invention. Figure 3 shows how the garment sizing dataset 64 of Figure 2 can be used to recommend an appropriate garment size to suit a user’s body size and body shape. Here, a browser 72 runs a JavaScript app 74 that receives product data 76, exemplified here by store ID, department ID and product code, characteristic of a garment chosen by the user on the browser 72. On being launched by a launch button 78 in the browser 72, a nested iframe 80 is created in which the user can upload an image 82 of their body. The image 82 is uploaded to an API 84 in conjunction with the product data 76.

[0040] In the API 84, the image 82 is applied to a regression model 86 to obtain customer measurements 88. The customer measurements 88 are applied to a size lookup 90 that outputs a recommended size 92 and displays that recommended size 92 to the user via the iframe 80 in the browser 72.

[0041] The size lookup 90 receives garment size input from the garment sizing dataset 64, using the product data 76 uploaded to the API 84 in conjunction with the user’s body image 82. A query is made at 94 as to whether the garment sizing dataset 64 contains a garment size chart for the garment chosen by the user. If not, the size lookup 90 uses a garment size derived from a generic size chart 96. Conversely, if the garment sizing dataset 64 contains a garment size chart for the chosen garment, a query is made at 98 as to whether the garment sizing dataset 64 also contains correction data for that garment. If not, the size lookup 90 uses a garment size derived from the size chart 100 held by the garment sizing dataset 64. If the garment sizing dataset 64 contains correction data for the chosen garment, a corrected size chart 102 is generated and used by the size lookup 90.

[0042] Turning finally to Figure 4, this shows a commercial application of the invention, namely garment shopping, as an example of the pairing steps described above with reference to Figure 3. Like numerals are used to denote like features. Figure 4 further shows how a corrected size chart, as described above with reference to Figure 3, may be generated. In a user flow, Figure 4 exemplifies a user’s experience of the invention where the user, while shopping for a garment 104 online using a browser 72, uploads a body image 82 via a window 80 embedded into a web page that displays the garment 104. A machine learning model 86 extracts the user’s relevant measurements 88 (in this example, the bust, waist and hips) from the image 82 and uses those measurements 88 to look up the appropriate size of the garment 104 for that user in an improved size chart 102 that has been corrected, better to reflect the corresponding actual measurements of that particular garment 104 or brand of garment. The lookup process selects and outputs the most appropriate garment size at 92, in this case recommending to the user to try the chosen garment 104 in a female size ten to suit the measurements 88 derived from the uploaded image 82.

[0043] As published size charts are often inaccurate, a size chart correction system of the invention can improve accuracy by taking the measurements of a human 106 shown wearing the relevant garment 104 in a given nominal size, in this case a female size twelve, on a product webpage 108 or other source of a reference image. A machine learning model 110 derives the relevant body measurements 112 of the human 106 (again, the bust, waist and hips in this example) from the reference image and / or from other sources such as a source site. The machine learning model 110 could be the same machine learning model 86 that is used to determine the measurements of the user.

[0044] The body measurements 112 of the human 106 are then compared to the nominal measurements for the garment 104 worn by the human 106, as provided in a corresponding published garment size chart 100. In this way, it is possible to calculate divergence of the actual size of the garment 104 from the published size chart 100 and to apply a correction factor at 114 to improve the accuracy of the published size chart 100. The corrected, hence improved, size chart 102 is then used in the aforementioned size lookup process that selects and outputs the most appropriate garment size for a particular user.

[0045] Thus, the machine learning model 112 need not determine the measurements of the garment 104 directly. Instead, the machine learning model 112 is used to determine the measurements of a human 106 wearing the garment 104. This gives the body dimensions that the nominal size of the garment 104 actually fits. For instance, if a human 106 with certain body dimensions fits a garment 104 of size twelve whereas the published size chart 100 for the garment 104 suggests that a female of similar body dimensions should buy that garment in size ten, a corrected size chart 102 can be generated based on an appropriate correction factor applied at 114.

Claims

Claims1 . A method of assessing an actual size of a garment, the method comprising: obtaining a reference image of a human wearing a garment of a corresponding type; obtaining nominal size data for the garment worn by the human in the reference image; determining body dimensions of the human depicted in the reference image; comparing the body dimensions with the nominal size data to determine variations between the body dimensions and the nominal size data; calculating correction values based on the variations; applying the correction values to the nominal size data to generate corrected size data for the garment; and storing the corrected size data in a database.

2. The method of Claim 1 , comprising determining the body dimensions by applying a machine learning model to the reference image.

3. The method of Claim 2, comprising training the machine learning model using image data representing the body of the human depicted in the reference image.

4. The method of Claim 3, comprising obtaining the image data from the reference image.

5. The method of Claim 3, comprising obtaining the image data from another image of the human depicted in the reference image.

6. The method of any of Claims 3 to 5, comprising:classifying images of the human depicted in the reference image to distinguish desired images from unwanted images; and deriving the image data from the desired images.

7. The method of Claim 6, wherein the desired images are full-body front-facing images of the human depicted in the reference image.

8. The method of Claim 6 or Claim 7, comprising: generating a dataset of the desired images and the unwanted images; assessing and correcting the dataset; and using the corrected dataset to train a classification model that classifies reference images.

9. The method of any preceding claim, comprising determining the body dimensions by looking up body dimension data for the human depicted in the reference image.

10. The method of Claim 9 when dependent on any of Claims 3 to 8, comprising training the machine learning model using the body dimension data in conjunction with the image data.

11. The method of Claim 10, comprising obtaining the body dimension data and the image data from the same website.

12. The method of Claim 10, comprising obtaining the body dimension data and the image data from different websites.

13. The method of any of Claims 9 to 12, comprising obtaining the body dimension data and the reference image from the same website.

14. The method of any of Claims 9 to 12, comprising obtaining the body dimension data and the reference image from different websites.

15. The method of any preceding claim, comprising obtaining the nominal size data from a size chart relating to the garment.

16. The method of any preceding claim, comprising obtaining the nominal size data and the reference image from the same website.

17. The method of any of Claims 1 to 15, comprising obtaining the nominal size data and the reference image from different websites.

18. The method of any preceding claim, further comprising: receiving a request for a garment from a user; receiving user body data uploaded by the user; looking up the corrected size data for the requested garment in the database; and recommending a garment size for the requested garment to the user based on a comparison between the uploaded user body data and the corrected size data.

19. The method of Claim 18, comprising obtaining the user body data from a body image uploaded by the user.

20. The method of Claim 19 when dependent on Claim 2, comprising obtaining the user body data by applying the same machine learning model to the uploaded body image.

Images