Zooming system and method based on estimated facial features

Abstract

A system and method for scaling a user's head based on estimated facial features is disclosed. In one example, a system includes a processor configured to acquire a set of images of a user's head; generating a model of the user's head based on the set of images; determining a scaling ratio based on the model of the user's head and the estimated facial features; and applying the scaling ratio to the model of the user's head to obtain a scaled user's head model; and a memory coupled to the processor, the memory configured to provide instructions to the processor.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 337,983, filed on May 3, 2022, entitled “USING ESTIMATED FACIAL FEATURES TO DETERMINE SCALING OF A MODEL OF A USER'S HEAD,” the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] The described embodiments generally relate to generating a scaled model of a user. More specifically, the embodiments relate to generating a scaled model of a user based on estimated facial features of the user, which can be used for virtual try-on of a product. Background Art

[0004] People who want to buy glasses usually have to go to an optometrist in person to obtain head measurements and then use these measurements to purchase eyeglass frames. In addition, people traditionally go to an optometrist or optical shop in person to try on several pairs of eyeglass frames to assess their fit. Typically, this requires hours of browsing through rows of frames and trying on many pairs of frames, most of the time without knowing in advance whether a particular frame will be a good fit for that person.

[0005] Allowing people to virtually obtain facial feature measurements and try on eyeglass frames will greatly improve the efficiency of selecting eyeglass frames. However, in order to better approximate people's real-world try-on experience, the size of the eyeglass frames in the virtual try-on experience should be accurate. In addition, the size of the eyeglass frames should fit the person's face based on the person's facial feature measurements. Summary of the invention

[0006] According to some aspects of the present disclosure, a system includes a processor and a memory coupled to the processor, the memory being configured to provide instructions to the processor. When accessing the instructions, the processor is configured to obtain a set of images of a user's head, generate a 3D model of the user's head based on the set of images, determine a scaling ratio based on the model of the user's head and estimated facial features, and apply the scaling ratio to the model of the user's head to obtain a scaled model of the user's head.

[0007] In some examples, the estimated facial features may include historical facial features. In some examples, determining the scaling ratio may include determining a measured facial feature from one of the set of images, updating the model of the user's head based on the measured facial feature, and determining scaling information based on the measured facial feature and at least a portion of the estimated facial feature.

[0008] In some examples, determining the scaling ratio can include determining a head width classification corresponding to the user's head using a machine learning model based on the set of images, obtaining a set of ratios corresponding to the head width classification, determining a measured facial feature from the model of the user's head, and determining the scaling ratio based on the measured facial feature and the estimated facial features. In some examples, the estimated facial features can include the set of ratios.

[0009] In some examples, the processor may also be configured to position an eyeglass frame model on the scaled user head model and determine a set of facial measurements associated with the user's head based on stored measurement information associated with the eyeglass frame model and the position of the eyeglass frame model on the scaled user head model.

[0010] In some examples, the processor may be further configured to determine a confidence level corresponding to a facial measurement in the set of facial measurements. In some examples, the processor may be further configured to compare the set of facial measurements to a set of stored dimensions of eyeglass frames and output a recommended eyeglass frame in a user interface based at least in part on the comparison.

[0011] In some examples, the processor may also be configured to input the set of facial measurements into a machine learning model to obtain a set of recommended eyeglass frames and output the set of recommended eyeglass frames in a user interface.

[0012] According to some examples, a method for generating a three-dimensional (3D) model may include receiving a set of images of an object, generating an initial model of the object based on the set of images, determining a first measurement value of a first feature of the object, classifying the object with a measurement classification associated with an estimated measurement value of the first feature, determining a scaling ratio of the initial model based on the first measurement value and the estimated measurement value, and scaling the initial model based on the scaling ratio to generate a scaled model.

[0013] In some examples, the object may be a user's head and the first feature may include facial width. In some examples, the measurement category may be selected from a list including narrow, medium, and wide.

[0014] In some examples, the method may also include positioning a 3D model on the scaled model and generating a measurement of the object based on the position of the 3D model on the scaled model and a comparison of the 3D model with the scaled model. In some examples, the 3D model may be associated with a real-world size.

[0015] In some examples, the method may also include determining measurements of the object based on the scaled model. In some examples, the method may also include determining a confidence level corresponding to each of the measurements.

[0016] In some examples, the method may further include receiving a second set of images and analyzing the second set of images using a machine learning model. In some examples, each image in the second set of images may include a learning object that includes a learning feature associated with a second measurement value and a corresponding measurement classification. In some examples, the machine learning model may associate each corresponding measurement classification in a set of measurement classifications with a corresponding second measurement value. In some examples, the measurement classification is selected from the set of measurement classifications to classify the object.

[0017] According to some examples, a computer program product embodied in a non-transitory computer-readable storage medium includes computer instructions for: receiving a set of images of a user's head; generating an initial three-dimensional (3D) model of the user's head based on the set of images; analyzing the set of images to detect a facial feature on the user's head; comparing the detected facial feature with an estimated facial feature to determine a scaling ratio, the estimated facial feature including at least one of an iris diameter, an ear connection distance, or a temple distance; and scaling the initial 3D model based on the scaling ratio to generate a scaled 3D model.

[0018] In some examples, the estimated facial feature may include an average measurement of the facial feature in a population, and the computer instructions may further include determining the estimated facial feature. In some examples, the estimated facial feature may include the iris diameter; and the iris diameter may be 11 mm to 13 mm.

[0019] In some examples, the computer instructions may also include positioning a 3D model of an eyeglass frame on the scaled 3D model and determining facial measurements of the user based on measurements associated with the 3D model of the eyeglass frame and the position of the eyeglass frame on the scaled 3D model.

[0020] In some examples, the computer instructions may also include determining a head width classification of the user's head and determining the estimated facial features based on the head width classification of the user's head.

[0021] In some examples, the computer instructions may also include: using a machine learning model including a set of image inputs, associating a head width classification in a set of head width classifications with a corresponding estimated facial feature in a set of estimated facial features. In some examples, each image in the set of images may include a head width classification and a facial feature measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0023] Figure 1 is a flow chart of a method of generating a scaled model of a user's head.

[0024] Figure 2 is a block diagram of a system for generating a scaled model of a user's head.

[0025] Figure 3 is a module diagram of a server for generating a scaled model of a user's head.

[0026] Figure 4 A set of images of a user's head is shown.

[0027] Figure 5 Reference points on the user's head are shown.

[0028] Figure 6 is a flow chart of a method of generating a scaled model of a user's head.

[0029] Figure 7 is a flow chart of a method of generating a scaled model of an object. DETAILED DESCRIPTION

[0030] The present example systems and methods may be implemented in a variety of ways, including as a process; an apparatus; a system; a material combination; a computer program product embodied on a computer-readable storage medium; and / or a processor, such as a processor configured to execute instructions stored on and / or provided by a memory coupled to the processor. In this specification, these implementations or any other form that the present systems and methods may take may be referred to as techniques. In general, the order of steps of the disclosed processes may be changed within the scope of the claimed invention. Unless otherwise stated, a component described as being configured to perform a task (e.g., a processor or memory) may be implemented as a general component that is temporarily configured to perform the task at a given time, or as a specific component manufactured to perform the task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data (e.g., computer program instructions).

[0031] A detailed description of one or more embodiments of the present system and method is provided below in conjunction with the accompanying drawings, which illustrate the principles thereof. The description of the present system is related to these embodiments and is not limited to any embodiment. Instead, the scope is limited only by the claims and includes numerous alternatives, modifications, and equivalents. In order to provide a thorough understanding of the present example system and method, many specific details are set forth in the following description. These details are provided for illustrative purposes, and according to the claims, the present system and method can be practiced without some or all of these specific details. For the sake of clarity, technical materials known in the technical field related to the present example system and method are not described in detail to avoid unnecessarily obscuring the present invention.

[0032] A model of a user's head can be generated and scaled based on a two-dimensional (2D) image showing the user holding a reference object (e.g., a standard-sized card such as a credit card) against their face. The model can be used to collect measurements of the user's head, which can be used to provide product recommendations to the user, and products can be overlaid and displayed on the model of the user's head. However, generating a model of the user's head based on a 2D image may not be accurate enough. For example, because a 2D image contains 2D information, the actual size and orientation of an object in the 2D image cannot be determined, or errors may occur. The apparent size of a reference object in a 2D image may appear different due to its tilt or orientation, so comparing the apparent size of a reference object in a 2D image to the apparent size of the user's head does not provide sufficient accuracy to correctly scale the model of the user's head.

[0033] In more detail, a method for scaling a 3D model of a user's head using a 2D image including a user's head and a reference object (e.g., a standard-sized card, such as a credit card or library card) may include obtaining an image of the user's head showing the reference object. The location of certain features of the reference object may be detected, such as two or more corners of a standard-sized card. 2D point features of the user's head may be detected and used to determine scaling. These features may include, for example, the outer corners of the user's eyes. The scaling factor may then be calculated using the known physical dimensions of the reference object (e.g., the height and width of a standard-sized card) and the detected feature locations on the user's head. However, it may be inconvenient for the user to obtain a suitable reference object to appear in the image with the user's head.

[0034] Some head measurements used to recommend products to users (e.g., eyeglass frames, prescription glasses, etc.), such as pupillary distance (PD), sub-lens height, and facial width, need to be determined with higher accuracy than can be provided by analyzing 2D images including reference objects. Pupil distance is a measurement of the distance between the centers of a user's pupils. Sub-lens height is a measurement from the bottom of the lens of a pair of glasses to the center of the user's pupils. Facial width is a measurement based on the distance between the user's contralateral ear connection or temple. In order to accurately measure the sub-lens height of a pair of glasses relative to the user's face, the position of the glasses on the user's face must be accurately located in three-dimensional (3D) space, which is difficult to achieve using methods based on 2D images. Similarly, in order to accurately measure the user's facial width, the exact direction of the user's head must be determined, which is also difficult to achieve using methods based on 2D images. Therefore, a technology is needed that can accurately determine user head measurements without the need for reference objects.

[0035] The following disclosure relates to systems and methods for determining the scaling of a user's head model using estimated facial features. These systems and methods can generate a 3D model of a user's head and scale the model based on the estimated facial features. The 3D model can then be used to determine the measurements of the user's head, recommend products to the user, display products to the user (e.g., through virtual try-on, etc.), and so on.

[0036] Various examples described herein eliminate the use of reference objects in input images used to generate and scale a model of a user's head. Instead, estimated facial features of the user are used to determine the appropriate scaling of a 3D model of the user's head. In some examples, a 3D model of the user's head is generated based on estimated sizes of points or features on the user's head. By avoiding the use of reference objects, the convenience of a user submitting an image to obtain head measurements, suitable recommendations for products that fit the user's head measurements, and a 3D preview of the product on the user is increased.

[0037] Figure 1 Flowchart of a method for generating a scaled model of a user's head based on estimated facial features. In step 102, images of the user's head are obtained. In some examples, the images include at least one frontal image of the user's head. The images may include a set of images (e.g., a video), such as a series of images capturing a user performing a head turn. In some examples, the user may be prompted to perform a specific head turn, or move the head to certain positions to obtain the set of images. In some examples, the set of images may include a minimum number of images of the user's head in a minimum number of positions for generating a 3D model of the user's head.

[0038] At step 104, a 3D model of the user's head is generated. The 3D model may be generated based on the set of images of the user's head. The 3D model may be a mesh model of the user's head. The 3D model may include one or more of: an image / video frame of the user's head, reference points on the user's head, or a set of rotation / translation matrices. In some examples, the 3D model is limited to reference points associated with the user's head. An initial 3D model may be generated based on a subset of the set of images of the user's head. The initial 3D model may then be adjusted to an adjusted 3D model using an iterative algorithm that incorporates additional information from the set of images of the user's head.

[0039] Each image in this set of images can be used together to generate a 3D model of the user's head. For example, each image in this set of images can be analyzed to determine the user's posture. The posture of the user's head in each image can include the rotation and / or translation of the user's head in the corresponding image. The posture information of each image can be referred to as external information. Reference points can be determined from this set of images and mapped to points on the 3D model of the user's head. Internal information can also be used to help generate a 3D model. Internal information may include a set of parameters related to the camera used to record this set of images of the user's head. For example, parameters related to the camera may include the focal length of the camera. Internal information can be calculated by correlating points on the user's head detected when generating a 3D model. Internal information can help provide depth information and measurement information for generating a 3D model.

[0040] In step 106, facial features of the user are detected. Facial features can be detected by analyzing the set of images of the user's head and can be marked on the 3D model or recorded in other ways. Facial features can include any facial features that can be used to scale the 3D model to real-world dimensions. Facial features can include the location of facial features, the size of facial features, etc. In some examples, facial features can include the location and / or size of the user's iris, which can be marked by an iris outline applied to the image and / or 3D model. In some examples, facial features can include the location of the user's temple, ear junction, pupil, eyebrow, corner of the eye, nose tip, nose bridge, cheekbone, etc. In examples where facial features include the location of the user's temple or ear junction, facial features can include the width of the user's face. In examples where facial features include the location of the user's pupil, facial features can include the interpupillary distance of the user's face. As will be discussed in detail below, the diameter of the user's iris, the width of the user's face, and / or the interpupillary distance of the user can be used to scale the 3D model to real-world dimensions.

[0041] At step 108, a scaling ratio is determined by comparing the detected facial features with estimated facial features. The estimated facial features may include average measurements of facial features in various populations. For example, the estimated facial features may include average measurements of facial features based on race, facial description, age, height, weight, region, or any other population or grouping. The estimated facial features may include empirical or historical measurements of the user's facial features. For example, historical measurements of the user's pupil distance, face width, iris diameter, etc. may be used. At step 110, the 3D model is scaled based on the scaling ratio. As long as the distance in real world dimensions between any points on the 3D model of the user's head is determined or known, the scaling ratio can be determined and applied to the 3D model of the user's head to generate a scaled 3D model of the user's head based on the known distance.

[0042] In some examples, the estimated facial features may include iris diameter. The average measurement of human iris diameter is approximately 11 mm to 13 mm. One image (e.g., a frontal image) in the set of images of the user's head may be analyzed to detect the location of the user's iris, and the iris outline may be marked. This frontal image and any other images of the user's head may be combined to generate a 3D model of the user's head, and the iris outline may be marked on the generated 3D model. The detected user's iris diameter may be compared to the average diameter of human irises, and a scaling ratio may be determined based on the comparison. The generated 3D model of the user's head may then be scaled so that the diameter of the iris outline in the scaled 3D model matches the average diameter of human irises. Therefore, in some examples, a scaled 3D model of the user's head may be generated based on a comparison of the detected iris diameter and the average human iris diameter. The scaled 3D model of the user's head is scaled to match the real-world size of the user's head.

[0043] In some embodiments, the estimated facial features may include proportions of facial features. The proportions of facial features may be associated with head width classifications. In some examples, the database may include associations between head width classifications and facial feature proportions. In some examples, user images labeled with corresponding head width classifications (e.g., narrow, medium, wide, etc.) may be used to train a machine learning model. In some examples, other head classifications may be associated with proportions of facial features, such as feature size (e.g., nose size, lip size, eye size, face shape, etc.).

[0044] The machine learning model or other algorithm may determine the relationship between the proportions of the user's facial features and their corresponding head classification (e.g., head width classification). The proportions of the facial features may include the distance between the user's eyes, the ratio of the face length to the face width, the distance between the user's eyebrows and lips, the width of the user's jawline, the width of the user's forehead, etc.

[0045] The proportions of facial features can be calculated in 2D space or 3D space, depending on the type of data available for each user in the training data. In examples where the data set available to the user in the training data only includes a frontal image of the user's head, the proportions of facial features can be calculated in 2D space after determining the facial features (e.g., eyes, eyebrows, facial contours, etc.) in the frontal image. In examples where the data set available to the user in the training data includes a set of images of the user (e.g., a set of head rotation images), the proportions can be calculated in 3D space after generating a 3D model of the user's head. The 3D model can be generated and scaled based on this set of images as described above. Therefore, whether in examples where a single frontal image of the user's head or a set of images of the user's head is included in the training data of the machine learning model, the trained machine learning model will output facial feature proportions corresponding to head width classifications or other head classifications.

[0046] Each head classification and head width classification is associated with a corresponding set of facial feature ratios. The scaling ratio used to scale the generated user head 3D model can be determined by dividing the detected user facial feature ratios (e.g., the distance between the user's eyes) by the corresponding facial feature ratios associated with the user head classification. The generated user head 3D model can then be scaled using the scaling ratio so that the facial ratios of the scaled 3D model match the facial feature ratios associated with the user head classification. Therefore, in some examples, a scaled 3D model of the user's head can be generated based on a comparison of the detected facial feature ratios with the facial feature ratios associated with the user head classification. The scaled 3D model of the user's head is scaled to match the real-world size of the user's head.

[0047] In some examples, the estimated facial features may include historical measurements of features of the user's head. For example, the estimated facial features may include a previously measured pupil distance of the user. In these examples, the scaling ratio may be determined by dividing the detected pupil distance of the user by the previously measured pupil distance of the user. The scaling ratio may then be used to scale the generated 3D model of the user's head so that the pupil distance of the scaled 3D model matches the previously measured pupil distance of the user. Thus, in some examples, a scaled 3D model of the user's head may be generated based on a comparison of the detected facial features with the previously measured facial features of the user. The scaled 3D model of the user's head is scaled to match the real-world size of the user's head.

[0048] In some examples, a scaled 3D model of the user's head can be used to derive measurements of the user's head. These head measurements can be used for any purpose. The head measurements can include a monopupillary distance measurement, a bipupillary distance measurement, a face width, or any other desired measurement. In some examples, the measurements of the user's head can be used to order eyeglass frames that fit the user's head.

[0049] In some examples, at least some of the user's head measurements derived from the scaled 3D model of the user's head are assigned corresponding confidences or other accuracy classifications. For example, confidences can be assigned to single pupil distance measurements, double pupil distance measurements, face width measurements, etc. In some examples, confidence estimates can be based on a machine learning approach that can assign confidences or accurate / inaccurate labels to facial measurements derived from the scaled 3D model of the user's head. Such machine learning approaches can use different features to make such estimates. Examples of features that can be used by the machine learning approach for confidence estimates include the pose of the user's head in a frontal image, and confidence associated with placing facial features on the generated 3D model of the user's face.

[0050] In optional step 112, the eyeglass frame is superimposed on the scaled 3D model of the user's head. In some examples, the user's head measurements derived from the scaled 3D model of the user's head can be used to recommend products to the user. For example, the derived head measurements (e.g., monopupillary distance, bipupillary distance, face width, nose bridge width, etc.) can be compared with the actual size of the eyeglass frame in the database. The eyeglass frame whose size best fits or corresponds to the head measurement derived by the user can be output on the user interface as a product recommended for the user to try and / or purchase. In some examples, the recommendation of a product (e.g., eyeglass frame) can be generated using machine learning. For example, the head measurement derived by the user can be input into a machine learning model for providing eyeglass frame recommendations, and the machine learning model can output eyeglass frame recommendations to the user based on the user's head measurement. In some examples, the eyeglass frame recommendation output by the machine learning model can be based on the user's head measurement, as well as eyeglass frames purchased by users with similar head measurements.

[0051] Any recommended eyeglass frames provided to the user may be a subset of a set of available eyeglass frames. The user may select a frame to view from the subset of recommended eyeglass frames or the set of available eyeglass frames. When the user selects an eyeglass frame, the eyeglass frame may be output and superimposed on the scaled 3D model of the user's head for virtual try-on of the selected eyeglass frame.

[0052] In some examples, the selected eyeglass frame can be changed to fit the user. For example, when the user virtually tries on the selected eyeglass frame, the 3D model of the selected eyeglass frame can be scaled using a scaling ratio or the user's head measurements. Thus, in the virtual try-on, the user can see the correctly sized eyeglass frame superimposed on the scaled 3D model of their head.

[0053] In some examples, the measurements of the user's head can be calculated by placing a 3D model of the eyeglass frame (e.g., a selected eyeglass frame) on a scaled 3D model of the user's head. In other words, the measurements of the user's head can be calculated by utilizing a fitting method that places the 3D model of the eyeglass frame on the scaled 3D model of the user's head. A digital eyeglass frame database with accurate real-world dimensions can be maintained, and the eyeglass frames in the database can be installed on the scaled 3D model of the user's head. After placing the 3D model of the eyeglass frame on the scaled 3D model of the user's head, the measurements of the user's head can be calculated based on the placement of the 3D model of the eyeglass frame on the scaled 3D model of the user's head. These measurements can include sub-piece height, temple length, monopupillary distance, bipupillary distance, face width, nose bridge width, etc.

[0054] In some examples, the position of the user's pupil on the scaled 3D model can be used to measure the monopupillary distance, bipupillary distance, etc. In some examples, the detection and back projection of the iris center key point can be used to determine the position of the user's pupil on the scaled 3D model. The sub-piece height is the vertical measurement from the bottom of the eyeglass frame lens to the center of the user's pupil. The temple length is the measurement from the front of the lens to the point where the temple is placed where the user's ear connects. The bridge width is the width of the user's nose bridge where the eyeglass frame is placed. Once the 3D model of the eyeglass frame is placed on the scaled 3D model of the user's head, all of these measurements can be calculated because the scaled 3D model of the user's head has been accurately scaled.

[0055] Although method 100 is described as being used to generate a scaled model of a user's head, method 100 can be used to generate a scaled model of any object. For example, method 100 can be used to generate a scaled model of a user's body, any inanimate object, or any desired object. The estimated features may depend on the specific object that needs to be scaled. For example, height can be used to generate a scaled model of a user's body. Any known or estimated measurements can be used to generate a model of an object.

[0056] Figure 2 is a block diagram of a system 200 for generating a user head scaling model based on estimated facial features (e.g., for implementing Figure 1 For simplicity, the system 200 is referred to as being used to generate a scaled model. The data generated by the system 200 can be used for various other applications, including using the measurement data and the scaled model to fit an eyeglass frame to a user. In some examples, the system 200 can also be used to position the eyeglass frame relative to the scaled model of the user's head.

[0057] System 200 may include client device 204, network 206, and server 208. Client device 204 may be coupled to server 208 via network 206. Network 206 may include a high-speed data network and / or a telecommunications network. User 202 may interact with client device 204 to generate a scaled model of the user. The scaled model of the user may be used to determine various head measurements of the user. The scaled model may be used to "try on" a product, for example, by providing an image of the user on the user's body via client device 204 and viewing a virtual installation of the product on the user's body according to the techniques further described herein.

[0058] The client device 204 is configured to provide a user interface for the user 202. For example, the client device 204 may receive input, such as an image of the user 202 captured by a camera of the client device 204, or observe the user 202 interacting with the client device 204. A scaled 3D model of the user may be generated based on at least some of the information collected by the client device 204. In some examples, a simulation of placing a product on the user's body (e.g., placing a frame of glasses on the user's head) may be output to the user 202.

[0059] In some examples, client device 204 includes an input component, such as a camera, a depth sensor, a lidar sensor, other sensors, or a combination of multiple sensors. In examples where client device 204 includes a camera, the camera can be configured to observe and / or capture an image of user 202, from which facial features (also referred to as physical features) can be determined. User 202 can be instructed to operate the camera or pose for the camera as further described herein. The information collected by the input component can be used to generate a scaled 3D model and / or stored for use in generating a scaled 3D model.

[0060] The server 208 is configured to determine facial features from the input image, determine the correlation between the facial features and the user's estimated facial features, and output a user-scaled 3D model scaled to real-world size. The server 208 can be remote from the client device 204 and accessed via a network 206 (such as the Internet). Various functions of the system 200 and the method 100 can be embodied in the client device 204 or the server 208. For example, functions traditionally associated with the server 208 can be performed not only by the server 208, but also by the client device 204, and vice versa. The output can be provided to the user 202 with almost no (if any) delay after the user 202 provides the input image. Therefore, the user 202 can experience the real-time installation of the product. There are many applications for virtually installing a product on the user's face, such as virtually trying on glasses, cosmetics, jewelry and other facial accessories. For simplicity, the examples in this article mainly describe the real-time installation of eyeglass frames on the user's face / head. However, this is not meant to be limiting, and these techniques can be applied to trying on other types of accessories and can be applied to video installation (for example, there may be some delay).

[0061] Figure 3 is a block diagram of a server 300 for generating a scaled model of a user's head. In some examples, the server 300 can be used to virtually mount glasses on the scaled model of the user's head and obtain measurements of the user's head. In some examples, the server 300 can be used to generate a scaled model of any object. In some examples, Figure 2 The server 208 of the system 200 uses Figure 3 The server 300 may include an image storage 302, a model generator 304, a 3D model storage 306, an estimated feature storage 308, an external information generator 310, an internal information generator 312, a scaling engine 314, a glasses frame information storage 316, a rendering engine 318, and an installation engine 320. The server 300 may be implemented as a Figure 3 More, different and / or fewer components are shown in the example. Each of the image storage 302, 3D model storage 306, estimated feature storage 308 and eyeglass frame information storage 316 can be implemented using one or more types of storage media. Each of the model generator 304, external information generator 310, internal information generator 312, scaling engine 314, rendering engine 318 and installation engine 320 can be implemented using hardware and / or software. The various components of the server 300 may include Figure 2 The system 200 may be implemented in and / or through a server 208 and / or a client device 204 .

[0062] Image storage 302 may be configured to store sets of images. In some examples, each set of images is associated with a recorded video or a series of snapshots of a user's head (e.g., a user's face) at various orientations. In some examples, each set of images is stored along with data associated with the entire set or a single image in the set. Image storage 302 may be configured to store Figure 1 The image set referenced in step 102 of method 100 .

[0063] The model generator 304 may be configured to determine a mathematical 3D model of the user's head associated with each set of images. The model generator 304 may generate an initial 3D model, such as Figure 1 The 3D model generated in step 104 of method 100 may be scaled and updated to generate a scaled 3D model, for example Figure 1 The method 100 includes step 110 of scaling the 3D model.

[0064] The model generator 304 can detect facial features of the user's head and determine facial feature measurements of the user's head, which can be associated with the generated 3D model of the user's head and stored in the model generator 304. For example, the model generator 304 can detect the edges of the user's irises and determine the distance between the opposite edges of the user's irises, referred to as iris distance or iris diameter. The model generator 304 can detect the user's ear junctions and determine the distance between the user's opposite ear junctions, referred to as ear junction distance or face width. The model generator 304 can detect the user's temples and determine the distance between the user's opposite temples, referred to as temple distance or face width. The iris distance, ear junction distance, and temple distance can be measured using any suitable units, such as pixels, etc. As will be discussed in detail below, the iris distance, ear junction distance, temple distance, combinations thereof, or any other suitable distance or measurement value can be used to scale the 3D model of the user's head. The model generator 304 can be configured to store the detected facial features (e.g., as reference points) and the determined user head measurements in the 3D model storage 306.

[0065] A mathematical 3D model of the user's head (e.g., a mathematical model of the user's head in 3D space) can be set at an origin. In some examples, the 3D model of the user's head includes a set of points in 3D space that define a set of reference points (e.g., locations) associated with features on the user's head (e.g., facial features) that are detected from an associated set of images. Examples of reference points include the endpoints of the user's eyes, the endpoints of the user's eyebrows, the bridge of the user's nose, the connection points of the user's ears, the tip of the user's nose, etc.

[0066] In some examples, the mathematical 3D model determined for the user's head is referred to as an M-matrix. The M-matrix can be determined based on a set of reference points associated with facial features on the user's head, which are determined from an associated set of images. In some examples, model generator 304 can be configured to store the M-matrix determined for a set of images with the set of images in image storage 302. In some examples, model generator 304 can be configured to store the 3D model of the user's head in 3D model storage 306. Thus, model generator 304 can perform Figure 1 Step 106 of method 100 .

[0067] The estimated facial features storage 308 may be configured to store estimated facial features. In some examples, the estimated facial features include average feature sizes in a user population. For example, the average diameter of a human iris is in the range of about 11 mm to about 13 mm, and the average diameter of a human iris may be stored as an estimated facial feature in the estimated facial features storage 308. In some examples, the estimated facial features may be associated with feature classifications. For example, a user may describe their head as narrow, medium, or wide, and the average face width for each feature classification may be stored as an estimated facial feature in the estimated facial features storage 308. The estimated facial features stored in the estimated facial features storage 308 may be used to Figure 1 Step 108 of method 100 .

[0068] The external information generator 310 may be configured to determine a set of external information for each image in a subset of at least one set of images. The set of images may be stored in the image storage 302. In some examples, a set of external information corresponding to an image in a set of images describes one or more directions and translations of a 3D model of a user's head determined for the set of images, which directions and translations result in a correct appearance of the user's head in the corresponding image. In some examples, a set of external information determined for an image in a set of images associated with the user's head is referred to as an (R, t) pair, where R is a rotation matrix and t is a translation vector corresponding to the corresponding image. The (R, t) pair corresponding to an image in a set of images may convert (RxM+t) the M matrix (representing the 3D model of the user's head) corresponding to the set of images into an appropriate direction and translation of the user's head displayed in the image associated with the (R, t) pair. In some examples, the external information generator 310 may be configured to store the (R, t) pair determined for each image in a subset of at least one set of images in the image storage 302 together with the set of images.

[0069] The internal information generator 312 may be configured to generate a set of internal information for a camera associated with recording a set of images. The camera may be a camera for recording a set of images stored in the image storage 302. In some examples, a set of internal information corresponding to the camera describes a set of parameters associated with the camera. For example, the parameters associated with the camera may include focal length. In some examples, a set of internal information associated with the camera may be found by points between the relevant user and the scaling reference object in different images, and a set of internal information representing the internal parameters of the camera may be calculated using camera calibration techniques. In some examples, a set of internal information associated with the camera is found by using an automatic calibration technique that does not require a scaling reference. In some examples, a set of internal information associated with the camera may be referred to as an I matrix. In some examples, the I matrix projects a version of the user's head 3D model transformed by the (R, t) pair corresponding to a particular image onto a 2D surface of the camera focal plane. In other words, I*(RxM+t) causes the 3D model (M matrix) to be projected onto the 2D surface in the direction and translation transformed by the (R, t) pair corresponding to the image. The result of the projection onto the 2D surface is a view of the user's head seen from the camera. In some examples, intrinsic information generator 312 may be configured to store an I matrix determined for a camera associated with a set of images in image storage 302 along with the set of images.

[0070] The scaling engine 314 may be configured to generate a scaled 3D model of the user's head. For example, the scaling engine 314 may retrieve from the 3D model storage 306 a 3D model of the user's head generated by the model generator 304 based on a set of images in the image storage 302. The scaling engine 314 may determine a scaling ratio for the 3D model of the user's head. For example, the scaling engine 314 may compare the detected facial features and the determined user head measurements generated by the model generator 304 and stored in the 3D model storage 306 with the estimated facial features stored in the estimated facial features storage 308 to determine the scaling ratio. The scaling engine 314 may then scale the 3D model based on the scaling ratio to generate the scaled 3D model so that the detected facial features and the determined user head measurements correspond to the estimated facial features. For example, the scaling engine 314 may scale the 3D model of the user's head so that the iris distance of the scaled 3D model corresponds to the average diameter of a human iris. In some examples, scaling engine 314 may scale the 3D model of the user's head so that the ear connection distance and / or the temple distance correspond to an average facial width for a particular feature classification of the user (e.g., a narrow, medium, or wide head). Scaling engine 314 may perform Figure 1 Step 110 of method 100 .

[0071] The eyeglass frame information storage 316 can be configured to store information associated with various eyeglass frames. For example, the information associated with the eyeglass frame can include measurements of various regions of the frame (e.g., bridge length, lens diameter, temple distance, etc.), eyeglass frame renderings corresponding to various (R, t) pairs, mathematical representations of eyeglass frame 3D models that can be used to render eyeglass images of various (R, t) parameters, prices, identifiers, models, descriptions, categories, types, eyeglass frame materials, brands, part numbers, etc. In some examples, the 3D model of each eyeglass frame includes a set of 3D points that define various locations / portions of the eyeglass frame, including, for example, one or more of the following: a pair of bridge points and a pair of temple bend points. In some examples, the information associated with the eyeglass frame can include a range of user head measurements for which the eyeglass frame has a suitable or recommended fit.

[0072] The rendering engine 318 may be configured to render the selected eyeglass frame to be superimposed on the scaled 3D model of the user's head. The selected eyeglass frame may be an eyeglass frame having information stored in the eyeglass frame information storage 316. The scaled 3D model may be stored in the 3D model storage 306, or the rendering engine 318 may render the selected eyeglass frame on an image, such as a corresponding image of a set of images stored in the image storage 302. In some examples, the rendering engine 318 may be configured to render the eyeglass frame (e.g., selected by the user) for each image in a subset of at least one set of images stored in the image storage 302. In some examples, the rendering engine 318 may be configured to transform the eyeglass frame by the (R, t) pair corresponding to the corresponding image. In some examples, the rendering engine 318 may be configured to perform occlusion on the transformed eyeglass frame using an occluder determined from the scaled 3D model of the user's head at a direction and translation associated with the (R, t) pair. The eyeglass frame occluded by the occluder at the direction and translation excludes certain portions hidden by the occluder at the direction / translation. For example, the occluder may include a generic facial 3D model, or an M-matrix associated with an image associated with the set of images. The rendered eyeglass frame of the image may display the state of the eyeglass frame at an orientation and translation corresponding to the image, and may be superimposed on the image when the set of images is played back to a user of the client device. The rendering engine 318 may execute Figure 1 Step 112 of method 100 .

[0073] Figure 4A received set 400 of images and / or video frames of a user's head is shown. The image set 400 shows various orientations of the user's head (images 402 to 410). The image set 400 may be captured by a camera that the user is facing. The user may be instructed to turn his head while the camera captures video frames of the user's head. The user may be instructed to look left and then right. A video clip or animation of a person turning his head may be shown to the user and the user may be instructed to do the same. The number of video frames captured may vary. The camera may be instructed by the processor to capture the user's head in a continuous video or snapshot manner. For example, the camera may capture a series of images with a delay between each image capture. The camera may capture images of the user's head in a continuous capture mode, where the frame rate may be lower than capturing video. The processor may be local or remote, such as on a server. The image set 400 may be processed to remove redundant and / or other unwanted images, and specific images in the set may be identified as representing different orientations of the user's head. The image set 400 may be used to determine a 3D model of the user's head, which may be scaled, used for measurement, and used to place or install the selected eyeglass frame.

[0074] Figure 5 Detected reference points obtained from a set of images of a user's head are shown. The reference points define the locations of various facial features and are used to scale the 3D model of the user's head. Figure 5 A frontal image of a user's head is shown 500. Reference points may be placed on opposite sides of the user's iris so that an iris diameter 502 may be determined. Reference points may be placed at opposite ear junctions of the user so that a first face width 504 may be determined. Reference points may be placed at opposite temples of the user so that a second face width 506 may be determined. Any of the iris diameter 502, the first face width 504, and / or the second face width 506 may be used along with estimated facial features to scale a 3D model of the user's head.

[0075] Figure 6 A method 600 for generating a scaled 3D model of a user's head using iris diameter is shown. In step 602, a 3D model of the user's head is generated. In some examples, the 3D model may be unscaled or may be scaled using an arbitrary measurement value (e.g., pixels). The 3D model may be generated based on a set of images of the user's head, such as a set of images recorded while the user performs head turns. The set of images may include at least one frontal image of the user's head. Step 602 may be similar to or the same as step 102 of method 100, as described above with respect to Figure 1 discussed.

[0076] In step 604, the diameter of the user's irises is determined. The user's irises can be detected by analyzing the set of images of the user's head (e.g., a frontal image of the user's head). The boundaries of the user's irises can be marked or otherwise recorded on the 3D model (e.g., as reference points on the 3D model). An iris outline can be applied to the set of images and / or the 3D model of the user's head. The diameter of each of the user's irises can be measured or determined based on the boundaries of the user's irises. In some examples, the diameter of the user's irises can be measured in pixels, although any suitable unit of measurement can be used. Step 604 can be similar to or the same as step 106 of method 100, as described above with respect to Figure 1 discussed.

[0077] In step 606, the diameter of the user's iris is compared to an average diameter of human irises to determine a zoom ratio. The average measurement of the diameter of a human iris is approximately 11 mm to 13 mm. The average diameter of a human iris can be compared (e.g., divided) to the determined diameter of the user's iris to determine the zoom ratio. Step 606 can be similar to or the same as step 108 of method 100, as described above with respect to Figure 1 discussed.

[0078] In step 608, the 3D model is scaled based on the scaling ratio. The 3D model may be scaled using the scaling ratio by multiplying the 3D model of step 602 by the scaling ratio. Thus, the diameter of the user's iris in the scaled 3D model may correspond to or otherwise match the average diameter of a human iris. In some examples, the scaled 3D model may be used to present a 3D model of an eyeglass frame on a user's head in a virtual try-on or similar scenario. In some examples, head measurements may be determined from the scaled 3D model, such as for ordering prescription glasses or similar purposes. Measurements specific to the eyeglass frame (e.g., temple length, sub-lens height, etc.) may be obtained by superimposing an eyeglass frame model on the scaled 3D model and determining the measurements based on the superposition. Step 608 may be similar to or identical to step 110 of method 100, as described above with respect to Figure 1 discussed.

[0079] Figure 7 A method 700 for generating a scaled 3D model of an object using object classification is shown. In step 702, a 3D model of an object is generated. The object can be a user's head, a user's body, an eyeglass frame, or any other suitable object. In some examples, the 3D model can be unscaled or can be scaled with an arbitrary measurement value (such as pixels). The 3D model can be generated based on a set of images of the object, such as a set of images recorded as a camera rotates around the object or otherwise moves relative to the object. The set of images can include at least one frontal image of the object. Step 702 can be similar to or the same as step 102 of method 100, as described above with respect to Figure 1 discussed.

[0080] In step 704, a first measurement of the object is determined. The first measurement can be any suitable measurement, depending on the identity of the object, such as height, width, length, etc. In examples where the object is a user's face, the first measurement can be the width of the user's face, the distance between the user's relative ears, the distance between the user's relative temples, etc. In examples where the object is a user's body, the first measurement can be the user's height, the user's width, etc. In examples where the object is a frame of glasses, the first measurement can be the width of the frame of glasses. The first measurement can be determined by analyzing the set of images of the object. The boundaries of the object can be marked or otherwise recorded on the 3D model (for example, as reference points on the 3D model). In some examples, the first measurement of the object can be measured in pixels, although any suitable unit of measurement can be used. Step 704 can be similar to or the same as step 106 of method 100, as described above with respect to Figure 1 discussed.

[0081] In step 706, estimated measurements of the object are determined. The estimated measurements of the object can be determined by associating a measurement classification with the object. The measurement classification can be a general description of the object. For example, in an example where the object is a user's face, the measurement classification can be a description of the width of the user's face, such as narrow, medium, or wide. In an example where the object is a user's body, the measurement classification can be a description of the user's body. For example, the measurement classification can refer to the user's height, such as tall, medium, or short; the user's body type, such as stocky, slender, etc.; or similar descriptions. In an example where the object is an eyeglass frame, the measurement classification can be a description of the width of the eyeglass frame, such as narrow, medium, or wide; a description of the height of the eyeglass frame, such as short, medium, or tall; or similar descriptions. A machine learning model can be trained with an image of the object associated with descriptions and real-world measurements. Therefore, the measurement classification can be associated with the estimated measurements of the object. For example, the narrow width of the user's face, the tall height of the user's body, and the medium width of the eyeglass frame can each be associated with specific real-world measurements, which can be used as the estimated measurements of the object.

[0082] In step 708, the first measurement of the object is compared to the estimated measurement of the object to determine a scaling ratio. For example, if the measurement classification associated with the object is average male face width, the estimated measurement may be approximately 14 centimeters. The estimated measurement (e.g., average male face width is 14 centimeters) can be compared to (e.g., divided by) the determined first measurement of the object (e.g., the measured / determined user face width) to determine the scaling ratio. Step 708 can be similar to or the same as step 108 of method 100, as described above with respect to Figure 1 discussed.

[0083] In step 710, the 3D model is scaled based on the scaling ratio. The 3D model may be scaled using the scaling ratio by multiplying the 3D model of step 702 by the scaling ratio. Thus, a first measurement of the object in the scaled 3D model may correspond to or otherwise match an estimated measurement of the object. In some examples, the scaled 3D model may be used to present 3D models of various products on the object in a virtual try-on or similar scenario. The scaled 3D model may be a product that may be presented on 3D models of other objects in a virtual try-on or similar scenario. In some examples, measurements of the object may be determined from the scaled 3D model, which measurements may be used for sizing or ordering of various products. Step 710 may be similar to or the same as step 110 of method 100, as described above with respect to Figure 1 discussed.

[0084] Facial features may include any facial features that can be used to scale the 3D model to real-world size. Facial features may include the location of facial features, the size of facial features, etc. In some examples, facial features may include the location and / or size of the user's iris, which may be marked by an iris outline applied to the image and / or 3D model. In some examples, facial features may include the location of the user's temple, ear junction, pupil, eyebrow, corner of the eye, nose tip, nose bridge, cheekbone, etc. In examples where facial features include the location of the user's temple or ear junction, facial features may include the width of the user's face. In examples where facial features include the location of the user's pupils, facial features may include the interpupillary distance of the user's face. As discussed in detail above, the diameter of the user's iris, the width of the user's face, and / or the interpupillary distance of the user may be used to scale the 3D model to real-world size.

[0085] Although the foregoing examples have been described in considerable detail to facilitate understanding, the invention is not limited to the details provided. There are many alternative ways to implement the invention. The disclosed examples are illustrative rather than restrictive.

Claims

1. A system comprising: A processor; as well as a memory coupled to the processor, and the memory is configured to provide instructions to the processor, the instructions, when executed by the processor, causing the processor to: Get a set of images of the user's head; generating a model of the user's head based on the set of images; determining a zoom ratio based on the model of the user's head and estimated facial features; as well as The scaling ratio is applied to the model of the user's head to obtain a scaled user's head model.

2. The system of claim 1 , wherein the estimated facial features include historical facial features, and wherein determining the scaling ratio includes: determining a measured facial feature from one of the set of images; updating the model of the user's head based on the measured facial features; as well as The scaling information is determined based on the measured facial features and at least a portion of the estimated facial features.

3. The system of claim 1 , wherein determining the zoom ratio comprises: determining, using a machine learning model, a head width classification corresponding to the user's head based on the set of images; obtaining a set of ratios corresponding to the head width classification, wherein the estimated facial features include the set of ratios; determining a measured facial feature from the model of the user's head; as well as The scaling ratio is determined based on the measured facial features and the estimated facial features.

4. The system of claim 1, wherein the processor is further configured to: Positioning a glasses frame model on the scaled user head model; and A set of facial measurements associated with the user's head is determined based on stored measurement information associated with the eyeglass frame model and a position of the eyeglass frame model on the scaled user head model.

5. The system of claim 4, wherein the processor is further configured to determine a confidence level corresponding to a facial measurement in the set of facial measurements.

6. The system of claim 4, wherein the processor is further configured to: comparing the set of facial measurements to a set of stored dimensions for eyeglass frames; and A recommended eyeglass frame is outputted at a user interface based at least in part on the comparison results.

7. The system of claim 4, wherein the processor is further configured to: inputting the set of facial measurements into a machine learning model to obtain a set of recommended eyeglass frames; and The set of recommended eyeglass frames is outputted in a user interface.

8. A method for generating a three-dimensional (3D) model, comprising: receiving a set of images of an object; generating an initial model of the object based on the set of images; determining a first measurement of a first characteristic of the object; classifying the object using a measurement classification, wherein the measurement classification is associated with an estimated measurement value of the first characteristic; determining a scaling ratio of the initial model based on the first measurement value and the estimated measurement value; as well as The initial model is scaled based on the scaling ratio to generate a scaled model.

9. The method of claim 8, wherein: The object includes a user's head; and The first characteristic includes a facial width.

10. The method of claim 9, wherein the measurement classification is selected from a list consisting of narrow, medium, and wide.

11. The method of claim 8, further comprising: positioning a 3D model on the scaled model, wherein the 3D model is associated with a real-world size; as well as A measurement of the object is generated based on a position of the 3D model on the scaled model and a comparison of the 3D model to the scaled model.

12. The method of claim 8, further comprising determining a measurement of the object based on the scaled model.

13. The method of claim 12, further comprising determining a confidence level corresponding to each of the measurements.

14. The method of claim 8, further comprising: receiving a second set of images, wherein each image in the second set of images comprises a learning object, the learning object comprising a learning feature associated with a second measurement value and a corresponding measurement classification; as well as The second set of images is analyzed using a machine learning model to associate each respective measurement classification in a set of measurement classifications with a respective second measurement value, wherein the measurement classification is selected from the set of measurement classifications to classify the object.

15. A computer program product, the computer program product being embodied in a non-transitory computer-readable storage medium and comprising computer instructions for: Receiving a set of images of a user's head; generating an initial three-dimensional (3D) model of the user's head based on the set of images; analyzing the set of images to detect a facial feature on the user's head; comparing the detected facial feature to an estimated facial feature to determine a scaling ratio, wherein the estimated facial feature comprises at least one of an iris diameter, an ear connection distance, or a temple distance; as well as The initial 3D model is scaled based on the scaling ratio to generate a scaled 3D model.

16. The computer product of claim 15, wherein: The estimated facial feature comprises an average measurement of the facial feature in a population; and The computer instructions further include determining the estimated facial features.

17. The computer product of claim 15, wherein: The estimated facial features include the iris diameter; and The iris diameter is 11 mm to 13 mm.

18. The computer product of claim 15, wherein the computer instructions further comprise: Positioning a 3D model of an eyeglass frame on the scaled 3D model; and Facial measurements of the user are determined based on measurements associated with the 3D model of the eyeglass frame and a position of the eyeglass frame on the scaled 3D model.

19. The computer product of claim 15, wherein the computer instructions further comprise: Determining a head width category of the user's head; as well as The estimated facial features are determined based on a head width classification of the user's head.

20. The computer product of claim 19, wherein the computer instructions further comprise: Using a machine learning model including a set of image inputs, a head width classification in a set of head width classifications is associated with a corresponding estimated facial feature in a set of estimated facial features, wherein each image in the set of images includes a head width classification and a facial feature measurement.

Images