Apparatus, method, and computer readable storage medium for contextualized device recommendation

Abstract

The present disclosure relates to a method for providing a contextualized assessment of the suitability of a spectacle frame to a user's face to a user. In particular, the present disclosure relates to a method comprising: receiving user data describing features of the user's face; receiving device data describing features of the spectacle frame; generating, from a first model, a value of a set of specific criteria describing the compatibility between the user's face and the spectacle frame, the first model being trained to associate user data and device data with values of specific criteria; generating, from a second model, a value of a global criterion based on the generated value of the set of specific criteria, the second model being trained to associate values of specific criteria with values of global criteria; determining a message about the user's face characterizing the spectacle frame.

Description

BACKGROUND TECHNICAL FIELD

[0002] The present disclosure relates to eyewear and in particular to the matching of a visual equipment with a user’s face.

[0003] RELATED ART

[0004] In the process of selecting a new visual equipment or eyewear, users often need to introspect to determine the aesthetic appeal of the new eyewear on their face. Moreover, when choosing between multiple pieces of eyewear, users can find it difficult to determine which one is the most attractive, the most practical or the most suitable to their specific facial bone structure and features. At the same time, patients can struggle with their own perception of the new eyewear on their face and the hypothetical perception of third parties (e.g. friends, family, professionals, etc.) about the suitability of the new eyewear on their face.

[0005] As mentioned above, the task of eyewear selection can be arduous, taking into account both the aesthetic appeal and the necessity of the eyewear for correct vision, and there is no efficient way to confidently purchase an eyewear that will be satisfactory to the user, the user’s doctor and the user’s friends. The present disclosure provides a solution to this problem.

[0006] The foregoing “background” description is for the purpose of generally presenting the context of the disclosure. The work of the inventors, to the extent the inventors were aware of it, as well as that of other individuals and SUMMARY

[0007] The present disclosure relates to an apparatus, a method and a computer readable storage medium for contextualized equipment recommendation.

[0008] According to an embodiment, the present disclosure further relates to a method for providing a contextual assessment of a spectacle frame on a user’s face, the method comprising: receiving user data describing features of the user’s face; receiving equipment data describing features of the spectacle frame; generating, according to a first model, a value of a set of specific criteria describing a compatibility between the user’s face and the spectacle frame based on the received user data and the received equipment data, the first model being trained to associate user data and equipment data with values of specific criteria; generating, by processing circuitry and according to a second model, a value of a global criterion based on the generated values of the set of specific criteria, the second model being trained to associate values of specific criteria with values of global criteria; determining a message about the user’s face characterizing the spectacle frame, the message being associated with the generated value of the global criterion and the generated values of the set of specific criteria; and outputting the message to the user.

[0009] According to an embodiment, the disclosure further relates to a device for providing a contextual assessment of a spectacle frame on a user's face, the device comprising processing circuitry configured to: receive user data describing features of the user's face; receive device data describing features of the spectacle frame; determine, according to a first model, a value of a set of specific criteria describing a compatibility between the user's face and the spectacle frame based on the received user data and the received device data, the first model being trained to associate user data and device data with values of specific criteria; generate, according to a second model, a value of a global criterion based on the generated values of the set of specific criteria, the second model being trained to associate values of the specific criteria with values of the global criterion; determine a message about the user's face representing the spectacle frame, the message being associated with the generated value of the global criterion and the generated values of the set of specific criteria; and output the message to the user.

[0010] According to an embodiment, the disclosure further relates to a non-transitory computer-readable storage medium storing computer-readable instructions which, when executed by a computer, cause the computer to perform a method for providing a contextual assessment of a spectacle frame on a user's face, the method comprising: receiving user data describing features of the user's face; receiving device data describing features of the spectacle frame; generating, according to a first model, a value of a set of specific criteria describing a compatibility between the user's face and the spectacle frame based on the received user data and the received device data, the first model being trained to associate user data and device data with values of specific criteria; generating, according to a second model, a value of a global criterion based on the generated values of the set of specific criteria, the second model being trained to associate values of the specific criteria with values of the global criterion; determining a message about the user's face representing the spectacle frame, the message being associated with the generated value of the global criterion and the generated values of the set of specific criteria; and outputting the message to the user.

[0011] The foregoing paragraphs are provided as a general introduction to the application and are not intended to limit the scope of the claims below. The described embodiments and further advantages will best be understood by reference to the following detailed description considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0012] A more complete appreciation of the disclosure and its many attendant advantages will be readily understood by reference to the following detailed description considered in connection with the accompanying drawings, in which:

[0013] Figure 1 is an illustration of a user wearing a spectacle frame according to an exemplary embodiment of the disclosure;

[0014] Figure 2 is a flowchart of a method for providing a contextual assessment of eyewear frames according to example embodiments of the present disclosure;

[0015] Figure 3A is a flowchart of a method for providing a contextual assessment of eyewear frames according to example embodiments of the present disclosure;

[0016] Figure 3B is a flowchart of a method for providing a contextual assessment of eyewear frames according to example embodiments of the present disclosure;

[0017] Figure 4A is an illustration of an image of a user's face according to example embodiments of the present disclosure;

[0018] Figure 4B is an illustration of an image of an eyewear frame according to example embodiments of the present disclosure;

[0019] Figure 4C is an illustration of an image of a user wearing eyewear frames according to example embodiments of the present disclosure;

[0020] Figure 5 is a schematic of a database comprising user features, device features and corresponding images of users wearing devices according to example embodiments of the present disclosure;

[0021] Figure 6A is a flowchart of an aspect of a method for providing a contextual assessment of eyewear frames according to example embodiments of the present disclosure;

[0022] Figure 6B is an illustration of a survey conducted to eye care professionals according to example embodiments of the present disclosure;

[0023] Figure 7A is a flowchart of an aspect of a method for providing a contextual assessment of eyewear frames according to example embodiments of the present disclosure;

[0024] Figure 7B is an illustration of a metric for determining a specific criterion according to example embodiments of the present disclosure;

[0025] Figure 7C is an illustration of a metric for determining a specific criterion according to example embodiments of the present disclosure;

[0026] Figure 7D is an illustration of a metric for determining a specific criterion according to example embodiments of the present disclosure;

[0027] Figure 7E is an illustration of a metric for determining a specific criterion according to example embodiments of the present disclosure;

[0028] Figure 7F is a graph illustrating a metric for determining a certain criterion according to an exemplary embodiment of the present disclosure;

[0029] Figure 7G is a graph illustrating a metric for determining a certain criterion according to an exemplary embodiment of the present disclosure;

[0030] Figure 7H is a graph illustrating a metric for determining a certain criterion according to an exemplary embodiment of the present disclosure;

[0031] Figure 7I is a graph illustrating a metric for determining a certain criterion according to an exemplary embodiment of the present disclosure;

[0032] Figure 7J is a graph illustrating a metric for determining a certain criterion according to an exemplary embodiment of the present disclosure;

[0033] Figure 7K is a graph illustrating a metric for determining a certain criterion according to an exemplary embodiment of the present disclosure;

[0034] Figure 7L is a graph illustrating a metric for determining a certain criterion according to an exemplary embodiment of the present disclosure;

[0035] Figure 8A is a graphical representation of responses to a survey of eye care professionals according to an exemplary embodiment of the present disclosure;

[0036] Figure 8B is a graphical representation of estimated responses to a survey of eye care professionals according to an exemplary embodiment of the present disclosure;

[0037] Figure 9A is a flowchart of aspects of a method for providing a situational assessment of eyewear frames according to embodiments of the present disclosure;

[0038] Figure 9B is a flowchart of a decision tree of aspects of a method for providing a situational assessment of eyewear frames according to embodiments of the present disclosure;

[0039] Figure 10A is a flowchart of aspects of a method for providing a situational assessment of eyewear frames according to embodiments of the present disclosure;

[0040] Figure 10B is a flowchart of an annotated decision tree of aspects of a method for providing a situational assessment of eyewear frames according to embodiments of the present disclosure;

[0041] Figure 11 is a hardware configuration of a frame fit assessment device according to an exemplary embodiment of the present disclosure; and

[0042] Figure 12 is a flowchart of aspects of a method for providing a contextualized assessment of device data when other device data is provided. DETAILED DESCRIPTION

[0043] The terms "a" or "an", as used herein, are defined as one or more than one. The term "plurality", as used herein, is defined as two or more than two. The term "another", as used herein, is defined as at least a second or more. The terms "including" and / or "having", as used herein, are defined as "comprising" (i.e., open language). The terms "visual equipment", "equipment", "equipments", "eyeglass frame", "eyeglass frames", "eyeglass", "eyeglasses", and "visual equipments" can be used interchangeably to refer to a device having both a frame and lenses. The term "visual equipment" can be used to refer to a single visual equipment, while the term "visual equipments" can be used to refer to more than one visual equipment. References in the specification to "one embodiment", "certain embodiments", "an embodiment", "an implementation", "example", or similar terms, mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure. Similarly, the terms "face image" and "person's face image" are corresponding terms that can be used interchangeably. Thus, appearances of such phrases, or appearances of these phrases in various places in the specification are not necessarily all referring to one same embodiment. Additionally, particular features, structures, or characteristics can be combined in any suitable way in one or more embodiments.

[0044] Today, patients, users, or consumers seeking eyewear generally have little guidance as to what is optically appropriate and aesthetically pleasing. For some patients or users, cultural trends drive their decision. For other patients or users, the opinions of friends and family are paramount. For still other patients or users who prioritize ergonomics and visual acuity, the opinions of trained eye care professionals (ECPs) are necessary.

[0045] Currently, users can use methods that provide some, but not all, of the above-mentioned features. For example, one method describes an implementation of a decision tree for matching eyewear frames to morphological features detected from key feature points of an individual’s face, which determines a pair of eyewear that best matches the individual. In another method, a user questionnaire can be used to match a user’s style preferences to available eyewear frames. In either case, and as is generally the case, these methods provide the user with knowledge of whether a particular pair of eyewear is a good or bad fit. However, these methods do not provide context regarding the determination. For example, while these methods can be able to recommend eyewear frames to a user based on style preferences according to bestseller rankings, etc., the recommendation ultimately reflects a single ‘best fit’ metric. While based on the basic features of the eyewear and the user, the single ‘best fit’ metric provides an oversimplification of the eyewear ‘fit’ and fails to convey to the user the reasons why the recommended eyewear frame has ‘best fit’. In some cases, the oversimplification can be a quantitative measure of a global label or specific criteria based on the eyewear (e.g., between 1 and 10), leaving the interpretation of the measure to the ECP. In this way, while the user is provided with knowledge of whether a particular pair of eyewear is a good or bad fit, these methods fail to provide the user with a textual context regarding the reasons for making such a determination (e.g., reasons why the frame is a good or bad fit according to its particular features).

[0046] U.S. Patent Application Publication No. 2017 / 169501 describes a database comprising eyewear models, user face models, and eyewear fit assessment models based on a fit assessment matrix. While a fit output is provided, the eyewear fit assessment models only generate a single global metric in determining the fit of a particular eyewear to a user’s face.

[0047] According to embodiments, the present disclosure describes an apparatus, method, and computer-readable storage medium for providing a contextual assessment of eyewear frames on a user’s face.

[0048] In embodiments, the present disclosure provides an association of different textual descriptions for each of a subset of possible values adopted by criteria related to global or specific attributes of the suitability of eyewear with respect to a user’s face.

[0049] In embodiments, the present disclosure includes an automated diagnostic system for determining the fit between eyewear and a user. The automated diagnostic system can generate at least one fit metric value and an associated textual description explaining the reasons why the eyewear is a good or bad fit for the user’s face. In examples, the at least one fit metric and the associated textual description can be based on one or more photos of the user, user information including eyewear prescription, age, gender, etc., and device characteristics including size, color, material, etc.

[0050] In embodiments, the present disclosure relates to an apparatus, method, and computer readable storage medium for providing a determination of user data describing user facial features and a determination of device data describing eyewear frame features from at least one photo of the user wearing the eyewear. Digital image processing and other image processing methods can be used to separate the user features from the frame features. The user data can be morphological, structural, and aesthetic features of the user’s face. The device data can be features including the overall width of the device frame, the dimensions of individual aspects of the device frame (e.g., size A, size B, size D, etc.), the vertical thickness of the top of the device frame, the horizontal thickness of the device frame at the hinge layer, the color of the device, the material of the device, etc. The at least one photo of the user’s face can be produced with a 2D or 3D camera or other image capture device configured to acquire images of the user, eye wear, etc. At least one fitment metric value and an associated textual description explaining the reasons why the eyewear is or is not a fit for the user’s face can be generated from the determined user data and device data.

[0051] According to exemplary embodiments, the present disclosure describes a machine learning based frame fitment assessment apparatus (i.e., eyewear assessment tool) for presenting a fit eyewear selection to a user based on morphological and structural features (i.e., user features and device features), ophthalmic needs (i.e., visual prescription), and aesthetic appeal.

[0052] According to exemplary embodiments, the present disclosure includes a method for device recommendation based on (1) user features, device features, and positional data of wearing the device, (2) a set of specific and global standard values related to the fitment of the device, (3) a global standard based on a global score, rating, or acceptance of the fitment of the device, and (4) specific standards based on a score, rating, or acceptance of specific fitment parameters, where the global standard can be derived from the specific standards.

[0053] As introduced above, the present disclosure provides eyewear recommendations and assessments with textual context, allowing the user to understand the reasons for the device fit or misfit. The assessment can be based on standard values determined from models generated by machine learning based methods, including global and specific. Each of the machine learning based methods can include a dataset annotated by ECPs providing labels for each of the specific and global standards for corresponding pairs of user features and device features.

[0054] In embodiments, the present disclosure can provide global standards modified only by relevant specific standards.

[0055] In embodiments, the present disclosure can provide global standards modified only by relevant specific standards.

[0056] In embodiments, the first and second models can be updated based on ECP recommendations. The ECPs correlate the frame model or its geometric features with a user facial photo and declare a score to a central server via a network. As can be appreciated, the network can be a public network such as the Internet, or a private network such as a LAN or WAN network, or any combination thereof, and can also include PSTN or ISDN sub-networks. The network can also be wired (such as an Ethernet network), or can be wireless (such as a cellular network including EDGE, 3G, 4G, and 5G wireless cellular systems). The wireless network can also be WiFi, Bluetooth, or any other wireless communication form known. The models are updated in real time and other ECPs benefit from the updates when determining the fit between eyewear and a user.

[0057] In embodiments, a global standard can be derived from the values of the specific standards using a machine learning based method. The machine learning based method can include a linear combination or a non-linear combination of classification and regression trees and / or specific standard models.

[0058] According to embodiments, the present disclosure provides multi-standard ECP input and multiple related models based on machine learning. These machine learning based models are not limited to a global standard, but provide textual information about the specific standard that is considered to be most relevant to the overall ‘fit’ of the frame or device. For example, it can be determined that the device is not a good fit for the user and that the primary driver of this poor fit is the thickness of the device frame. Accordingly, the user can search for a similar relevant device with a reduced thickness of the frame in order to improve its overall ‘fit’.

[0059] According to embodiments, the present disclosure relates to an apparatus, method and computer readable storage medium for providing a contextualized assessment of device data when other device data is provided. InThis method is illustrated in Figure 12 . When a first set of device data is presented in the absence of device data, the method of the present disclosure compares said set of data to a set of data in a database comprising at least one set of device data. An average value of the missing data from frames with similar other features in the database is then calculated. For example, an average thickness of a frame can be calculated in the case where it is known that the frame is a plastic female frame. The database can provide a suggestion of a device with a missing data value equal to or close to the average value of the calculated missing data and with other values equal to or close to the other values of the first set of device data. The user can select a missing data value individually for each data or a tolerance value for one or several other data.

[0060] ​According to embodiments, the present disclosure describes an apparatus, method and computer readable storage medium for providing a contextualized assessment of the suitability of a device and a user's face. When presenting an image of a user wearing a device, the method of the present disclosure processes the image so that a context of features contributing to a global criterion (i.e. a specific criterion) can be provided for a global criterion (e.g. a generic suitability metric). In other words, the method herein can determine a good match of a device and a user's face, however, a recommendation can be qualified by stating that the good match of the device and the user's face is aesthetically suitable because the relative distance between the lens centers of the device and the interpupillary distance are suitable. Alternatively, in embodiments, a recommendation can be qualified by stating that the good match of the device and the user's face is aesthetically suitable because the distance between the lens centers is less than the interpupillary distance, the user's pupils are thereby positioned closer to the nose piece of the device.

[0061] According to embodiments, the multi-criteria ECP input and the plurality of machine learning based correlation models described above provide increasingly robust and accurate results for the contextualized global criterion.

[0062] Turning now to the drawings, Figure 1 is an illustration of a user wearing a spectacle frame according to an exemplary embodiment of the present disclosure. In the example, Figure 1 The illustration of The input of

[0063] may be applied to the method of the present disclosure as described in Figure 1 the embodiments of the method of the present disclosure. Figure 2 The input of

[0064] Figure 2 is a flowchart of a method for providing a contextual assessment of a spectacle frame according to an exemplary embodiment of the present disclosure. It can be appreciated that the method 200 can be performed by a frame suitability assessment apparatus comprising processing circuitry configured to perform the steps described herein. The frame suitability assessment apparatus will be described in more detail with reference to Figure 11 The frame suitability assessment apparatus will be described in more detail with reference to

[0065] At step 210 of the method 200, user data can be received. As discussed with reference to Figure 3A and Figure 3B The user data can be provided directly as user features or can be determined from an image containing a user's face.

[0066] At step 220 of the method 200, device data can be received. As discussed with reference to Figure 3A andFigure 3B The device data can be provided directly as device features or can be determined from images containing the device, as discussed.

[0067] At step 230 of the method 200, a value for the particular criteria can be generated by applying the user features and device features described above to a machine learning based model for the particular criteria. The value for the particular criteria metric can be based on a set of particular criteria metrics for the suitability of the device and the user face according to the particular modality, aesthetic, or visual consideration. In embodiments, the particular criteria can be a numerical and continuous quantity, such as a probability, a combination of quantities, a score, etc. For example, the particular criteria can define the inter-pupillary distance of the user. In embodiments, the particular criteria can be a qualitative quantity that can be defined by an alphanumeric value. For example, the qualitative quantity can represent an ECP’s assessment of the relative width of the device frame width to the user face width. The ECP can then decide whether the relative width is (a) too wide, (b) acceptable, or (c) too narrow. In another example, the qualitative quantity can represent an ECP’s assessment of the presence of the user’s eyebrows within the device frame. The ECP can then decide whether (a) the eyebrows are visible within the frame, (b) the eyebrows are positioned acceptably, or (c) the eyebrows are too high above the frame. This example merely represents various qualitative quantities related to frame suitability and that can be assessed by the ECP.

[0068] At step 240 of the method 200, a value for the global criteria can be generated by applying the generated value for the particular criteria to a machine learning based model for the global criteria. In embodiments, the value for the global criteria can be further based on the user features and device features defined above. The value for the global criteria can be a numerical or qualitative value indicative of the global suitability of the device to the user face. In embodiments, the machine learning based model can be used to generate the value for the global criteria from the generated value for the particular criteria. The machine learning based model can be a decision tree generated by classification and regression trees or can be a linear regression of the value for the particular criteria.

[0069] According to embodiments, in each of step 230 and step 240, the value for the particular criteria and the value for the global criteria can be determined according to a machine learning based method generated based on input from the ECP. For example, the machine learning based method can be based on ECP assessments and inputs for the value for the particular criteria and the global criteria for a given set of images of the wearable device, as will be described below.

[0070] At step 250 of method 200, the value of the particular criterion generated at step 230 and the value of the global criterion generated at step 240 can be evaluated to determine a message characterizing the suitability. In this way, the value of the global criterion can be contextualized by the value of the relevant particular criterion, providing a comprehensive, text and language based output as an alternative to value based output that lacks meaning. In an embodiment, the evaluation of the global criterion and the particular criterion can be performed by a decision tree, where the value of the particular criterion informs and contextualizes the global criterion found at the end of a branch of the decision tree. The global criterion (e.g., “suitable,” “not suitable”) can be provided with the relevant particular criterion (e.g., “color mismatch”) that forms the basis of the global criterion. The decision tree can be an annotated decision tree and include bifurcations defining different semantic text templates, each bifurcation or path defined by the evaluation of the value of the particular criterion and yielding a value of the global criterion. Each resulting value of the global criterion can then be described using descriptive terms that are contextualized by the features of the particular criterion and informed by the ECP that defines its path (as described in step 260). In an embodiment, the resulting text description can include a text translation of the bifurcation reason. Reference will now be made to Figure 10A and Figure 10B The annotated decision tree described above is further described.

[0071] At step 260 of method 200, the message determined at step 250 can be output to the user. In an embodiment, the determined message can be provided directly to the user as output. In another embodiment, the determined message can be modified according to an automated natural language generation tool to produce a more natural message according to the preferences and habits of the frame suitability evaluation device. For example, the modification can result in a given message being provided in multiple ways so as not to appear redundant and insincere when different devices follow the same bifurcation of the tree. In an example, the result can be a paraphrasing of the text message or a different contextualization of the value of the particular criterion to the value of the global criterion. Instead of defining the value of the global criterion by the size of the device relative to the user’s face, the value of the global criterion can be defined as the value of the particular criterion according to the color of the device relative to the user’s skin tone. In an embodiment, the automated natural language generation tool can take into account the influence of certain particular criteria on the value of the global criterion such that when providing an alternative human form explanation of the value of the global criterion, the textual context can be meaningful. For example, the color of the device and the size of the device can be particular criteria that have equal influence on the value of the global criterion and thus can be used interchangeably by the automated natural language generation tool.

[0072] In an embodiment, the message can be output to the user in a variety of ways including audio, video, haptics, etc. In an example, the message can be delivered to the user in a spoken manner.

[0073] Referring now to each step of method 200,Figure 3A and Figure 3B Steps 210 and 220 are described in further detail. Figure 3A and Figure 3B Each is a flowchart of a method for providing a contextualized assessment of eyewear frames according to exemplary embodiments of the present disclosure.

[0074] With respect to Figure 3A Steps 210 of method 200 and steps 220 of method 200 are performed simultaneously and each on separate images and / or data. For example, a user can provide an image of their face and wish to request a contextualized assessment of a hypothetical match between the image and a selected device from a database of devices.

[0075] Accordingly, at step 311 of method 201, user features can be extracted at step 312 from a user image of the user face or from user features stored within a user feature database. The user feature database can include a user profile associated with the user in which user features have been inputted for storage and access by the user. The user features can be morphological features, structural features, visual prescriptions, and aesthetic features of the user face. At step 321 of method 201, device features can be extracted at step 322 from a device image of the device or from device features stored within a device feature database. The device feature database can include device features associated with a plurality of devices stored in an online inventory, each set of device features defining structural features, aesthetic features, and visual features of the device. Alternatively, as suggested, user features and device features can be extracted at steps 311 and 321 from the respective images.

[0076] Similar to Figure 2 method 200, the user features and device features extracted at steps 311 and 321 can be applied to a model of a particular criterion at step 331 of method 201. By applying the model on the user features and device features, a value of a particular criterion metric can be generated according to a set of particular criterion metrics for suitability of the device with respect to the user face. Suitability of the device can be defined according to particular morphological, aesthetic, or visual considerations. In embodiments, the particular criterion can be a numerical and continuous quantity, such as a probability, a combination of quantities, a score, and the like. In embodiments, the particular criterion can be a qualitative quantity that can be defined by an alphanumeric value.

[0077] Similar to Figure 2The value of the particular criterion determined at step 331 of the method 200 can be applied to a model of the global criterion at step 341 of the method 201. The resulting value of the global criterion can be a numerical or qualitative value indicative of the global suitability of the device to the user’s face. In embodiments, a machine learning based model can be used to generate the value of the global criterion from the generated value of the particular criterion. The machine learning based model can be a decision tree generated by a classification and regression tree or can be a linear regression of the value of the particular criterion.

[0078] In examples where the value of the particular criterion and the value of the global criterion have been generated via an annotated decision tree, the method 201 can return to step 301 of the method 200. Figure 2 The method 200 can then return to step 250 of the method 200 where a message can be determined and output to the user at steps 250 and 260 respectively.

[0079] With respect to Figure 3B Steps 210 of the method 200 and step 220 of the method 200 are performed simultaneously and each on an image and / or data reflecting an image of a face of a user of a fitting device. For example, a user can provide an image of a face of their fitting device and can wish to request a contextualized evaluation of the ‘suitability’ therebetween. It can be appreciated that this can be relevant when a user is in a retail store and is trying on various devices, or when a user is shopping online and is virtually ‘trying on’ various devices.

[0080] Thus, at step 311 of the method 201, user features can be extracted from the user image of the face of the user of the fitting device at step 302. The user features can be morphological, structural and aesthetic features of the face of the user. At step 321 of the method 201, device features can be extracted from the user image of the face of the user of the fitting device at step 302. The device features can define structural, aesthetic and visual features of the device.

[0081] Similar to the method 200 of Figure 2 The user features and device features extracted at steps 311 and 321 can be applied to a model of the particular criterion at step 331 of the method 201. By applying the model to the user features and device features, a value of the particular criterion measure can be generated according to a set of particular criterion measures of suitability for the device and the face of the user. The suitability of the device can be defined according to particular morphological, aesthetic or visual considerations. In embodiments, the particular criterion can be a numerical and continuous quantity, such as a probability, a combination of quantities, a score, etc. In embodiments, the particular criterion can be a qualitative quantity that can be defined by an alphanumeric value.

[0082] Similar to the method 200 of Figure 2In method 200, the value of the specific criterion determined at step 331 of method 201 can be applied to the global criterion model at step 341 of method 201. The resulting global criterion value can be a numerical or qualitative value indicating the global suitability of the device for the user's face. In an embodiment, a machine learning-based model can be used to generate the global criterion value based on the generated value of the specific criterion. The machine learning-based model can be a decision tree generated from classification and regression trees or a linear regression of the value of the specific criterion.

[0083] In the example, values ​​for specific criteria and global criteria have been generated via an annotated decision tree, and method 201 can return to... Figure 2 Method 200, wherein the message can be determined and output to the user at steps 250 and 260 respectively.

[0084] In view of the above, Figure 4A , Figure 4B and Figure 4C express Figure 2 Possible inputs to method 200. In the example, the user can provide a user image 412 independently of the device. In this way, the user can select any one of multiple images of the device 422, which can be evaluated to provide a contextualized assessment of the user's face and the device. Alternatively, or as a virtual 'try-on', the user can provide a combined image 402 of the user and device image 412. In any of the cases presented above, the images can be processed to extract user features and device features.

[0085] The above description focuses on the flowchart experienced by the end user. In an exemplary embodiment of this disclosure, the end user can provide an image of their face while wearing the device. To allow this operation, specific standard models and global standard models must be developed.

[0086] refer to Figure 5 A specific standard model can be initially developed by generating a database of user features, device features, and corresponding images of users wearing the device. For this purpose, the original database A533 may include multiple datasets, which include user features 511, device features 521, and corresponding images 502 of users wearing the device. The corresponding image 502 of the user wearing the device may be one image or multiple images. In an embodiment, the multiple datasets of the original database A533 may be sufficient to generate an accurate model of the specific standard while minimizing the computational burden.

[0087] According to example embodiments, user features 511 can be extracted from corresponding images 502 of a user wearing a device, from a user image without a device, from morphological data associated with the user, etc. Images can be acquired using two-dimensional imaging devices or three-dimensional imaging devices.

[0088] According to example embodiments, device features 521 can be extracted from corresponding images 502 of a user wearing a device, from a device image without a user face, from structural data associated with a three-dimensional rendering of a device according to a device design, from structural data acquired through measurements using a frame tracking device, or from another source. Images can be acquired using two-dimensional imaging devices or three-dimensional imaging devices.

[0089] According to example embodiments, user images 502 of a wearing device can be images of a real user wearing a device, images of a real user wearing a virtual 'try-on' device, images of a virtual user (i.e., avatar) wearing a virtual 'try-on' device, etc. From which user features 511 and device features 521 can be acquired according to the techniques described above.

[0090] Returning to method 200, specific standard models and global standard models can be developed based on user features and device features in consultation with ECPs. Specific standard models and global standard models can be based on ECPs' evaluation of images of a user's face wearing a device. As Figure 6B shown, by completing an ECP survey, an ECP can evaluate an image of a user's face wearing a device according to a set of specific standards related to various aspects of the 'combined' image and according to global standards.

[0091] As an overview of the ECP evaluation process, and with reference to Figure 6A , a raw dataset A 633 comprising images 602 of a user's face wearing a device can be evaluated by a plurality of ECPs 634 according to an ECP survey. As Figure 6B shown, the ECP survey can comprise specific standard survey questions and global standard survey questions. For example, the survey can comprise the following specific standards:

[0092] Specific Standard 1 : Provide feedback on the width of the device relative to the size of the user's head. A high negative score indicates that the device width is too small, while a high positive score indicates that the device width is too large.

[0093] Specific Standard 2: Provide feedback on the position of the pupils relative to the shape of the device / lenses. A high negative score indicates that the pupils are too close to the nasal portion of the device / lenses, while a high positive score indicates that the pupils are too close to the temporal portion. Not included in the ECP survey, but it should be noted that, in general, the pupils are preferably slightly closer to the nasal portion.

[0094] Particular criterion 3: provide feedback on the horizontal position of the external part of the eyebrow with respect to the shape of the device.

[0095] Particular criterion 4: provide feedback on the vertical position of the eyebrow with respect to the top aspect of the device.

[0096] Particular criterion 5: provide feedback on the vertical position of the bottom aspect of the device with respect to the cheeks of the user.

[0097] Particular criteria 3, 4 and 5 can be evaluated on a similar continuum of negative and positive values.

[0098] Particular criterion 6: provide feedback on the size of the bridge of the nose of the device with respect to the width of the nose of the user. A high negative score indicates that the bridge of the nose is too narrow, while a high positive score indicates that the bridge of the nose is too wide.

[0099] The results of each ECP investigation associated with a dataset of corresponding images of the user features, the device features and the user wearing the device can be stored in an annotated database A together with the dataset, which will be referred to with reference to Figure 7A Further details are discussed. Thus, the results of each ECP investigation contribute to a model describing the value of the particular criteria and a model describing the value of the global criteria, wherein the models are based on the evaluation by the ECP of the various aspects of the suggested combination of the user face and the device.

[0100] In embodiments, the results of each ECP investigation can be used to provide constraints for mathematical expressions, equations and inequality conditions and logical conditions defining each of the particular criteria models and the global criteria models as a function of the user features and the device features. Coefficients and parameters in such expressions can be obtained using machine learning based tools, as will be discussed.

[0101] For example, for a given set of particular criteria (Sc1, Sc2,..., SC N ), such expressions can be based on user features including the width of the temporal part of the user face, the width of the sphenoid bone of the user face, the horizontal position of the barycenter of the user eyelid opening, the average vertical position of the user eyebrows, the length of the user nose, etc., and device features including the total width of the device frame, the dimensions of the various aspects of the device frame (e.g. size A, size B, size D, etc.), the vertical thickness of the top aspect of the device frame, the horizontal thickness of the device frame at the hinge level, the color of the device, the material of the device, etc. In one expression related to Sc1, the expression can be written as

[0102] Sc1 = too small if (temporal part width - total device width

[0103] > 20 mm) and too big if (temporal part width - total device width

[0104] ≤ 20 mm and temporal part width - total device width

[0105] Too long if (sphenoid width - (2A + D) > 20mm

[0106] In one expression related to Sc2, the expression can be written as

[0107] Sc2 = Too small if (sphenoid width - (2A + D) > 20mm

[0108] Too long if (sphenoid width - (2A + D) > 20mm

[0109] OK if (sphenoid width - (2A + D) > 20mm and sphenoid width - (2A + D) > -20mm

[0110] Too long otherwise

[0111] where A denotes 'Size A', a dimension of the device frame, and B denotes 'Size B', a dimension of the device frame.

[0112] In one expression related to Sc3, the expression can be written as

[0113] Sc3 = Too high if (max(Y_eyebrow)

[0114] - (device frame's Y 顶部 ) > 10mm)

[0115] eyebrow is too high above the device frame, if max(Y_eyebrow)

[0116] - (device frame's Y 顶部 <= -5mm) then eyebrow is observable within the frame, otherwise OK

[0117] In the above examples, the thresholds defining, for example, the boundaries between too small, OK, and too large, can be determined by a machine learning based method. The thresholds can be determined depending on the answer scale used. For example, the answer scale can be a text bin describing three possible modes or a number bin describing a suitability level between -5 and +5. In embodiments, the thresholds can be defined by a machine learning based method applied to the results of the ECP survey shown in Table 1. It can also be understood that the thresholds can depend on the relationships and interactions between the specific criteria, where, for example, Sc1 can influence Sc3. Figure 6B

[0118] Alternatively, and as will be described more in reference to Figure 8A and Figure 8B The specific criteria models and global criteria models can also be developed according to statistical methods or other machine learning based methods applied to the results of the ECP survey with respect to the specific criteria and global criteria for a given user dataset. For example, such statistical methods can include linear discriminant analysis. ​

[0119] To this end, and according to exemplary embodiments, the ECP survey can be directed to a dataset acquired from the original database A 533. The ECP survey can comprise a series of images 502 of the user wearing the device, and next to each image, a series of questions about specific points of suitability of the device and the person’s face. For each question, a limited number of possible answers can be provided to the ECP. In examples, these limited number of possible answers can be a scale between -5 and +5, a scale between 0 and 10, or a selection of an item among a set of N items. Exemplary questions and answers as submitted to the ECP during the completion of the ECP survey are described below.

[0120] Question 1. How to evaluate the width of the device relative to the width of the user’s face? (a) too small, (b) can, or (c) too big.

[0121] Question 2. How to evaluate the user’s pupils relative to the aperture of the device? (a) too internal, (b) can, or (c) too external.

[0122] Question 3. How to see the position of the user’s outer eyebrow corners relative to the aperture of the device? (a) too internal, (b) can, or (c) too external.

[0123] Question 4. How to evaluate the position of the top of the device’s frame relative to the user’s eyebrows? (a) too low, (b) can, or (c) too high.

[0124] Question 5. How to evaluate the position of the bottom of the device’s frame relative to the user’s cheeks? (a) too low, (b) can, or (c) too high.

[0125] Question 6. How to evaluate the bridge of the device’s frame relative to the wearer’s nose? (a) too narrow, (b) can, or (c) too wide.

[0126] The above exemplary questions provide an introduction to the myriad of features that can be considered during the development of a specific standard model, as referenced in Figure 7B to Figure 7L will be shown. Moreover, although described above with reference to a specific standard model, a similar approach can be readily implemented when developing a global standard model.

[0127] Thus, reference is made to Figure 7A A flowchart of the generation of a specific standard model and a global standard model will now be described. In particular, Figure 7A depicts the submission of the annotated database A 736 of user datasets to a first machine learning method 704 in order to generate a specific standard model 737. In embodiments, the submission of the annotated database A 736 of user datasets to the first machine learning method 704 can also generate a global standard model 747.

[0128] Annotated database A 736 includes Figure 6A User features 711 and device features 721 of the original database A as Figure 6B illustrated are acquired corresponding ECP survey results 735. The dataset of annotated database A 736 can then be provided to the first machine learning method 704.

[0129] According to embodiments, the first machine learning method 704 can be a linear discriminant analysis or similar method for determining a linear combination of features that characterizes or separates multiple classes of objects or events, including neural networks and the like.

[0130] In an example, where a particular standard model 737 is being generated, the first machine learning method 704 can be a linear discriminant analysis (LDA), where the first machine learning method 704 attempts to explain the ECP survey results 735 by identifying statistical laws that relate the ECP survey results 735 to the user features 711 and device features 721. For example, an LDA according to the ECP survey results 735 can support a law that gives the probabilities (p a , p b , p c )i (i = 1, 6) that a corresponding user and device are in a state a, b, c for a given question i. The probabilities can be rewritten as (p a + p b + p c = 1, where 0 ≤ p a ≤ 1, 0 ≤ p b ≤ 1, and 0 ≤ p c ≤ 1), where the values of (p a , p b , p c ) i describe the particular standard of the model. It can be appreciated that similar methods can be used to determine a global standard model.

[0131] Having applied the first machine learning method 704 to the user dataset of annotated database A 736, the output of the first machine learning method 704 can be understood as the particular standard model 737 or the global standard model 747, as the case can be.

[0132] Figure 7B to Figure 7L A graphical illustration of representative measures of user features and device features that contribute to a particular standard model and a global standard model are provided.

[0133] Figure 7B is a graphical illustration of representative measures of user's temporal width values according to example embodiments of the present disclosure.

[0134] Figure 7Cis an illustration of a representative measurement of a user's inner eye corner distance value according to example embodiments of the present disclosure.

[0135] Figure 7D is an illustration of a representative measurement of a user's outer eye corner distance value according to example embodiments of the present disclosure.

[0136] Figure 7E is an illustration of a representative measurement of a user's nose length value according to example embodiments of the present disclosure.

[0137] Figure 7F is an illustration of a representative measurement of a user's nose length value as a base and original of morphological features according to example embodiments of the present disclosure.

[0138] Figure 7G is an illustration of a representative measurement of a user's right eyebrow's maximum height value according to example embodiments of the present disclosure.

[0139] Figure 7H is an illustration of a representative measurement of a user's right eyebrow's average height value according to example embodiments of the present disclosure.

[0140] Figure 7I is an illustration of a representative measurement of a distance value between inner eye corners of a user according to example embodiments of the present disclosure.

[0141] Figure 7J is an illustration of a representative measurement of a distance value between lateral eye corners of a user according to example embodiments of the present disclosure.

[0142] Figure 7K is an illustration of a representative measurement of a value of a left pupil distance and a right pupil distance of a user's eyes according to example embodiments of the present disclosure.

[0143] Figure 7L is an illustration of a representative measurement of a distance value between inner eye corners of a user according to example embodiments of the present disclosure.

[0144] As introduced above, the relationship between ECP survey results, user features, and device features can be defined according to an implementation of the first machine learning method or LDA in the example. Thus, when applied to an unknown user dataset comprising user features and device features, LDA training will ensure its accurate classification. Thus, Figure 8A is a graphical representation of responses to an ECP survey to be used as training data according to example embodiments of the present disclosure.

[0145] According to embodiments, can be understood in view of a specific question from the ECP survey, such as "How to evaluate the width of the device relative to the width of the user's face?" Figure 8AThe response to the problem can be represented numerically as (0) too small, (1) okay, or (2) too large. Figure 8A This shows the responses to the questions considered as training data. Each response, indicated by a number, represents an evaluation of the question, as each response relates to an image of a user with the corresponding device. The horizontal axis is the first discriminant axis (LD1) and the vertical axis is the second discriminant axis (LD2). Each is a linear combination of the variables used in the model. Figure 8A In the example, the linear combination can be written as LD1 = 0.5291134 * size A + 0.4516208 * size D - 0.1110664 * temporal width - 0.1571796 * hinge thickness - 0.0956928 * sphenoid bone width and LD2 = 0.05257000 * size A + 0.64085819 * size D + 0.12367100 * temporal width - 0.09174694 * hinge thickness - 0.15042168 * sphenoid bone width, where, as Figure 7B to Figure 7L The description defines 'size A', 'size D', 'temporal width', 'hinge thickness', and 'sphenoid bone width'.

[0146] like Figure 8A As shown, and as defined by the above expressions, LD1 and LD2 provide the maximum interval between the answers to the survey question: (0) too small, (1) okay, and (3) too large.

[0147] LDA has been trained based on ECP survey results, and the expressions for LD1 and LD2 that best separate their responses have been identified. LDA can be applied to unclassified user features and device features. Therefore, Figure 8B This is a graphical representation of the estimated response to a survey conducted on eye care professionals according to exemplary embodiments of the present disclosure. (As shown in...) Figure 8B In this context, LDA provides the conditional probabilities that a device's frame is classified as (0) too small, (1) okay, or (2) too large for the user's face. Figure 8B The data shown reflects the maximum conditional probability.

[0148] Now for reference Figure 9A The flowchart above illustrates how the specific standard model and the global standard model can be applied to the context of a new database for the user dataset in order to generate relationships between the global standard and the specific standard. Figure 9A The output of the flowchart can be a decision tree populated with specific criteria, and each branch of the decision tree ends with a global criterion, such that the global criterion can be interpreted from the values ​​of the specific criteria that populate the same branch of the decision tree.

[0149] In particular, the user dataset from original database B 943 having a structure similar to the structure of original database A and annotated database A can be submitted to the specific standard model 937 and the global standard model 947. The output of each of the specific standard model 937 and the global standard model 947 can then be provided to the second machine learning method 905 in order to generate a decision tree 944. The decision tree 944 can reflect the relationship between the output of the specific standard model 937 and the output of the global standard model 947 as determined by the second machine learning method 905. In an embodiment, the second machine learning method 905 can be a classification and regression tree. For example, a classification tree can be implemented if each specific standard has only a few discrete modes (e.g., “too small,” “ok,” “too big”). In another example, a regression tree can be implemented if each standard can be described by a continuum (e.g., “-10 to +10”).

[0150] Figure 9B is an exemplary decision tree determined from the output of the specific standard model and the global standard model and applied to the second machine learning method. It can be appreciated that each branch of the decision tree 944 can yield a value of the global standard 942, wherein the value of the global standard 942 is defined by the value of the specific standard 932 that fills the same branch of the decision tree 944.

[0151] Reference is now made to Figure 10A , Figure 9A and Figure 9B The decision tree generated in Figure 9A may be further processed in order to provide a textual context regarding the suitability of the device with respect to the user’s face. To this end, the decision tree 1044 generated by the second machine learning method of Figure 10B may be annotated by the ECP, the ECP annotation 1045 thereby contextualizing the decision tree 1044 and generating an annotated decision tree 1046, as shown in

[0152] In other words, while existing methods can provide a value of the global standard that is located on a scale of one to ten, one being a poor suitability and ten being a good suitability, the present disclosure provides mechanisms by which the value of the global standard can be defined from the value of the specific standard generated based on the user’s face and the device being evaluated.

[0153] In an embodiment, and as in Figure 10B , the annotated decision tree as in Figure 9BThe annotated decision tree 1046 can provide a contextualized assessment as a fit message. The fit message can then be output to the user to provide a frame fit assessment, as previously described.

[0154] In embodiments, and with reference again to the method 200 of Figure 2 the annotated decision tree 1046 can provide a contextualized assessment as a fit message. The fit message can then be output to the user to provide a frame fit assessment, as previously described.

[0155] According to embodiments, the method 200 of the present disclosure allows the user to understand how the device fits on their face and the reasons for it. To this end, the user can provide an image of themselves wearing the device (i.e. eyeglasses frame), and the user features and device features can be computed therefrom. The user features and device features can then be applied to the specific criteria models developed above to determine the values of the specific criteria. The values of the specific criteria are applied to the annotated decision tree of Figure 10B the annotated decision tree can allow the determination of global fit criteria, wherein the path of the annotated decision tree provides context to the user in the form of a fit message.

[0156] Reference is now made to the following drawings in which: Figure 11 , Figure 11 is a hardware description of a frame fit assessment device according to exemplary embodiments of the present disclosure.

[0157] In Figure 11 , the frame fit assessment device includes a CPU 1185 that executes the processes described above. The frame fit assessment device can be a general purpose computer or a specific special purpose machine. In one embodiment, the frame fit assessment device becomes a specific special purpose machine when the processor 1185 is programmed to perform the visual device selection (and in particular, any of the processes discussed with reference to the above disclosure).

[0158] Alternatively, or additionally, the CPU 1185 can be implemented on an FPGA, ASIC, PLD, or using discrete logic circuitry, as would be understood by one of ordinary skill in the art. Further, the CPU 1185 can be implemented as multiple processors working in parallel, to execute the instructions of the processes of the present disclosure described above.

[0159] The frame suitability assessment device also includes a network controller 1188, such as an Intel Ethernet PRO network interface card, to interface to a network 1199. As can be appreciated, the network 1199 can be a public network, such as the Internet, or a private network, such as an LAN or WAN network or any combination thereof and can also include PSTN or ISDN sub-networks. The network 1199 can also be wired, such as an Ethernet network, or can be wireless such as a cellular network including EDGE, 3G, and 4G wireless cellular systems. The wireless network can also be WiFi, Bluetooth, or any other wireless communication means.

[0160] The frame suitability assessment device further includes a display controller 1189, such as a graphics card or a graphics adapter, to interface to a display 1190, such as a monitor. A general purpose I / O interface 1191 interfaces to a keyboard and / or mouse 1192 and to a touch panel 1193 on or separate from the display 1190. The general purpose I / O interface 1191 also connects to various peripheral devices 1194, such as printers and scanners. In embodiments of the disclosure, the peripheral devices 1194 can include a 2D or 3D camera, or other image capture device configured to acquire images of a user, eyewear, etc.

[0161] A sound controller 1195 is also provided in the frame suitability assessment device to interface to speakers / microphones 1196 to provide sound and / or music.

[0162] A general storage controller 1197 connects a storage media disk 1187 to a communication bus 1198 to interconnect all of the components of the frame suitability assessment device, which can be an ISA, EISA, VESA, PCI, etc. A description of the general features and functions of the display 1190, keyboard and / or mouse 1192, and display controller 1189, storage controller 1197, network controller 1188, sound controller 1195, and general purpose I / O interface 1191 are omitted for brevity, as these features are known.

[0163] The exemplary circuit elements described in the context of the present disclosure can be replaced with other elements and constructed in a different manner than the examples provided herein. Moreover, the circuitry configured to perform the features described herein can be implemented in multiple circuit units (e.g., chips), or the features can be combined in circuitry on a single chip set.

[0164] The functions and features described herein can also be performed by various distributed components of a system. For example, one or more processors can perform these system functions, where the processors are distributed across multiple components in communication with a network. Distributed components can include one or more client and server machines that can share processing, in addition to including various human-interactive and communication devices (e.g., display monitors, smart phones, tablet computers, personal digital assistants (PDAs)). The network can be a private network such as a LAN or WAN, or can be a public network such as the Internet. Input to the system can be received via direct user input, and can be received remotely, in real-time or as batch processing. Additionally, some implementations can be performed on different modules or hardware than those described. Accordingly, other implementations are within the scope of what can be claimed.

[0165] Figure 12 is a flowchart of a method for providing a contextualized assessment of device data when other device data is provided. When a first set of device data is presented in the absence of device data, the set of data is compared to a set of data in a database comprising at least one set of device data. An average of the missing data from frames with similar other characteristics in the database is then calculated. The database then provides a suggestion of a device having missing data values equal to or close to the calculated average of the missing data and having other values equal to or close to the other values of the first set of device data.

[0166] It will be apparent that many modifications and variations can be possible according to the above teachings. It is, therefore, to be understood that within the scope of the appended claims, the application can be practiced otherwise than as specifically described herein.

[0167] Embodiments of the present disclosure can also be described as follows in brackets.

[0168] (1) A method for providing a contextual assessment of eyewear frames on a user's face, the method comprising: receiving user data describing characteristics of the user's face; receiving device data describing characteristics of the eyewear frames; generating, according to a first model, values of a set of specific criteria describing compatibility between the user's face and the eyewear frames based on the received user data and the received device data, the first model trained to associate user data and device data with values of specific criteria; generating, by processing circuitry and according to a second model, values of global criteria based on the generated values of the set of specific criteria, the second model trained to associate values of the specific criteria with values of global criteria; determining a message about the user's face characterizing the eyewear frames, the message associated with the generated values of the global criteria and the generated values of the set of specific criteria; and outputting the message to the user.

[0169] (2) The method of (1), wherein the outputting outputs the message to the user by applying a natural language generator to the determined message.

[0170] (3) The method of (1) or (2), wherein the received user data is based on an image of the user’s face.

[0171] (4) The method of any one of (1) to (3), wherein the received device data is based on an image of the eyewear frame.

[0172] (5) The method of any one of (1) to (4), wherein the first model is generated by applying a first machine learning to a database comprising user data, device data, and images of faces of users wearing eyewear frames, the user data and the device data in the database being associated with respective ones of the images of faces of users wearing eyewear frames in the database, the first machine learning being trained to associate the user data and the device data in the database with reference values of a particular criterion and reference values of a global criterion.

[0173] (6) The method of any one of (1) to (5), wherein the reference values of the particular criterion and the reference values of the global criterion are determined by human evaluation of the images of faces wearing eyewear frames in the database.

[0174] (7) The method of any one of (1) to (6), wherein the human evaluation is performed by an eye care professional.

[0175] (8) The method of any one of (1) to (7), wherein the first machine learning is linear discriminant analysis.

[0176] (9) The method of any one of (1) to (8), wherein the second model is generated by applying a second machine learning to reference values of a particular criterion and reference values of a global criterion, the second machine learning being trained to associate the reference values of the particular criterion with the reference values of the global criterion.

[0177] (10) The method of any one of (1) to (9), wherein the second model is a decision tree.

[0178] (11) An apparatus for providing a contextual assessment of a spectacle frame on a user’s face, the apparatus comprising processing circuitry configured to: receive user data describing features of the user’s face; receive device data describing features of the spectacle frame; determine, from a first model, a value of a set of specific criteria describing compatibility between the user’s face and the spectacle frame based on the received user data and the received device data, the first model trained to associate user data and device data with values of specific criteria; generate, from a second model, a value of a global criterion based on the generated value of the set of specific criteria, the second model trained to associate values of the specific criteria with values of the global criterion; determine a message about the user’s face representing the spectacle frame, the message associated with the generated value of the global criterion and the generated value of the set of specific criteria; and output the message to the user.

[0179] (12) The apparatus of (11), wherein the first model is generated by applying a first machine learning to a database comprising user data, device data, and images of user’s faces wearing spectacle frames, the user data and the device data in the database being associated with respective ones of the images of user’s faces wearing spectacle frames in the database, the first machine learning trained to associate the user data and the device data in the database with reference values of specific criteria and reference values of a global criterion.

[0180] (13) The apparatus of (11) or (12), wherein the reference values of specific criteria and the reference values of the global criterion are determined by human evaluation of the images of faces wearing spectacle frames in the database, the human evaluation being performed by eye care professionals.

[0181] (14) The apparatus of any one of (11) to (13), wherein the second model is generated by applying a second machine learning to reference values of specific criteria and reference values of a global criterion, the second machine learning trained to associate the reference values of specific criteria with the reference values of the global criterion.

[0182] (15) A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a computer, cause the computer to perform a method for providing a contextual assessment of a spectacle frame on a user's face, the method comprising: receiving user data describing features of the user's face; receiving device data describing features of the spectacle frame; generating, from a first model, a value of a set of specific criteria describing compatibility between the user's face and the spectacle frame based on the received user data and the received device data, the first model trained to associate user data and device data with values of specific criteria; generating, from a second model, a value of a global criterion based on the generated value of the set of specific criteria, the second model trained to associate values of the specific criteria with values of the global criterion; determining a message about the user's face characterizing the spectacle frame, the message associated with the generated value of the global criterion and the generated value of the set of specific criteria; and outputting the message to the user.

[0183] Thus, the foregoing discussion discloses and describes exemplary embodiments of the present application. As will be understood by those skilled in the art, the present application can be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, the disclosure of the present application is intended to be illustrative, but not limiting, of the scope of the application and the other claims. This disclosure, including any readily discernable variants of the teachings herein, defines the scope of the foregoing claims, so that no inventive subject matter is devoted to the public.

Claims

1. A method for providing a contextual assessment of an eyeglass frame on a user’s face when incomplete device data is provided, the method comprising: receiving user data describing features of the user’s face; receiving a set of device data describing features of the eyeglass frame, wherein a portion of the features of the eyeglass frame have been provided for the eyeglass frame in the set of device data, and another portion of the features of the eyeglass frame have not been provided for the eyeglass frame in the set of device data; comparing the set of device data to a set of data in a database comprising at least one set of device data; computing an average of data from eyeglass frames in the database having similar features corresponding to the other portion of the features of the eyeglass frame that have not been provided for the eyeglass frame in the set of device data, as if the other portion of the set of device data had been provided for the eyeglass frame; generating, according to a first model, a value of a set of specific criteria describing compatibility between the user’s face and the eyeglass frame based on the received user data, the received device data, and the computed average, the first model trained to associate user data and device data with values of specific criteria; generating, by processing circuitry and according to a second model, a value of a global criterion based on the generated value of the set of specific criteria, the second model trained to associate values of the specific criteria with values of the global criterion; determining a message about the user’s face representing the eyeglass frame, the message associated with the generated value of the global criterion and the generated value of the set of specific criteria; and outputting the message to the user.

2. The method of claim 1, wherein, the outputting outputs the message to the user by applying a natural language generator to the determined message.

3. The method of claim 1, wherein, the received user data is based on an image of the user’s face.

4. The method of claim 1, wherein, the received device data is based on an image of the eyeglass frame.

5. The method of claim 1, wherein, the first model is generated by applying a first machine learning to a database comprising user data, device data, and images of user’s faces wearing eyeglass frames, the user data and the device data in the database associated with respective ones of the images of user’s faces wearing eyeglass frames in the database, the first machine learning trained to associate the user data and the device data in the database with reference values of specific criteria and reference values of a global criterion.

6. The method of claim 5, wherein, the reference values of specific criteria and the reference values of the global criterion are determined by human evaluation of the images of faces wearing eyeglass frames in the database.

7. The method of claim 6, wherein, the human evaluation is performed by eye care professionals.

8. The method of claim 5, wherein, the first machine learning is linear discriminant analysis.

9. The method of claim 1, wherein, the second model is generated by: applying a second machine learning to reference values of specific criteria and reference values of a global criterion, the second machine learning trained to associate the reference values of specific criteria with the reference values of the global criterion.

10. The method of claim 6, wherein, the second model is a decision tree.

11. An apparatus for providing a contextual assessment of an eyeglass frame on a user’s face when incomplete device data is provided, the apparatus comprising: processing circuitry configured to: receive user data describing features of the user's face, receive a set of device data describing features of the eyewear frame, wherein a portion of the features of the eyewear frame has been provided for the eyewear frame in the set of device data, and another portion of the features of the eyewear frame is not provided for the eyewear frame in the set of device data, compare the set of device data to a set of data in a database comprising at least one set of device data; compute an average of data from frames with similar features in the database corresponding to the other portion of the features of the eyewear frame in the set of device data that is not provided for the eyewear frame, as if the other portion of the set of device data had been provided for the frame; determine, according to a first model, a value of a set of specific criteria describing compatibility between the user's face and the eyewear frame based on the received user data, the received device data, and the computed average, the first model being trained to associate user data and device data with values of specific criteria, generate, according to a second model, a value of a global criterion based on the generated value of the set of specific criteria, the second model being trained to associate values of the specific criteria with values of the global criterion, determine a message about the user's face representing the eyewear frame, the message being associated with the generated value of the global criterion and the generated value of the set of specific criteria, and output the message to the user.

12. The apparatus of claim 11, wherein, the first model is generated by: applying a first machine learning to a database comprising user data, device data, and images of user's faces wearing eyewear frames, the user data and the device data in the database being associated with respective ones of the images of user's faces wearing eyewear frames in the database, the first machine learning being trained to associate the user data and the device data in the database with reference values of specific criteria and reference values of a global criterion.

13. The apparatus of claim 12, wherein, the reference values of specific criteria and the reference values of the global criterion are determined by human evaluation of the images of faces wearing eyewear frames in the database, the human evaluation being performed by eye care professionals.

14. The apparatus of claim 11, wherein, the second model is generated by: applying a second machine learning to reference values of specific criteria and reference values of a global criterion, the second machine learning being trained to associate the reference values of specific criteria with the reference values of the global criterion.

15. A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a computer, cause the computer to perform a method for providing a contextual assessment of an eyewear frame on a user's face when incomplete device data is provided, the method comprising: receiving user data describing features of the user's face, receiving a set of device data describing features of the eyeglasses frame, wherein a portion of the features of the eyeglasses frame have been provided for the eyeglasses frame in the set of device data and another portion of the features of the eyeglasses frame have not been provided for the eyeglasses frame in the set of device data; comparing the set of device data to a set of data in a database comprising at least one set of device data; computing an average of data from eyeglasses frames with similar features in the database corresponding to the other portion of the features of the eyeglasses frame not provided for the eyeglasses frame in the set of device data, as if the other portion of the set of device data had been provided for the eyeglasses frame; generating, according to a first model, a value of a set of specific criteria describing compatibility between the user's face and the eyeglasses frame based on the received user data, the received device data, and the computed average, the first model trained to relate user data and device data to values of specific criteria; generating, according to a second model, a value of a global criterion based on the generated value of the set of specific criteria, the second model trained to relate values of specific criteria to values of global criteria; determining a message about the user's face characterizing the eyeglasses frame, the message associated with the generated value of the global criterion and the generated value of the set of specific criteria; and outputting the message to the user.

Images