Method and device for determining the fit of a visual device

Abstract

This disclosure relates to a method and apparatus for evaluating the fit of a visual device. In an embodiment, the method (105) includes: receiving (110) an image of a user's face; identifying (115) a subset of eyeglasses in a database that meet a preliminary fit threshold; determining (120) the lens plane of a candidate eyeglass; determining (125) the nose pad plane of the candidate eyeglass; generating (130) at least one 2D profile of the user's nose; generating (140) at least one 2D profile of the nose segment of the frame of the candidate eyeglass; virtually docking (150) the candidate eyeglasses onto the nose of the user's face; and calculating (170) the final fit of the virtually docked candidate eyeglasses when the virtually docked candidate eyeglasses meet a secondary fit threshold, the final fit being defined by at least one of a comfort criterion and / or an aesthetic criterion.

Description

background Technical Field

[0002] This disclosure relates to a method for determining the virtual fit of an eyewear device on a user's face based on comfort and aesthetics. A properly fitted eyewear device is an important step in ensuring the desired optical performance.

[0003] Related technical specifications

[0004] Choosing new eyewear can be daunting, given the sheer number of options available to your eye care needs from brick-and-mortar retailers, online retailers, and their combinations. Brick-and-mortar retailers have failed to provide sufficient inventory and convenience for the modern consumer. Online businesses strive to offer consumer-specific recommendations, but the lack of physical interaction with the consumer's proportions often makes it difficult to determine the optimal fit for a given eyewear product. Similarly, consumer uncertainty about fit often leads to in-person trying on and fitting. While finding a good fit gives consumers greater confidence in purchasing eyewear, limiting consumers to in-store inventory and the pressure associated with trying on clothes in public can complicate and negatively impact the shopping experience.

[0005] Document FR 3 065 821 A1 discloses an apparatus for obtaining measurements and 3D images to determine facial morphological parameters in order to create custom-made glasses.

[0006] Document WO 2015 / 027196 A1 discloses a method and system for creating custom products from scratch.

[0007] To leverage the convenience and inventory of online eye care services, a new method is needed to determine the fit of eyeglasses or vision devices. If the fit quality of a virtual try-on of eyeglass frames can be determined, it may be possible to recommend eyeglasses with confidence.

[0008] Therefore, a new approach is needed to provide an online eye care experience that generates personalized product recommendations and ensures good fit.

[0009] The foregoing "Background" section is intended to provide an overall overview of the context of this disclosure. The inventors' work within the scope described in this background section, and aspects of the description that may not be otherwise considered prior art at the time of submission, are neither explicitly nor implicitly acknowledged as prior art to this disclosure. Summary of the Invention

[0010] This disclosure relates to an apparatus and method for determining the suitability of a vision device.

[0011] According to an embodiment, this disclosure further relates to a method for determining the suitability of a vision device.

[0012] According to an embodiment, this disclosure further relates to an apparatus for determining the compatibility of a vision device.

[0013] The preceding paragraphs are provided as a general description and are not intended to limit the scope of the appended claims. The described embodiments and further advantages will be best understood by referring to the following detailed description taken in conjunction with the accompanying drawings. Attached Figure Description

[0014] A more comprehensive understanding of this disclosure and its many incidental advantages will be readily obtained when considered in conjunction with the accompanying drawings, as this will be better understood by referring to the following detailed description, in which the drawings are shown:

[0015] Figure 1 This is a flowchart of a method for determining the compatibility of a vision device according to exemplary embodiments of the present disclosure;

[0016] Figure 2A This is an illustration of the steps of a method for determining the suitability of a vision device according to exemplary embodiments of the present disclosure;

[0017] Figure 2B This is an illustration of the steps of a method for determining the suitability of a vision device according to exemplary embodiments of the present disclosure;

[0018] Figure 2C This is an illustration of the steps of a method for determining the suitability of a vision device according to exemplary embodiments of the present disclosure;

[0019] Figure 3A This is a flowchart of a sub-process of a method for determining the suitability of a vision device according to exemplary embodiments of the present disclosure;

[0020] Figure 3B This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0021] Figure 3C This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0022] Figure 3D This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0023] Figure 3E This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0024] Figure 4A This is a flowchart of a sub-process of a method for determining the suitability of a vision device according to exemplary embodiments of the present disclosure;

[0025] Figure 4B This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0026] Figure 4C This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0027] Figure 4D This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0028] Figure 5A This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0029] Figure 5B This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0030] Figure 5C This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0031] Figure 5D This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0032] Figure 5E This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0033] Figure 5F This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0034] Figure 5G This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0035] Figure 5H This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0036] Figure 5IThis is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0037] Figure 5J This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0038] Figure 5K This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0039] Figure 5L This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0040] Figure 5M This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0041] Figure 5N This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0042] Figure 5O This is a diagram illustrating the steps of a sub-process of a method for determining the suitability of a vision device according to an exemplary embodiment of this disclosure;

[0043] Figure 6A This is an illustration of the steps of a method for determining the suitability of a vision device according to exemplary embodiments of the present disclosure;

[0044] Figure 6B This is an illustration of the steps of a method for determining the suitability of a vision device according to exemplary embodiments of the present disclosure;

[0045] Figure 7A This is a flowchart of a sub-process of a method for determining the suitability of a vision device according to exemplary embodiments of the present disclosure;

[0046] Figure 7B This is an illustration of the steps of a sub-process of a method for determining the adaptability of a vision device according to exemplary embodiments of this disclosure; and

[0047] Figure 8 This is a hardware schematic diagram of an apparatus for performing a method for determining the suitability of a vision device, according to an exemplary embodiment of this disclosure. Detailed Implementation

[0048] As used herein, the term "a" or "an" is defined as one or more. The term "multiple" is defined as two or more. The term "another" is defined as at least a second or more. The terms "including" and / or "having" are defined as comprising (i.e., open-ended language). Throughout this document, references to "an embodiment," "some embodiments," "embodiment," "implementation," "example," or similar terms mean that a specific feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of this disclosure. Therefore, the appearance of such phrases throughout this specification or in various places does not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics can be combined in any suitable manner without limitation in one or more embodiments.

[0049] In this article, the terms "visual device" and "glasses" are used interchangeably. Similarly, the terms "consumer" and "user" are used interchangeably as appropriate.

[0050] When consumers purchase eye care products from online retailers, they are asked to choose from a seemingly endless array of frame designs to reflect their preferences regarding the design of the glasses or vision devices. Typically, due to this overwhelming experience, consumers seek out physical retailers that offer a real try-on experience, allowing them to assess how the vision device looks and feels on their face. For example, consumers can be confident in the comfort, size, and aesthetic appeal of the vision device.

[0051] Therefore, the expectation for future online eye care services must be to recommend suitable visual devices to consumers, which have been identified as meeting various fit factors, including comfort, size, and aesthetic appeal.

[0052] Therefore, a method is needed that can (1) automatically score all available visual devices in a visual device database (i.e., catalog) for a given face of a consumer based on comfort criteria (e.g., comfort on the nose, ears, temples, and cheeks) and / or aesthetic criteria (e.g., the matching of face shape, face color, face proportions with the shape of the visual device, the color of the visual device, general aspects of the visual device, lens geometry, prescription data, lens tint, etc.); (2) display the scores to the consumer in a useful manner; and (3) predict whether a given visual device will be well-fitted for a given face of a consumer.

[0053] In one embodiment, the method may include automatically scoring all available visual devices in a visual device database for a given face of a consumer, based on an estimate of the perceived weight of the visual device.

[0054] According to an embodiment, this disclosure describes a method for determining the fit of a vision device to a user's face. The determination of this fit can be performed iteratively for each vision device in a database of vision devices. The determined fit may include assessments of comfort criteria, aesthetic criteria, and perceived weight of the vision device.

[0055] In one embodiment, the method includes generating a nose contour of the user's nose and glasses-based planes that inform the positioning of candidate glasses during virtual docking, etc. The generated nose contour of the user's nose can be an estimated nose contour or an estimated nose model. The estimated nose contour can be a three-dimensional (3D) estimated nose contour.

[0056] In one embodiment, virtual docking can be integrated with virtual try-on to provide users with an augmented reality experience.

[0057] In one embodiment, the method includes generating at least one two-dimensional (2D) profile of a user's nose and at least one 2D profile of candidate glasses. The at least one 2D profile of the candidate glasses includes the nose key position of the candidate glasses.

[0058] In one embodiment, the method includes virtually mating candidate glasses onto the user's nose based on at least one 2D profile of the user's nose and at least one 2D profile of the candidate glasses.

[0059] In one embodiment, the method includes evaluating the positioning of candidate glasses for virtual docking relative to a user's facial landmark, the evaluation indicating whether the candidate glasses for virtual docking have acceptable fit and should be further considered.

[0060] In one embodiment, the method includes scoring candidate eyeglasses to achieve acceptable fit, with scores defining values ​​for comfort criteria, aesthetic criteria, and perceived weight.

[0061] Now refer to the attached diagram, Figure 1 A flowchart of the method described in this disclosure is provided. Figure 1 Method 105 will be described as applying to a subset of visual devices or a subset of eyeglasses from a visual device database or an eyeglasses database, for a single visual device or eyeglass. It should be understood that method 105 can be performed iteratively for each visual device within the subset of visual devices to determine which visual device is suitable for the user and to determine a score related to the suitability of the visual device for the user.

[0062] Accordingly, in step 110 of method 105, an image of the user can be received. In an embodiment, the user's image can be one or more 2D images and / or one or more 3D image datasets. The image can be the user's face. The one or more 2D images can be two 2D images, the first image being a front view of the user's face and the second image being a side view of the user's face.

[0063] In this embodiment, the user's morphological data can be determined from the received images. For example, two 2D images may include a scale factor that allows facial measurements (metric, imperial, or other measurement systems) to be determined. In one example, the scale factor can be determined using a ruler, a card of standard size, or any object with a known dimension. In another example, the scale factor can be determined using features of the user's face, such as pupillary distance or other known morphological distances on the user's face. In yet another example, the scale factor can be obtained knowing the measurement distance and camera focal length. In yet another example, the scale factor can be extracted directly from a 3D image dataset of the user's face.

[0064] In an embodiment, the user's morphological data, which is attached to or replaces the user's image, can be directly provided to method 105.

[0065] In step 115 of method 105, a subset of glasses from the glasses database can be identified with respect to the user.

[0066] Therefore, eyeglass data for each eyeglass within the eyeglasses database must be obtained. Similar to step 110 of method 105, the eyeglass data can be generated based on images of each eyeglass in the eyeglasses database. Each eyeglass image can be one or more 2D images and / or one or more 3D image datasets. Each eyeglass image can also be two 2D images: a first image is a front view of the eyeglass, and a second image is a side view of the eyeglass.

[0067] In this embodiment, eyeglass feature data can be determined from images. For example, two 2D images may include a scale factor that allows eyeglass feature measurements (metric, imperial, or other measurement systems) to be determined from them. In one example, the scale factor can be determined using known dimensions of the eyeglasses, such as frame width, frame height, temple length, etc. In another example, the scale factor can be determined using another object of known size located within the field of view of the image. In yet another example, the scale factor can be determined directly from a dataset of 3D images of the frame.

[0068] In this embodiment, the image attached to or replacing the glasses, or the glasses feature data, can be directly provided to method 105. For example, the glasses feature data can be provided by the glasses manufacturer or wholesaler.

[0069] In another embodiment, the eyeglass feature data may be a combination of data determined from an image and data provided directly to method 105 by the eyeglass manufacturer or wholesaler. For example, the shape, material composition, color, and size of the eyeglasses may be obtained directly from the manufacturer or wholesaler, while facial angles, slant angles, etc., may be determined from an image.

[0070] Returning to step 115 of method 105, and after obtaining morphological data about the user's face and eyeglass feature data about the eyeglasses in the eyeglasses database, initial screening of the eyeglasses database can be performed. Initial screening or pre-filtering may include comparing the user's face with each pair of eyeglasses in the eyeglasses database. This comparison may be related to comfort, aesthetics, or both. Comparisons related to comfort may be based on the adaptability of each pair of eyeglasses in the eyeglasses database to the user's face width.

[0071] Therefore, the first method involves measuring the user's temporal width ( The temporal width is used as the user's face width. In one embodiment, the temporal width can be determined directly using a first image of the user's face from a frontal view or a dataset of 3D images of the user's face. In another embodiment, the user's temporal width can be determined indirectly. First, the sphenoid sinus width can be measured from a first image of the user's face from a frontal view. Then, the sphenoid sinus width can be used to estimate the temporal width of the user's face. ).Apart from In addition, for The estimation can be based on gender information and the detected pupillary distance. Gender information ( (This can be targeted at male input) And for women, the input is Using this information, the temporal width can be calculated as follows: ,in These are hyperparameters that are adjusted based on the training dataset.

[0072] To determine whether a given pair of glasses meets the initial fit threshold, the calculated temporal width can be compared with the total width of the glasses determined for each pair of glasses in the glasses database. A comparison was made. The total width determined for each pair of glasses in the eyeglasses database was defined as the widest distance from the nose pad plane, excluding the temples. The deviation limit between the total width of the glasses and the temporal width was... This can be defined as the initial fitness threshold. Accordingly, if... The visual device was eliminated in the pre-filtering step, and the glasses currently under consideration are considered to have uncomfortable fit.

[0073] The second method involves measuring the user's ear width ( The auricle width is used as the user's face width. In one embodiment, the auricle width can be determined directly using a first image of the user's face from a frontal view or a dataset of 3D images of the user's face. In another embodiment, the user's auricle width can be determined indirectly. First, the sphenoid sinus width can be measured from a first image of the user's face from a frontal view. ) or temporal width ( Then, the width of the user's auricle can be estimated using the sphenoid sinus width or the temporal width. ).Apart from or In addition, for The estimation can be based on gender information and the detected pupillary distance. Gender information ( (This can be targeted at male input) And for women, the input is Using this information, and assuming the sphenoid sinus width is used, the auricle width can be calculated as follows: ,in These are hyperparameters that are adjusted based on the training dataset.

[0074] After determining the ear width, and in order to determine whether a given pair of glasses meets the initial fit threshold, the temple opening of the given pair of glasses must be determined. In an embodiment, the temple opening can be determined directly using a first image of the front view of the glasses or a 3D image dataset of the glasses. For example, it can be... The temple opening is compared to the distance between the distal ends of the temples of the glasses. Laterally, temple opening is the distance between the distal ends of the temples of the glasses and the distal ends of the temples. The degree of inward movement is measured. In another embodiment, the temple opening of the eyeglasses can be determined indirectly. The indirect determination of the temple opening can be based on information about the material used for each arm of the eyeglasses (in order to deduce the rigidity of the temple). This information informs the possible optical guidelines that can be used. For example, thin and smooth temples indicate a temple opening of -15 mm, semi-rigid temples indicate a temple opening of -10 mm, rigid or strong temples indicate a temple opening of -5 mm, and very rigid temples indicate a temple opening of approximately -2-3 mm.

[0075] Temple opening and total width of the glasses can be used to determine the width of the glasses at ear level or In this way, .

[0076] Accordingly, in order to determine whether a given pair of glasses meets the initial fit threshold, the deviation limit between the auricle width and the width of the glasses at ear level or This can be defined as the initial fitness threshold. Accordingly, if... The visual device was eliminated in the pre-filtering step, and the glasses currently under consideration are considered to be uncomfortable to wear.

[0077] In an embodiment, comparisons related to aesthetics can be based on the fit of each pair of glasses in the eyewear database, particularly the shape and color, to face shape and / or skin color and / or eye color.

[0078] To this end, the first approach involves categorizing both the user's face shape and the eyeglass frame shape into several shape categories. For example, the face shape category could include square, rectangle, heart, rhombus, circle, and oval. The eyeglass frame shape category could include square, rectangle, circle, pantothenic, and D-shape, among others.

[0079] In this embodiment, the facial shape can be determined indirectly. First, the temporal width can be determined in step 115 of method 105. ), making the vertical position of the temples ( Use the vertical position of the lowest point of the mandible as the starting level. The ending level can be determined from the starting level to the ending level. , ... Each level is equidistant from its adjacent level. Then, using facial landmark information around the jawline, the maximum width of the face can be calculated accordingly. , ... Based on these absolute and relative width values, one of the shape categories listed above can be selected as the face shape.

[0080] In one embodiment, the eyeglass frame shape can be obtained directly from frame data from the manufacturer or retailer. In another embodiment, the eyeglass frame shape can be obtained indirectly. For this purpose, one of the lens regions (e.g., left or right) can be extracted from one of the eyeglass frame images, and the curvature of that region can be calculated. The digital curvature value can then be converted into the different shape categories listed above, as skin shapes. Finally, the fit of the eyeglass frame can be evaluated using a predefined binary shape matching matrix. If the evaluation fails, the visual device is eliminated in the pre-filtering step, and the eyeglasses currently under consideration are deemed to have an ineffective fit from an aesthetic point of view.

[0081] The second method involves categorizing the user's skin color and the eyeglass frame color into several color categories. For example, light, olive, and dark colors can be used as categories for skin color, while black, blue, pink, and white can be used as categories for eyeglass frame colors.

[0082] In this embodiment, skin color can be determined indirectly. Using facial landmark information, the cheek area can be extracted, and the average RGB value of that area can be calculated. The numerical RGB value can then be converted into the different color categories listed above as skin color.

[0083] In one embodiment, the eyeglass frame color can be obtained directly from frame data from an eyeglass frame manufacturer or retailer. In another embodiment, the eyeglass frame color can be determined indirectly. This can be done by extracting the eyeglass frame outline region from one of the eyeglass frame images and calculating the average RGB value within that region. The digital RGB values ​​can then be converted into the different categories listed above as eyeglass frame colors. Finally, a predefined binary color matching matrix is ​​used to evaluate the frame fit. If the evaluation fails, the visual device is eliminated in the pre-filtering step, and the eyeglasses currently under consideration are deemed to have an uncomfortable fit from an aesthetic point of view.

[0084] Now back Figure 1 After pre-filtering the glasses database to identify a subset of glasses to be considered, method 105 proceeds to step 120, and the remaining steps and sub-processes of method 105 will be described in light of a single glasses or candidate glasses. It should be understood that method 105 can be performed iteratively.

[0085] Next, in step 120 of method 105, the lens rim plane or "frame plane" of the candidate eyeglasses can be determined. It is understood that the primary purpose of eyeglasses is to ensure that the lenses are positioned at a relevant distance from the user's eyes and at the optical center of the lenses to provide optimal optical performance to the wearer. For this reason, the lens rim plane is defined as ensuring that the position of the lens rim relative to the user's eyes is a fixed constraint. In the example, the lens rim plane includes the bridge of the nose, the lenses, and the nose pad position rim, etc.

[0086] In this embodiment, the lens plane can be defined by one or more parameters. The first parameter may be the vertex distance (…). The vertex distance is defined as the horizontal distance from the lens plane to the center of the cornea of ​​the user's eye. The second parameter can be the anterior tilt angle (…). The tilt angle is defined as the angle between the plane of the eyeglass lens rim and the user's coronal or vertical plane. Figure 2A The Chinese illustration shows and Ideal instances of these two parameters can be determined using one of a variety of methods. In the example, sales statistics for eyeglasses could be used, taking into account factors such as race, lens usage, prescription, and pupillary distance. The ideal lens should be determined by factors such as the appropriate height, A, B, and D distances, and lens profile. or In some cases, It can be replace. and The associated value can be a regular value or a personalized value within a group or for an individual. In this way, It can be statistics based on sales statistics. Therefore, these two parameters can be determined as follows: and statistics In another example, the image of the user received in step 110 of method 105 may include an image of the user wearing personal glasses. The image of the user wearing personal glasses can be a 2D or 3D image. In this way, vertex distance, tilt angle, wrap angle, etc., can be determined. In some cases, this information can be determined from fitting measurements. Furthermore, using the received image, the true... and lean angle Therefore, these two parameters can be determined as follows: and When a user's image is available, some markers may be hidden by the lens rim of the glasses. A user's photo without their glasses can be used, and homography can be calculated from the markers.

[0087] In this embodiment, the tilt angle can be determined from a side-view 2D image. The bridge position and bottom position of the lens can be detected from the side-view 2D image. This detection allows for estimation of the lens rim plane and calculation of the tilt angle. Furthermore, the corneal apex can be detected from the side-view 2D image, and the corneal apex is combined with the lens rim plane to calculate the apex distance. Additionally, when employed, the wrap angle can be estimated based on an estimation of the lens hinge position and the detected bridge position.

[0088] According to an embodiment, if the plane of the eyeglass lens rim cannot be [adjusted / deformed] due to contact with the brow bone... , and / or Placement can be implemented in two ways. The first method involves increasing... Continue until the plane of the eyeglass lens no longer contacts the brow bone. The second method involves maintaining the established... , and / or Furthermore, the diameter of the possible candidate eyeglass rims is limited to avoid contact with the brow bone. Therefore, the maximum diameter of the eyeglass rims can be calculated after the frame is fitted onto the nose, as will be described in later chapters.

[0089] After step 120 of method 105 determines the lens rim plane, step 125 of method 105 describes determining the nose pad plane or "nose pad plane" of the eyeglasses.

[0090] Many eyeglasses include nose pads, which help position the glasses on the nose. Therefore, as... Figure 2B As shown, determining the nose pad plane is crucial for fitting candidate glasses onto the user's nose. In embodiments, the nose pad plane can be considered as being located at a specific position relative to the geometry of the nose pad. In embodiments, the nose pad plane is more than one plane of the nose pad and is defined on different aspects of the nose pad. In the example, the nose pad plane is a single plane located in the middle of the nose pad. This means that, in embodiments, the nose pad plane can be located at a specific distance from the lens plane ( ).

[0091] In an embodiment, the nose pad plane of the eyeglasses can be defined relative to the lens rim plane.

[0092] In an embodiment, It can be dynamically measured for each candidate frame.

[0093] In an embodiment, Information from the manufacturer can be used to make statistical estimates (e.g., the nose pads of glasses are typically located 4 mm from the lens rim).

[0094] In this embodiment, the nose pads of the glasses can be fixed, such as... Figure 2B In another embodiment, the nose pads of the glasses can be flexible rotating nose pads, such as... Figure 2C This change in the nose pads of glasses will be discussed later.

[0095] Now back Figure 1 In order to place the glasses on the user's nose, method 105 proceeds to sub-process 130, in which at least one 2D contour of the user's nose is generated. The generation of at least one 2D contour of the user's nose allows for iterative browsing of the nose contour in an effort to align the nose pad plane of the glasses with the nose. To do this, markers on the user's nose must be identified. In embodiments, such markers may include the highest possible contact point between the nose pad plane of the glasses and the user's nose, and corresponding contact points between the lens plane of the glasses and the user's nose.

[0096] refer to Figures 3A to 3E The sub-procedure 130 of method 105 is described in more detail. Regarding... Figure 3AThe purpose is to generate at least one 2D contour of the user's nose. The first 2D contour of the user's nose in at least one 2D contour can be the intermediate nose contour determined in step 131 of subprocess 130, and the first 2D contour of the user's nose in at least one 2D contour corresponds to the 2D nose contour at the intersection of the nose contour and the nose pad plane of the glasses. Mathematically, the intermediate nose contour can be described as for any , The second 2D contour in at least one 2D contour of the user's nose can be the main nose contour determined in step 132 of subprocess 130, whereby the second 2D contour in at least one 2D contour of the user's nose corresponds to the 2D contour of the nose at the intersection of the nose contour and the lens plane. Mathematically, the main nose contour can be described as for any , . and Reflecting on the nose The width of the nose at horizontal level and the angle of the face. The lower limit of the nose. This can be identified as the lowest physical point of the nose that can support the eyeglass components. In other words, It can be defined as the height level of the nose, at which the curvature of the nose is inverted. Figure 3B The diagram illustrates the above description at a high level.

[0097] To determine the midline and main contour of the nose, different strategies can be employed based on the type of user image received in step 110 of method 105.

[0098] In an embodiment, if the user's image includes a 2D front view of the user's face, it can be used in conjunction with an estimation of a standard nose angle to determine at least one 2D contour of the user's nose. For this purpose, it is assumed that facial landmarks detected within the 2D front view of the user's face (e.g., Figure 3C The symbol in the image corresponds to the 2D outline of the base of the nose or It can be observed as the intersection of the nose and the plane passing through the root and alar of the nose, or the "plane of the base of the nose". Figure 3C The plane at the base of the nose can be observed. Therefore, in Figure 3C As shown, it can be retrieved , and ,in It is the root of the nose, and and These are the highest possible contact points between the nose pad plane and the lens rim plane of the glasses and the nose, respectively.

[0099] In an embodiment, You can directly use facial markers , and calculate. Transformation can be applied based on Calculation. In the example, the applied transformation could be a reductive transformation. To do this, the function... (Its simulated nose with a V-shaped angle) is incorporated into the following equation:

[0100] (1)

[0101] (2)

[0102] Then Limited to Within the range. Similar to the above, Transformation can be applied to it based on Calculation. In the example, the transformation could be a reductive transformation. As mentioned above, the function... It can be incorporated into the following equation:

[0103] (3)

[0104] (4)

[0105] Then Limited to Within the range. During generation and When both are present, such as Figure 3D As shown, the function This can be determined by using statistics related to the standard nose's angle of retraction. Figure 3E The figure illustrates an overview of generating at least one 2D outline of the nose of a user's face based on a 2D image of the user's face.

[0106] In another embodiment, wherein the user's image includes a 3D image dataset of the user's face, and a 3D model of the user's nose contour can be extracted from the 3D image dataset and can be used to derive at least one 2D contour of the user's nose as the intersection of the 3D model of the nose contour or the nose contour with the intersection of the eyeglass lens plane and the eyeglass nose pad plane determined in steps 120 and 125 of method 105, respectively.

[0107] In another embodiment, where the user's image does not include a 3D image dataset of the user's face, a 3D model of the user's nose contour can be generated based on a 2D dataset including images and videos (e.g., pictures, videos) acquired via a smartphone. This 3D model generation method can be performed via 2D photogrammetry, which may be related to or unrelated to simultaneous localization and mapping, motion structures, deep learning, etc. The acquisition of the 2D dataset can be done by the user or another person and can be scaled down depending on the 3D reconstruction method employed. For example, a single image may be used instead of several images.

[0108] In the example, the user's images consist of a 2D dataset instead of a 3D image dataset of the user's face. The facial markers of the 2D images of the user's face can be matched with 3D nose patterns, assuming a set of representative 3D nose pattern models (including nose patterns from different morphologies) is available. For this purpose, B-splines can be used to interpolate the facial markers of the 2D images within the 3D nose patterns.

[0109] In another example, when the user's image includes a 2D dataset of the user rather than a 3D image dataset of the user's face, a 3D model of the user's nose can be generated based on 2D images of the user obtained from the front and side views as a first approximation. The front view provides the nose in... The left and right edges within the plane. The side view provides an image of the nose. The midline in the plane belongs to the vertical plane that passes through the middle of the nose. From this, an approximate model of the nose can be constructed, and the cross-sectional shape of the nose can be derived.

[0110] Now back Figure 1 Method 105 proceeds to sub-process 140, wherein at least one 2D profile of the nose segment of the lens rim of the candidate eyeglasses is generated. The at least one 2D profile of the nose segment of the lens rim of the candidate eyeglasses is generated in a manner similar to the 2D profile of the user's nose.

[0111] refer to Figures 4A to 4D The sub-procedure 140 of method 105 is described in more detail. Regarding... Figure 4A The objective is to generate at least one 2D profile of the nose segment of the lens rim for candidate eyeglasses. The first 2D profile of the at least one 2D profile of the nose segment of the lens rim is the nose pad profile determined in step 141 of subprocess 140. Mathematically, the nose pad profile can be described as... , The second 2D profile of at least one 2D profile of the nose segment of the eyeglass rim can be the eyeglass rim profile determined in step 142 of subprocess 140. Mathematically, the eyeglass rim profile can be described as for any , . and These respectively reflect the contours of the nose pads on the glasses. The width of the nose and the angle of the face at horizontal level. Figure 4B In the context of eyeglasses, the nose pad position is reflected by points arranged lining the nose area of ​​the eyeglass frame.

[0112] To determine the nose pad profile and lens rim profile of eyeglasses, two methods can be considered, depending on the mobility of the nose pads.

[0113] In an embodiment, for eyeglasses with static nose pads (e.g., plastic lens rims), the positional features of the nose pads (i.e., the top and bottom of the nose pads) can be detected from the received image or determined statistically based on a standard position of the nose pads. For example, the standard position may specify that the top of the nose pads is located at the height of the box (see [link to documentation]). Figure 4B )above The distance is mm, and the bottom of the nose pads is below the height of the frame. The distance is mm. Since it relates to the above equation concerning at least one 2D profile of the nose section of the eyeglass rim, It's the top of the nose pads on the glasses, and It's the bottom of the nose pads on the glasses.

[0114] In the case of candidate eyeglasses with static nose pads, the discontinuous static profile of the nose pad contour can be calculated by extracting the eyeglass rim profile. For example, the area defined by the eyeglass nose pads or It can be used for , Isolate them. Then, it can be done through... , The lens rim profile is calculated as a continuous static profile.

[0115] Figure 4C The diagram illustrates the method described above for static nose pads. Alternatively, at least one 2D outline of the eyeglasses can be provided by the eyeglass designer / manufacturer.

[0116] In embodiments, for eyeglasses with movable or rotating nose pads (e.g., certain metal eyeglass frames), the positional features of the nose pads (i.e., the top and bottom of the nose pads) can be detected from the received image or determined statistically based on a standard position of the nose pads. For example, the standard position may specify that the top of the nose pads is located at the height of the box (see...). Figure 4B )above The distance is mm, and the bottom of the nose pads is below the height of the frame. The distance is mm. Since it relates to the above equation concerning at least one 2D profile of the nose section of the eyeglass rim, It's the top of the nose pads on the glasses, and It's the bottom of the nose pads on the glasses.

[0117] When the candidate eyeglasses have rotating nose pads, the continuous static profile of the eyeglass rim profile can be determined as follows: , Next, the discontinuous parametric profile for simulating the flexibility of the nose pads in eyeglasses can be determined as follows: , ,in It is the distance between the nose pads of the glasses, and It refers to the angle of the nose pads on the glasses, assuming that the nose pads of both glasses have the same face angle.

[0118] Therefore, in the rotating nose pad design, it is assumed that the position of the rotation axis of the nose pad is known, the possible rotation amplitude is known, the attachment point of the nose pad relative to the rotation axis is known, and the length of the nose pad is known. Thus, a parametric model can be created using different rotation angles of the nose pads (the same for both nose pads) and different distances between the attachment points of each nose pad (adjusted by slightly bending the metal rod supporting the nose pads). This parametric model generates different nose pad profiles.

[0119] The above "limits" for at least two 2D profiles of the nose section of the eyeglasses will be described below. Regarding the lens rim profile, The curvature of the nose pad position of the glasses (e.g.) ) begins to exceed a certain limit (e.g. The height of ) and This refers to the height of the bridge of the nose on the eyeglasses. For example, if the bridge of the nose is thick, the "upper limit" might equal the "lower limit" height. Regarding the contour of the nose pads on the eyeglasses... It is the "upper limit" height of the nose pads for glasses, and It is the "lower limit" height of the nose pads for glasses.

[0120] Now back Figure 1 Furthermore, after generating at least one 2D contour of the user's nose and at least one 2D contour of the nasal segment of the candidate glasses, the candidate glasses can be virtually docked onto the user's nose in sub-procedure 150 of method 105.

[0121] refer to Figures 5A to 5OSub-process 150 of method 105 is described in more detail. Sub-process 150 may include different steps based on the type of nose pads being used. For example, glasses with static nose pads may be handled differently than glasses with parametric or rotating (or moving) nose pads. Furthermore, each type of nose pad may be processed differently to virtually mate the glasses to the user's nose.

[0122] about Figures 5A to 5C A first method for static nose pads of eyeglasses can be implemented. In an embodiment, the first method for static nose pads of eyeglasses may include a first step of aligning the nose pads of the eyeglasses with the nose (and thus in the nose pad plane) and a second step of confirming that the lens rim profile of the eyeglasses does not intersect with the nose (and thus in the lens rim plane). The first step is performed to prioritize contact between the user's nose and the nose pads of the eyeglasses. For this purpose, a method can be used... and The second step is performed to confirm that the glasses are positioned correctly on the nose. For this, one method can be used... and .

[0123] The first step may include, from facial markers (As discussed earlier) Begin browsing with a predetermined step size (e.g., step size = 0.1 mm). At a given level The nose pads of glasses can be placed in the area where they are placed. Above (i.e., finding the exact level where the nose contour will contact the frame), and the feasibility of that position can be confirmed. In other words, as Figure 5A As shown, the first step can be performed as follows: (1) Calculate Nasal width at horizontal level ( ) and face width ( (2) Find the level of the nose pads on your glasses. ,in and (3) inspection Is it in (Right now Within. If within The current docking location is acceptable or "OK", such as... Figure 5B and Figure 5C As shown, the subprocess proceeds to the second step.

[0124] The second step may include checking whether the outline of the eyeglass rim intersects with the nose at the current docking position (within the plane of the eyeglass rim). To do this, a given... In scope Browse between The browsing result determines which one. Corresponding to the nose contour reference Therefore, it can then be confirmed. The width of the nose is greater than or equal to The width of the nose.

[0125] If the second step determines that the current docking position is feasible, then nose browsing can be stopped, and the following content can be recorded in the database: (1) The corresponding vertical coordinate of the nose (In the nose reference), (2) the corresponding ordinate of the glasses (In the eyeglasses reference). The recorded coordinates allow for the calculation of comfort standards for eyeglass positioning during sub-procedure 170 of method 105. Of course, it is understood that if no feasible mating position is found at the end of the aforementioned nose scan, the eyeglasses frame can be discarded.

[0126] In addition to the above, a second method for static nose pads on eyeglasses can be implemented.

[0127] In an embodiment, a second method for static nose pads on eyeglasses may include calculating a smoothing model based on frame contour points. In an example, the smoothing model may be a B-spline model, such as... Figure 5D and Figure 5E As shown in the diagram. The smoothing model helps to account for symmetry and reduce noise. In an embodiment, the smoothing model can be defined as... In the example, these points can be limited to the contact area. For example, for the contour of an eyeglass lens rim, the range can be limited to... and Between, and for the nose pad contour of glasses, the range can be limited to and between.

[0128] In one embodiment, a second method for static nose pads on eyeglasses may include calculating a smoothed model based on nose contour points. Because the nose contour has a discontinuous profile, such as... Figure 5F As shown, two smooth models can be generated, such as Figure 5G As shown. One for the right eye side or One for the left eye side or In the example, and .

[0129] For both the frame contour and the nose contour, two standards can be calculated. First, for a given parameter... The width of the nose and the glasses can be defined separately. and The truncated angles of the nose contour can be defined for the right and left eye sides respectively. and Furthermore, the chamfer angles of the eyeglasses outline can be defined separately for the right and left eye sides. and .

[0130] More precisely, for a given , It can be calculated as such that .therefore, And targeting , It can be calculated as such that .therefore, .

[0131] Given the above, the mating can be calculated by searching for contact points with the same width and the same chamfer (on the nose and glasses). This can be accomplished by solving an optimization problem defined by a cost function. In the example, the cost function could be... ,in These are parameters that define the contact points on the nose, and These are parameters that define the contact points on the glasses; the cost function can be applied to a given... and The cost function is evaluated. As described in this paper, the cost function is a classic function used to solve least squares problems, but it can also be any other cost function that is easy to apply and relevant. As mentioned above, and as... Figure 5H As shown, the solution is to have a position with the same width and the same chamfer at the contact point between the eyeglasses outline and the nose outline.

[0132] In addition to the above, a third method can be implemented for static nose pads on eyeglasses.

[0133] In an embodiment, a third method for static nose pads for eyeglasses may include optimizing the position of the eyeglasses through two translations. In the example, the translations may be in the x-direction and the y-direction, such as... Figures 5I to 5K As shown. The translation distance can be determined using a cost function, such that points on the nose pad contour of the glasses are projected onto the nose contour, in order to calculate the distance between the nose pad and the nose, as shown. Figure 5L and Figure 5M As shown.

[0134] As described above, sub-process 150 of method 105 can also be considered for eyeglasses with parameterized or rotated (or moved) nose pads. For parameterized nose pads, three different methods can be implemented to determine the position of the eyeglasses and the nose pads thereon on the user's nose.

[0135] In an embodiment, the first method includes processing each pair of parameter values. Generate candidates for "glasses nose pad contour" or Each pair of parameter values ​​can vary within a given range. Therefore, for each candidate "glasses nose pad profile", the corresponding glasses (current) can be... Candidate+ This is every The common (all candidate) mating points are placed on the user's nose, as described above in the first method for static nose pads for eyeglasses, and the level of comfort can be assessed according to the method described herein. Based on the mating points, parameterized pairs can be selected. Therefore, the candidate "eyeglass nose pad position profile" and the associated frame positioning on the nose corresponding to the maximum level of comfort were selected.

[0136] In one embodiment, the second method involves adapting the nose pads of the glasses to the contours of the nose. For this purpose, an angle A_F can be adapted to the nose contour for each available nose pad width. Therefore, the process is adjusted considering that the contact point on the glasses is determined by the center point of the nose pads. In this way, only the contact point on the nose is sought.

[0137] For each given nose bridge width ( A docking solution can be defined where the noses have the same width. Therefore, given the spline nose model, the docking solution can be calculated for each... Nasal width ( And the following cost function can be used to calculate the docking solution: For example, for a given width The cost function and solution can be modeled as Figure 5N and Figure 5O As shown in the image.

[0138] In an embodiment, the third method includes employing information related to the nose pads of the glasses. In the case of parameterized nose pads, the position of the glasses can include the orientation of the nose pads with two translations and rotation angles. The position of the glasses can be determined through optimization: this optimization generates a parameter for the translation... Modifications and adjustments to the nose pad angle of the glasses ( The modification involves a cost function where points of the nose pad contour are projected onto the nose contour to calculate the distance between the nose pad and the nose. A similar approach is followed in the third method for static nose pads.

[0139] Now back Figure 1 After virtually attaching the candidate glasses to the user's nose, method 105 proceeds to step 160, where the global reference position of the glasses is compared with the user's face. Step 160 of method 105 is a secondary filtering step, where glasses can be eliminated as an option based on comparisons between high-level size features.

[0140] For example, when the plane of the eyeglass lens is placed on the nose, and therefore... It is known that the entire frame (i.e., the lens rim plane along with the temples) can be considered to assess certain size-driven comfort aspects of candidate eyeglasses. Size-driven comfort aspects of candidate eyeglasses include temple length matching, temple levelness, and so on.

[0141] Therefore, in the example, the features of the candidate glasses or the position of the temples of the candidate glasses (upper limit) lower limit ) can be like Figure 6A The image shows the measured image or a result retrieved from a 3D frame design database. If unavailable, the temple positions of candidate eyeglasses can be estimated based on factors such as the type of eyeglass rim and the material of the rim.

[0142] After identification and Then, you can view the glasses from the side view, such as... Figure 6B As shown. This allows , , and Points are identified in the side view. Correspondingly, the intersection of the temple of the eyeglasses with the vertical line can be detected as a point. The temples of the eyeglasses can be provided based on the angle between the temple and the plane of the eyeglass lens rim (measured from a side view of the eyeglass frame, or assumed to be 90°). Furthermore, the contact point between the temple and the user's ear can be detected as a point. .

[0143] Point With point The comparison of distances can be evaluated relative to a predetermined threshold. In the example, the predetermined threshold could be a distance in centimeters. For example, if... Then glasses can be discarded. If Then the glasses can be preserved, and at a distance (It provides the ideal useful length for the temples) and (It defines the angle between the temples and the horizontal plane) can be measured and then used to assess the comfort of the glasses (e.g., comfort on the ears). If Then the plane of the eyeglass lens rim and therefore the forward tilt angle It can be adjusted to make Therefore, the glasses positioning can be recalculated, as described in the previous sections of this disclosure.

[0144] Now back Figure 1After determining in step 160 of method 105 that the candidate glasses meet the global positioning criteria, method 105 proceeds to sub-process 170, where the final fit of the virtually docked candidate glasses is calculated. The final fit calculation in sub-process 170 of method 105 may include assessing comfort, aesthetics, etc., individually or in combination, such as referencing... Figure 7A and Figure 7B The description is as follows. In the example, the final fitness score could be a weighted average.

[0145] First, regarding comfort, in step 171 of subprocess 170, several comfort criteria can be defined. For example, comfort of the nose, eye positioning within the lens, eyebrow comfort, cheek comfort, temporal comfort, and ear comfort. Furthermore, each standard... It can consist of several sub-standards Therefore, for each criterion (or subcriterion), the following can be defined. In step 172 of subprocess 170, the evaluation function for the criterion (or subcriterion) can be defined. The input includes one or more morphological features of the user's face or eyes. In step 173 of subprocess 170, a target value representing the ideal value can be defined. In step 174 of subprocess 170, a scoring function can be defined. This scoring function evaluates the distance between the result of the evaluation function and the desired result or target value. This distance can be an absolute value or a squared distance. The score for a given subcriterion can be defined as... Then, the score of a given standard can be determined as a linear combination of each sub-standard score, or determined as...

[0146]

[0147] Understandable. Each sub-standard The weighted average of the calculated values, where These are the corresponding weights. Weights can be dynamically determined based on available training data, which is accumulated and generalized to allow the algorithm to be applied to more people. Alternatively, weights can be statically assigned based on relative importance. For example, the temporal comfort subcritique might be more important to overall eyewear comfort than the cheek comfort subcritique, and therefore the corresponding weights should reflect this relative importance.

[0148] Furthermore, the final fit score can be a combination of weighted criteria, such that each comfort criterion... (For example, comfort, aesthetics, and perceived weight) can be weighted according to their relative contribution to the final fit score. For instance, comfort may be generally considered more important than aesthetics; however, users may hold the opposite preference. Therefore, the final fit score can reflect this preference by assigning appropriate fitting weights. In an embodiment, a user can indicate their comfort preference in their user profile or during real-time interaction with a service that provides personalized eyewear recommendations.

[0149] The above will now be explained using examples of standards and substandards.

[0150] The first example focuses on nasal comfort (i.e., is it possible for the frames to press against the nose or slip off?). The standard is a combination of three sub-standards. The first sub-standard It refers to the height at which the eyeglass frame is positioned on the nose. In other words, it's the distance from the point where the nose contacts the outline of the nose pads (e.g., the frame height of the glasses). How far is it? It's understandable that the higher the frame is positioned, the greater the pressure it puts on the nose. Second sub-standard This represents a match near the contact point for the following: (1) and , and (2) and To evaluate this sub-criteria, calculations can be performed on... and The area of ​​the space near the contact point between them, and similarly, the area can be calculated in... and The area of ​​the space near the point of contact. Then each area can be added together. The larger the area, the greater the pressure the frame puts on the nose, and the more likely the frame is to slip off. Third sub-standard This represents the position of the bridge of the glasses (i.e., the highest point of the lens outline) relative to the nose. In other words, the distance from the bridge of the glasses to the top of the nose (i.e., the facial landmark point described earlier). How far is it? Therefore, the nasal comfort standard can be defined as a weighted sum of the above sub-standards, or defined as...

[0151]

[0152] The following describes the sub-standards of nasal comfort standards. Examples. Described and Matching near the contact point. For both the left and right eye sides, the distance between the nose pad and the nose near the contact point can be calculated. (Nose pad) It can be discretized around the contact point. Points: ( ). Each point The image is projected onto the nose using, for example, orthographic projection. Then, for each point, the distance can be calculated as follows: Knowing the threshold The percentage of the nose bridge length closest to the nose can then be calculated as: .otherwise, The length of the nose bridge, near the nose, can be given for each side (left eye side and right eye side) as follows: The total length is Then, the standard is based on one side. and by The ones given, among which .

[0153] In the embodiments, the following is described Substandard of the standard Examples. yes and as well as and Matching near the contact point. If It is the range of calculable areas on the nose and It is by and For the area of ​​a shape defined by a line, the evaluation function can be defined as follows:

[0154] The target value can be set to 0, and the scoring function can be described by the following equation, assuming the optimal score is 1: ,if (That is, the glasses and nose contours fit each other perfectly). ,if (That is, the contact area may be uncomfortable, but it is not necessarily a pain point), and ,if (That is, there is likely only one point of contact, and the shapes of the nose and the nose pads of the glasses are very different.)

[0155] In embodiments, a standard example is eye position comfort, focusing on the eye's position within the lens, as described below. For example, whether the frame might be positioned too high, too low, too close to the nose, or too close to the temple, etc. The position of the eye within the lens or It can be a combination of two sub-standards.

[0156] For example, eyes positioned vertically or This measures how far the actual positioning of the glasses deviates from the ideal positioning. We can assume that the "ideal" positioning is when the frame height is at eye level, and the eye level is the lowest of the four eye corners. Further, the distance from the VD ideal, or... or It can be measured as the actual vertex distance on the user's face and The distance between them is as described above.

[0157] In this embodiment, a standard example is brow comfort, and the focus is on brow comfort (i.e., is it possible for the frame to touch the brow bone?). To this end, the coordinates of the highest point of the frame can be measured. Coordinates of the lowest point of the eyebrow .distance It can be defined where the threshold of the eyebrow ( The fraction is less than zero and the fraction is Accordingly, the evaluation of candidate eyeglasses can be conducted as follows. If Then the frames can be phased out. .if ,but Of course, if ,but In other words, eyebrow comfort assesses whether the eyeglass frame obstructs the eyebrows, and if so, to what extent. This assessment can be made by comparing the relative positions between the uppermost point of the frame and the lowermost point of the brow bone.

[0158] In this embodiment, a standard example is cheek comfort, and the focus is on cheek comfort (i.e., is it possible for the glasses to touch the cheek?). To do this, the coordinates of the lowest point of the glasses can be measured. Coordinates of the highest point of the cheek .distance Among them, the threshold of the cheek ( ) and fractions As described below. Accordingly, the evaluation of candidate eyeglasses can be performed as follows. If Then the candidate glasses can be eliminated. .if ,but .if ,but .

[0159] Additional comfort criteria that can be assessed include temporal comfort (i.e., is it possible for the glasses to compress the temporal region?), ear comfort (i.e., is it possible for the glasses to compress the ears and / or is the frame worn horizontally enough?), etc.

[0160] According to the embodiments, an appropriate threshold must be selected for a given criterion. For example, regarding the above... A standard example, which is understandable. It is aimed at one of its sub-standards threshold ( In some cases, choosing an appropriate threshold can be tricky. However, all the thresholds applied in this paper represent the relative distance between two scales, meaning that none of the thresholds are negative. In other words, the minimum value of each threshold is 0, and the only requirement is to set an appropriate upper limit for each threshold. To this end, the appropriate threshold for each criterion can be derived using the following method.

[0161] First, the range of each threshold can be selected based on expert knowledge. In the example, It can be selected as belonging to the range [0-50]. Next, the increment step size ( The value can be set to 1, causing the method to iterate over all possibilities within a defined range, increasing by only a single step size in each iteration. Then it can be used... The scoring algorithm runs on the current value. This can be done for each evaluator or tester. Obtain the final score for all frames. And take that into consideration. In the examples, the scores can be compared with the tester's subjective scores or To make a comparison, a test can be applied to evaluate how well the rankings (i.e., scores) of the two series can be described using a monotonic function. In the example, this test could be... and The Spearman ranking is used between the two series. The ranking test scores range from -1 to 1, where 1 indicates that the two series are perfectly correlated (i.e., identical), -1 indicates that the two series are perfectly negatively correlated, and 0 indicates that there is no correlation between the two series. If the score is greater than 0.7, the two series can be considered well-matched. The goal is to find a set of thresholds that ensures all scores are greater than 0.7. In this way, if any one or more thresholds result in no significant difference in the ranking test across all well-matched series, the more stringent (i.e., the most stringent) threshold is chosen as the final threshold.

[0162] In addition to the above, sub-process 170 of method 105 can be implemented according to a machine learning-based algorithm to calculate a comfort score. For this purpose, user feedback on the criteria can be collected. Feedback on the 1-9 semantic scale (as illustrated above). Standard It can be relative to glasses Nasal comfort can be scored as ,in The signage information can be related to this information. For example, signage information. It can be the nose. All marker coordinates can be tagged as... . It can be considered as input, while It can be a machine learning-based model The output of this model is for the standard. minimize This function can be used in all standards. Upgrade, and can be based on relevant Establish corresponding .

[0163] According to an embodiment, in addition to the aesthetic and comfort criteria outlined above, each candidate pair of glasses in the eyewear database can be evaluated in terms of perceived weight, thereby providing an estimate of the perceived weight of the glasses based on the specific features of the user's face. Therefore, the final fit score can be a weighted score reflecting each of the aesthetic, comfort, and perceived weight criteria.

[0164] First, after determining the weight of the glasses, they can be ranked. The weight of the glasses can be the sum of the weight of the rims and the weight of each individual lens. The weight of a lens can depend on its geometry (i.e., volume) and its material (i.e., volumetric mass). To obtain the geometry of the lens, (1) the geometry can be determined by knowing... (1) Calculate using standard curvature charts and lens profiles, or (2) approximate the geometry based on statistical values ​​stored in an eyewear database. Therefore, the geometry of candidate eyewear can be determined to be similar to... The average value of the group. The weight of the lens can be (1) based on the volume of the lens and its material, (2) based on the lens according to The average volume and material of the group, and (3) based on the lens The average volume of the group and the amount of material ordered are used to determine this.

[0165] Secondly, through knowing Knowing the lens geometry using a standard curvature chart and lens profile, glasses can be ranked by estimating the number of contact points between the face and the glasses, and the weight of the glasses, assuming the contact points are evenly distributed on the face. Therefore, the method described above can be used to fit the frame onto the nose and estimate the number of contact points on the face, where each contact point with its axis of symmetry (the axis of symmetry being a vertical line through the center between the eye and the tip of the nose) is assigned a score of 1. The total scores can be summed, and the original weight of each pair of glasses can be divided by the total score. Each pair of glasses can then be ranked based on perceived weight.

[0166] Third, and assuming that 3D scans of the lenses and frames are known, the perceived weight of each pair of glasses can be estimated using objective criteria. For this purpose, the methods described above can be used to mount the glasses onto the nose. Young's models of the contact points (e.g., nose pads, bridge, temples) and the face (e.g., skin, bones) can be used to perform mechanical modeling via finite element method (FEM). The glasses can then be ranked based on their mechanical properties.

[0167] According to an embodiment, the final fit score generated in subprocess 170 of method 105 can be generated for each candidate eyeglass in a subset of eyeglasses and can be used to present one or more eyeglasses to a user. This presentation may include the display of the final fit score. Each final fit score may reflect each assessment criterion (including a weighted fit score) or a display of recommended eyeglasses based on the final fit score.

[0168] In this embodiment, and as described above, each final fit score can be a combination of calculated values ​​for comfort criteria and calculated values ​​for aesthetic criteria. In this example, the combination can be a weighted combination of the calculated values. In another embodiment, each final fit score can include the perceived weight of the candidate glasses, which is also assigned a weight.

[0169] When more than one pair of glasses is highly rated, the number of frames presented to the user can be modified to avoid overwhelming the user. For example, if many frames perfectly satisfy a set of constraints (e.g., distance close to 0, or the maximum value of a criterion), the glasses can be categorized and / or sorted so that they are displayed to the user by criterion, by type of glasses, etc. Of course, if only a few glasses satisfy the constraints, the number of glasses displayed to the user can be increased so that those glasses with lower rankings are also presented. For this purpose, the glasses can be reordered with relaxed constraints. In another example, according to an embodiment, method 105 can be implemented such that the final fit score places each pair of glasses into a category of glasses, and then the glasses by category can be presented to the user.

[0170] According to an embodiment, method 105 can also, from the perspective of aesthetics and comfort, attempt to automatically check whether eyeglasses in the catalog are acceptable, knowing the user's visual prescription. This may include, based on knowing (1) the user's prescription (i.e., + pupillary distance), and (2) the maximum acceptable lens thickness for the user determined by the focal group. The estimated (i.e., aesthetics + weight), and (3) the lens diameter to achieve the maximum acceptable lens thickness, which is determined by the lens index selected by the user or based on the default index corresponding to the best sales index in a given country, region, city, state, etc. where the user may reside.

[0171] This includes following the Gullstrand formula, as follows:

[0172]

[0173] in, Equals 1, equal (That is, the radius of the lens's anterior refractive power). equal (That is, the radius of the lens's posterior refractive power). It is the diameter of the lens, and It refers to the thickness of the lens at the center of the lens.

[0174] Therefore, the first equation can be derived as follows:

[0175]

[0176] also,

[0177]

[0178] Based on the first equation and the second question above, It can be calculated as the lens diameter to achieve a given edge thickness (depending on...) and ).

[0179] The standard curvature chart of the lens index (selected by the user or assumed, as described above) can be used to calculate... and Therefore, based on the user's maximum acceptable lens thickness... It can be calculated ,in It refers to the diameter of the lens that achieves this thickness. Then, This can be compared to size E (the maximum distance between the optical center of the lens and the lens profile) (i.e., the useful lens diameter = size E × 2). Therefore, it can be understood that size E can be (a) provided by the retailer or (b) estimated based on the inner profile of the eyeglasses, assuming the pupil height positioning is known (e.g., size B + 4).

[0180] In the embodiment, if size E < If so, the glasses are acceptable, and the lens thickness is estimated to be acceptable from an aesthetic / comfort perspective. However, if If the size is less than E, the glasses are rejected because the lens thickness is estimated to be too high, resulting in an unattractive / uncomfortable appearance.

[0181] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed herein and their structural equivalents, or combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more computer program instruction modules encoded on a tangible, non-transitory program carrier, for execution by or control of the operation of a data processing device. Alternatively or additionally, program instructions may be encoded on artificially generated propagated signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiving device for execution by the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or combinations thereof.

[0182] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The device may also be or further include dedicated logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0183] Computer programs (also referred to or described as programs, software, software applications, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A computer program may (but does not necessarily) correspond to a file in a file system. A program may be stored as a part of a file containing other programs or data (e.g., one or more scripts stored in a markup language file), in a single file dedicated to a related program, or in multiple collaborating files (e.g., a file storing one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on a single computer or on multiple computers located at one site or distributed across multiple sites and interconnected via a communication network.

[0184] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating outputs. These processes and logic flows can also be executed by a dedicated logic circuit system, and the device can also be implemented as a dedicated logic circuit system, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit).

[0185] For example, a computer suitable for executing computer programs includes a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for making or executing instructions and one or more storage devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled thereto to receive data from or transfer data to or to them, or both. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory can be supplemented by dedicated logic circuitry systems or incorporated into dedicated logic circuitry systems.

[0186] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including voice input, voice input, or tactile input. Additionally, the computer can interact with the user by sending and receiving files to and from a device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser.

[0187] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), middleware components (e.g., an application server), or front-end components (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with embodiments of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0188] A computing system may include clients and servers. Clients and servers are generally geographically distant and typically interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to a user device, which acts as a client, for example, to display data to a user interacting with the user and to receive user input. Data generated at the user device (e.g., the result of user interaction) can be received from the user device at the server.

[0189] Figure 8 An example of such a computer is shown in the figure, illustrating a schematic diagram of a general-purpose computer system 800. According to one embodiment, system 800 can be used for operations described in association with any of the computer implementation methods described above. System 800 includes a processor 810, memory 820, storage device 830, and input / output device 840. Each component 810, 820, 830, and 840 is interconnected using a system bus 850. Processor 810 is capable of processing instructions to execute within system 800. In one embodiment, processor 810 is a single-threaded processor. In another embodiment, processor 810 is a multi-threaded processor. Processor 810 is capable of processing instructions stored in memory 820 or storage device 830 to display graphical information to a user interface on input / output device 840.

[0190] The memory 820 stores information within the system 800. In one embodiment, the memory 820 is a computer-readable medium. In one embodiment, the memory 820 is a volatile memory cell. In another embodiment, the memory 820 is a non-volatile memory cell.

[0191] Storage device 830 provides large-capacity storage for system 800. In one embodiment, storage device 830 is a computer-readable medium. In various other embodiments, storage device 830 may be a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device.

[0192] Input / output device 840 provides input / output operations for system 800. In one embodiment, input / output device 840 includes a keyboard and / or pointing device. In another embodiment, input / output device 840 includes a display unit for displaying a graphical user interface.

[0193] Although this specification contains many specific implementation details, these should not be construed as limiting the scope of possible claims, but rather as a description of features that may be specific to particular embodiments.

[0194] Certain features described in this specification in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although the features described above may be described as functioning in certain combinations, or even originally claimed in this way, in some cases, one or more features from a claimed combination may be removed from that combination, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[0195] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or in an ordered sequence, or to perform all the operations illustrated to achieve the desired result. In some cases, multitasking in parallel may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0196] Specific embodiments of the subject matter have been described. Other embodiments are also within the scope of the appended claims. For example, the actions recited in the claims can be performed in different orders and still achieve the desired results. As an example, the processes described in the figures do not necessarily require the specific order or sequential order shown to achieve the desired results. In some cases, multitasking in parallel may be advantageous.

[0197] Clearly, many modifications and variations are possible based on the above teachings. Therefore, it should be understood that the invention can be practiced in ways other than those specifically described herein, within the scope of the appended claims.

[0198] Examples of the disclosure may also be described as shown in parentheses below.

[0199] (1) A method for determining the fit of a visual device, the method comprising: receiving at least one image of a user's face via a processing circuit system; identifying, via the processing circuit system, a subset of eyeglasses from a visual device database that meets a preliminary fit threshold, the preliminary fit threshold being anatomical features of the user determined based on at least one image of the user's face; determining, via the processing circuit system and for candidate eyeglasses in the subset of eyeglasses, an eyeglass rim plane defined by a tilt angle; determining, via the processing circuit system and based on the eyeglass rim plane, an eyeglass nose pad plane intersecting with the nose pads of the candidate eyeglasses, the eyeglass nose pad plane being located at a position offset from the eyeglass rim plane by a predetermined distance; and via the processing circuit system... The processing circuitry generates at least one two-dimensional (2D) contour of the user's nose based on one or more facial landmarks detected in at least one image of the user's face and a three-dimensional (3D) nose contour estimated based on at least one image of the user's face. The at least one 2D contour of the user's nose includes at least one of the following: a first 2D contour of the at least one 2D contour of the user's nose, defined as the intersection of a plane parallel to the nose pad plane of the glasses and the estimated 3D nose contour, the first 2D contour of the at least one 2D contour located at a predetermined distance from a facial reference landmark among the one or more facial landmarks; and a second 2D contour of the at least one 2D contour of the user's nose. A 3D nose profile is defined as the intersection of a plane parallel to the lens rim plane and the estimated 3D nose profile. A second 2D profile of the at least one 2D profile is positioned relative to the first 2D profile of the at least one 2D profile based on a predetermined distance of deviation between the nose pad plane and the lens rim plane. Through the processing circuitry system and for candidate eyeglasses in the subset of eyeglasses, at least one 2D profile of the nose segment of the candidate eyeglass frame is generated, wherein the first 2D profile of the at least one 2D profile of the nose segment of the frame is associated with the nose pad of the candidate eyeglass, and the second 2D profile of the at least one 2D profile of the nose segment of the frame is associated with the lens rim of the candidate eyeglass. Through the processing... The circuit system virtually mates the candidate glasses onto the user's nose based on at least one 2D profile of the user's nose and at least one 2D profile of the nose segment of the candidate glasses' frame; the processing circuit system compares the position of features of the lens rims of the virtually matetized candidate glasses with a subset of one or more facial landmarks detected in at least one image of the user's face; and the processing circuit system calculates the final fit of the virtually matetized candidate glasses on the user's nose when the virtually matetized candidate glasses meet a secondary fit threshold, the final fit of the virtually matetized candidate glasses being defined by at least one of a comfort criterion and / or an aesthetic criterion.

[0200] (2) The method as described in (1) further includes, through the processing circuitry system and in order to calculate the final fit of the virtual docking candidate glasses on the user's nose, combining calculated values ​​of each of a plurality of comfort criteria and / or calculated values ​​of each of a plurality of aesthetic criteria.

[0201] (3) The method as described in (1) or (2), wherein the combination comprises: calculating a weighted sum of the calculated values ​​of each of the plurality of comfort criteria by means of the processing circuit system; and / or calculating a weighted sum of the calculated values ​​of each of the plurality of aesthetic criteria by means of the processing circuit system.

[0202] (4) The method of any one of (1) to (3) further includes, through the processing circuit system and for each remaining eyeglass in the subset of eyeglasses, iteratively performing the determination of the eyeglass rim plane, the determination of the eyeglass nose pad plane, the generation of at least one 2D profile of the nose segment of the frame of the candidate eyeglasses, the virtual docking, the comparison, and the calculation.

[0203] (5) The method of any one of (1) to (4) further includes ranking each candidate glasses for virtual docking that meets the secondary fit threshold by means of the processing circuit system, based on a comparison of the final fit of each candidate glasses for virtual docking.

[0204] (6) The method of any one of (1) to (5) further includes generating a visual device recommendation for the user through the processing circuitry system and based on the ranking.

[0205] (7) The method of any one of (1) to (6), wherein the position of the virtual docking candidate glasses is calculated based on a second 2D profile in at least one 2D profile of the user's nose and a second 2D profile in at least one 2D profile of the nose segment of the frame associated with the eyeglass rim of the candidate glasses, by optimizing the relationship between a first 2D profile in at least one 2D profile of the user's nose and a first 2D profile in at least one 2D profile of the nose segment of the frame associated with the nose pad of the candidate glasses, wherein the second 2D profile in at least one 2D profile of the user's nose is associated with the intersection of the eyeglass rim plane and the estimated 3D nose profile.

[0206] (8) The method of any one of (1) to (7), wherein generating a first 2D profile of at least one 2D profile of the user's nose comprises: generating a subsequent 2D profile of at least one 2D profile of the user's nose corresponding to a posterior nasal plane associated with an axis extending through the last aspect of the user's nose by means of the processing circuit system; and transforming the subsequent 2D profile of at least one 2D profile of the user's nose by means of the processing circuit system, the transformation resulting in the first 2D profile of at least one 2D profile of the user's nose, the transformation including a nose slant.

[0207] (9) The method of any one of (1) to (8), wherein at least one 2D profile of the nose section of the frame of the candidate eyeglasses includes a plurality of 2D profiles associated with the position of the movable nose pad of the candidate eyeglasses, each of the plurality of 2D profiles being a configuration of the candidate eyeglasses and representing a possible position of the movable nose pad of the candidate eyeglasses.

[0208] (10) The method of any one of (1) to (9), wherein the determination of the nose pad plane of the eyeglasses, the virtual docking, the comparison and the calculation are performed for each configuration of the candidate eyeglasses, each configuration of the candidate eyeglasses being an additional candidate eyeglasses.

[0209] (11) The method of any one of (1) to (10), wherein the anatomical features of the user are determined as the distance between the tempors of the user or the distance between the ears of the user.

[0210] (12) The method of any one of (1) to (11), wherein the spectacle lens plane is further defined by vertex distance.

[0211] (13) An apparatus for determining the fit of a visual device, the apparatus comprising: a processing circuit system configured to: receive at least one image of a user's face; identify a subset of eyeglasses from a visual device database that satisfy a preliminary fit threshold, the preliminary fit threshold being an anatomical feature of the user determined based on at least one image of the user's face; for candidate eyeglasses in the subset of eyeglasses, determine an eyeglass rim plane defined by a tilt angle; and, based on the eyeglass rim plane, determine an eyeglass nose pad plane intersecting with the nose pads of the candidate eyeglasses, the eyeglass nose pad plane being located at a position offset from the eyeglass rim plane by a predetermined distance; and, based on the user's face, determine an eyeglass rim plane intersecting with the nose pads of the candidate eyeglasses. The system generates at least one two-dimensional (2D) contour of the user's nose by detecting one or more facial landmarks in at least one image of the user's face and estimating a three-dimensional (3D) nose contour based on at least one image of the user's face. The at least one 2D contour of the user's nose includes at least one of the following: a first 2D contour of the at least one 2D contour of the user's nose, defined as the intersection of a plane parallel to the nose pad plane of the glasses and the estimated 3D nose contour; the first 2D contour of the at least one 2D contour located at a predetermined distance from a facial reference landmark among the one or more facial landmarks; and a first 2D contour of the at least one 2D contour of the user's nose. A second 2D profile is defined as the intersection of a plane parallel to the lens rim plane and the estimated 3D nose profile. The second 2D profile of the at least one 2D profile is positioned relative to the first 2D profile of the at least one 2D profile based on a predetermined distance of deviation between the nose pad plane and the lens rim plane. For candidate eyeglasses in the subset of eyeglasses, at least one 2D profile of the nose segment of the frame of the candidate eyeglasses is generated, wherein the first 2D profile of the at least one 2D profile of the nose segment of the frame is associated with the nose pad of the candidate eyeglasses, and the second 2D profile of the at least one 2D profile of the nose segment of the frame is associated with the nose pad of the candidate eyeglasses. The eyeglasses are associated with a lens rim; the candidate eyeglasses are virtually fitted onto the nose of the user's face based on at least one 2D profile of the user's nose and at least one 2D profile of the nose segment of the candidate eyeglasses frame; the position of a feature of the lens rim of the virtually fitted candidate eyeglasses is compared with a subset of one or more facial landmarks detected in at least one image of the user's face; and when the virtually fitted candidate eyeglasses meet a secondary fit threshold, the final fit of the virtually fitted candidate eyeglasses on the user's nose is calculated, the final fit of the virtually fitted candidate eyeglasses being defined by at least one of a comfort criterion and / or an aesthetic criterion.

[0212] (14) The device as described in (13), wherein the processing circuitry is further configured to combine the calculated values ​​of each of a plurality of comfort criteria and the calculated values ​​of each of a plurality of aesthetic criteria in order to calculate the final fit of the virtual docking candidate glasses on the user’s nose.

[0213] (15) The device as described in (13) or (14), wherein the processing circuitry is further configured to: rank each candidate glasses for virtual docking that meets the secondary fit threshold based on a comparison of the final fit of each candidate glasses for virtual docking; and generate a visual device recommendation for the user based on the ranking.

[0214] (16) The device as described in any one of (13) to (15), wherein the position of the virtual docking candidate glasses is calculated based on a second 2D profile in at least one 2D profile of the user's nose and a second 2D profile in at least one 2D profile of the nose segment of the frame associated with the eyeglass rim of the candidate glasses, by optimizing the relationship between a first 2D profile in at least one 2D profile of the user's nose and a first 2D profile in at least one 2D profile of the nose segment of the frame associated with the eyeglass rim of the candidate glasses, wherein the second 2D profile in at least one 2D profile of the user's nose is associated with the intersection of the eyeglass rim plane and the estimated 3D nose profile.

[0215] (17) The device of any one of (13) to (16), wherein the processing circuitry is configured to generate a first 2D profile of at least one 2D profile of the user's nose by: generating a subsequent 2D profile of at least one 2D profile of the user's nose corresponding to a posterior nasal plane associated with a vertical axis extending through the last aspect of the user's nose; and transforming the subsequent 2D profile of at least one 2D profile of the user's nose, the transformation resulting in the first 2D profile of at least one 2D profile of the user's nose, the transformation including a nose splay.

[0216] (18) The device of any one of (13) to (17), wherein at least one 2D profile of the nose section of the frame of the candidate eyeglasses includes a plurality of 2D profiles associated with the position of the movable nose pad of the candidate eyeglasses, each of the plurality of 2D profiles being a configuration of the candidate eyeglasses and representing a possible position of the movable nose pad of the candidate eyeglasses.

[0217] (19) The device as described in any one of (13) to (18), wherein the processing circuitry is further configured to perform the determination of the nose pad plane of the glasses, the virtual docking, the comparison, and the calculation for each configuration of the candidate glasses, each configuration of the candidate glasses being an additional candidate glasses.

[0218] (20) The device as described in any one of (13) to (19), wherein the anatomical features of the user are determined as the distance between the tempors of the user or the distance between the ears of the user.

[0219] (21) The device as described in any one of (13) to (20), wherein the eyeglass lens plane is further defined by vertex distance.

[0220] (22) A non-transitory computer-readable medium storing computer-readable instructions, which, when executed by a computer, cause the computer to perform a method for determining the fit of a visual device, the method comprising: receiving at least one image of a user's face; identifying a subset of eyeglasses from a visual device database that satisfy a preliminary fit threshold, the preliminary fit threshold being an anatomical feature of the user determined based on at least one image of the user's face; determining, for candidate eyeglasses in the subset of eyeglasses, an eyeglass rim plane defined by a tilt angle; and, based on the eyeglass rim plane, determining an eyeglass nose pad plane intersecting with the nose pads of the candidate eyeglasses, the eyeglass nose pad plane being located at the eye... The position of the lens ring plane deviating from a predetermined distance; based on one or more facial landmarks detected in at least one image of the user's face and a three-dimensional (3D) nose contour estimated based on at least one image of the user's face, at least one two-dimensional (2D) contour of the user's nose is generated, the at least one 2D contour of the user's nose including at least one of the following: a first 2D contour of the at least one 2D contour of the user's nose, defined as the intersection of a plane parallel to the nose pad plane of the glasses and the estimated 3D nose contour, the first 2D contour of the at least one 2D contour being located at a predetermined distance from a facial reference landmark among the one or more facial landmarks, and the use of A second 2D profile of at least one 2D profile of the nose of the user is defined as the intersection of a plane parallel to the lens rim plane and the estimated 3D nose profile. The second 2D profile of the at least one 2D profile is positioned relative to the first 2D profile of the at least one 2D profile based on a predetermined distance of deviation between the nose pad plane and the lens rim plane. For candidate eyeglasses in the subset of eyeglasses, at least one 2D profile of the nose segment of the frame of the candidate eyeglasses is generated, wherein the first 2D profile of the at least one 2D profile of the nose segment of the frame is associated with the nose pad of the candidate eyeglasses, and the second 2D profile of the at least one 2D profile of the nose segment of the frame is associated with... The candidate glasses are associated with the lens rims; the candidate glasses are virtually fitted onto the nose of the user's face based on at least one 2D profile of the user's nose and at least one 2D profile of the nose segment of the frame of the candidate glasses; the position of the features of the lens rims of the virtually fitted candidate glasses is compared with a subset of one or more facial landmarks detected in at least one image of the user's face; and when the virtually fitted candidate glasses meet a secondary fit threshold, the final fit of the virtually fitted candidate glasses on the user's nose is calculated, the final fit of the virtually fitted candidate glasses being defined by at least one of a comfort criterion and / or an aesthetic criterion.

[0221] Therefore, the foregoing discussion discloses and describes only exemplary embodiments. As those skilled in the art will understand, this disclosure can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Accordingly, this disclosure is intended to be illustrative and not to limit the scope of the invention and the other claims. This disclosure (including any readily identifiable variations of the teachings herein) partially defines the scope of the foregoing claims, leaving no inventive subject matter open to the public.

Claims

1. A method (105) for determining a fit of a visual device for a user to wear, the method comprising: receiving (110), by processing circuitry, at least one image of a face of the user; identifying (115), by the processing circuitry, a subset of eyeglasses from a database of visual devices that satisfy a preliminary fit threshold, the preliminary fit threshold being a limit of deviation of parameters of the eyeglasses from corresponding anatomical parameters of the face of the user determined based on the at least one image of the face of the user, wherein whether an eyeglass in the database satisfies the preliminary fit threshold is determined by comparing the parameters of the eyeglass to the corresponding anatomical parameters of the face of the user and eliminating the eyeglass if the preliminary fit threshold is exceeded, the comparison being related to comfort based on adaptability of eyeglasses to a width of a face of a user; determining (120), by the processing circuitry and for a candidate eyeglass in the subset of eyeglasses, an eyeglass rim plane defined by a pantoscopic angle; determining (125), by the processing circuitry and based on the eyeglass rim plane, an eyeglass nose pad plane that intersects an eyeglass nose pad of the candidate eyeglass, the eyeglass nose pad plane being located at a predetermined distance from the eyeglass rim plane; generating (130), by the processing circuitry, at least one two-dimensional, 2D, profile of a nose of the user based on one or more facial landmarks detected in the at least one image of the face of the user and a three-dimensional, 3D, nose profile estimated from the at least one image of the face of the user, the at least one 2D profile of the nose of the user comprising at least one of: a first 2D profile in the at least one 2D profile of the nose of the user defined as an intersection of a plane parallel to the eyeglass nose pad plane and the estimated 3D nose profile, the first 2D profile in the at least one 2D profile of the nose of the user being located at a predetermined distance from a facial reference landmark in the one or more facial landmarks, and a second 2D profile in the at least one 2D profile of the nose of the user defined as an intersection of a plane parallel to the eyeglass rim plane and the estimated 3D nose profile, the second 2D profile in the at least one 2D profile of the nose of the user being located relative to the first 2D profile in the at least one 2D profile based on the predetermined distance of the deviation of the eyeglass nose pad plane from the eyeglass rim plane; generating (140), by the processing circuitry and for a candidate eyeglass in the subset of eyeglasses, at least one 2D profile of a nose section of a frame of the candidate eyeglass, a first 2D profile in the at least one 2D profile of the nose section of the frame being associated with an eyeglass nose pad of the candidate eyeglass and a second 2D profile in the at least one 2D profile of the nose section of the frame being associated with an eyeglass rim of the candidate eyeglass; virtually docking (150), by the processing circuitry and based on the at least one 2D profile of the nose of the user and the at least one 2D profile of the nose section of the frame of the candidate eyeglass, the candidate eyeglass on the nose of the face of the user. comparing (160), by the processing circuitry, a position of a feature of a lens rim of a virtually docked candidate eyeglass with a subset of the one or more facial landmarks detected in the at least one image of the user's face; and calculating (170), by the processing circuitry and when the virtually docked candidate eyeglass satisfies a secondary fitment threshold representative of a distance between a global reference position of the eyeglass and the user's face obtained from the received at least one image, a final fitment of the virtually docked candidate eyeglass on the user's nose, the final fitment of the virtually docked candidate eyeglass being a score obtained by a scoring function resulting from: defining (171) a comfort criterion; defining (172) an evaluation function measuring the defined comfort criterion, wherein inputs include one or more morphological features of the user's face or eyeglasses, wherein the one or more morphological features of the user's face are determined from the received at least one image; defining (173) a target value representative of an ideal result of the evaluation function; and defining (174) a scoring function assessing a distance between a result of the evaluation function and the ideal result.

2. The method of claim 1, further comprising combining, by the processing circuitry and for calculating the final fitment of the virtually docked candidate eyeglass on the user's nose, a calculated value of each of a plurality of comfort criteria.

3. The method of claim 2, wherein, The combining includes: calculating, by the processing circuitry, a weighted sum of the calculated value of each of the plurality of comfort criteria.

4. The method of claim 1, further comprising iteratively performing, by the processing circuitry and for each remaining eyeglass of the subset of eyeglasses, the determining the eyeglass rim plane, the determining the eyeglass temple plane, the generating at least one 2D profile of a nose section of a frame of the candidate eyeglass, the virtually docking, the comparing, and the calculating.

5. The method of claim 4, further comprising ranking, by the processing circuitry, each virtually docked candidate eyeglass satisfying the secondary fitment threshold based on a comparison of the final fitment of each virtually docked candidate eyeglass.

6. The method of claim 5, further comprising generating, by the processing circuitry and based on the ranking, a visual device recommendation for the user.

7. The method of claim 1, wherein, The position of the virtually docked candidate eyeglass is calculated by optimizing a relationship between a first one of the at least one 2D profile of the user's nose and a first one of the at least one 2D profile of a nose section of the frame associated with a lens temple of the candidate eyeglass, the second one of the at least one 2D profile of the user's nose being associated with an intersection of the eyeglass rim plane and the estimated 3D nose profile.

8. The method of claim 1, wherein, The generating a first 2D profile of the at least one 2D profile of the user’s nose comprises: generating, by the processing circuitry, a subsequent 2D profile of the at least one 2D profile of the user’s nose corresponding to a posterior nasal plane of the user, the posterior nasal plane being associated with an axis extending through a last aspect of the user’s nose, and transforming, by the processing circuitry, the subsequent 2D profile of the at least one 2D profile of the user’s nose, the result of the transformation being the first 2D profile of the at least one 2D profile of the user’s nose, the transformation comprising a nasal splay angle.

9. The method of claim 4, wherein, The at least one 2D profile of the nosepiece segment of the candidate eyeglasses comprises a plurality of 2D profiles associated with positions of the eyeglasses mobile nose pads of the candidate eyeglasses, each of the plurality of 2D profiles being a configuration of the candidate eyeglasses and representing a possible position of the eyeglasses mobile nose pads of the candidate eyeglasses.

10. The method of claim 9, wherein, The determining (125) the eyeglasses nose pads plane, the virtual docking (150), the comparing (160) and the calculating (170) are performed for each configuration of the candidate eyeglasses, each configuration of the candidate eyeglasses being an additional candidate eyeglasses.

11. The method of claim 1, wherein, The anatomical feature of the user is determined as a distance between temples of the user or a distance between ears of the user.

12. The method of claim 1, wherein, The eyeglasses rim plane is further defined by a vertex distance.

13. A device for determining a fit of a visual equipment to be worn by a user, the device comprising: processing circuitry configured to: receive (110) at least one image of a face of a user, identify (115) a subset of eyeglasses from a database of visual equipment satisfying a preliminary fit threshold, the preliminary fit threshold being a limit of deviation of parameters of the eyeglasses from corresponding anatomical parameters of the face of the user determined based on the at least one image of the face of the user, wherein whether an eyeglass in the database satisfies the preliminary fit threshold is determined by comparing the parameters of the eyeglass with the corresponding anatomical parameters of the face of the user and eliminating the eyeglass if the preliminary fit threshold is exceeded, the comparison being related to comfort, based on adaptability of eyeglasses to a width of a face of a user, for a candidate eyeglass of the subset of eyeglasses, determine (120) an eyeglasses rim plane defined by a pantoscopic angle, based on the eyeglasses rim plane, determine (125) an eyeglasses nose pads plane intersecting eyeglasses nose pads of the candidate eyeglass, the eyeglasses nose pads plane being located at a position deviated from the eyeglasses rim plane by a predetermined distance, based on one or more facial landmarks detected in the at least one image of the face of the user and a three-dimensional, 3D, nose profile estimated from the at least one image of the face of the user, generate (130) at least one two-dimensional, 2D, profile of a nose of the user, the at least one 2D profile of the nose of the user comprising at least one of: a first one of the at least one 2D profile of the user's nose defined as an intersection of a plane parallel to the eyeglasses nose pad plane and the estimated 3D nose profile, the first one of the at least one 2D profile of the user's nose being located at a predetermined distance from a face reference landmark of the one or more face landmarks, and a second one of the at least one 2D profile of the user's nose defined as an intersection of a plane parallel to the eyeglasses rim plane and the estimated 3D nose profile, the second one of the at least one 2D profile of the user's nose being located relative to the first one of the at least one 2D profile of the user's nose based on a predetermined distance of the eyeglasses nose pad plane from the eyeglasses rim plane, generating (140), for each candidate eyeglasses of the subset of eyeglasses, at least one 2D profile of a nose section of a frame of the candidate eyeglasses, a first one of the at least one 2D profile of the nose section of the frame of the candidate eyeglasses being associated with an eyeglasses nose pad of the candidate eyeglasses and a second one of the at least one 2D profile of the nose section of the frame of the candidate eyeglasses being associated with an eyeglasses rim of the candidate eyeglasses, virtually docking (150) the candidate eyeglasses on the nose of the user's face based on the at least one 2D profile of the user's nose and the at least one 2D profile of the nose section of the frame of the candidate eyeglasses, comparing (160) a position of a feature of the eyeglasses rim of the virtually docked candidate eyeglasses with a subset of the one or more face landmarks detected in the at least one image of the user's face, and when the virtually docked candidate eyeglasses satisfy a secondary fitment threshold representative of a distance between a global reference position of the eyeglasses and the user's face obtained from the received at least one image, computing (170) a final fitment of the virtually docked candidate eyeglasses on the user's nose, the final fitment of the virtually docked candidate eyeglasses being a score obtained as a result of a scoring function: defining (171) a comfort criterion; defining (172) an evaluation function measuring the defined comfort criterion, wherein inputs include one or more morphological features of the user's face or eyeglasses, wherein the one or more morphological features of the user's face are determined from the received at least one image; defining (173) a target value representative of an ideal result of the evaluation function; and defining (174) a scoring function assessing a distance between a result of the evaluation function and the ideal result.

14. The apparatus of claim 13, wherein, the processing circuitry is further configured to: rank each virtually docked candidate eyeglasses satisfying the secondary fitment threshold based on a comparison of the final fitment of each virtually docked candidate eyeglasses, and generate, for the user, a visual device recommendation based on the ranking.

15. A non-transitory computer-readable medium having computer-readable instructions stored therein that, when executed by processing circuitry of a computer, cause the computer to perform a method for determining a fitment of a visual device for a user, the method comprising: receiving (110) at least one image of a face of a user; identifying (115), by processing circuitry, a subset of eyeglasses from a database of visual equipment that satisfy a preliminary fit threshold, the preliminary fit threshold being a limit of deviation of parameters of the eyeglasses from corresponding anatomical parameters of the face of the user determined based on the at least one image of the face of the user, wherein whether an eyeglass in the database satisfies the preliminary fit threshold is determined by comparing the parameters of the eyeglass with the corresponding anatomical parameters of the face of the user and eliminating the eyeglass if the preliminary fit threshold is exceeded, the comparison being related to comfort and based on the adaptability of the eyeglasses to the width of the face of the user; determining (120), for a candidate eyeglass in the subset of eyeglasses, an eyeglass rim plane defined by a pantoscopic angle; determining (125), based on the eyeglass rim plane, an eyeglass nose pad plane intersecting an eyeglass nose pad of the candidate eyeglass, the eyeglass nose pad plane being located at a predetermined distance from the eyeglass rim plane; generating (130), based on one or more facial landmarks detected in the at least one image of the face of the user and a three-dimensional, 3D, nose profile estimated from the at least one image of the face of the user, at least one two-dimensional, 2D, profile of a nose of the user, the at least one 2D profile of the nose of the user comprising at least one of: a first 2D profile in the at least one 2D profile of the nose of the user defined as an intersection of a plane parallel to the eyeglass nose pad plane and the estimated 3D nose profile, the first 2D profile in the at least one 2D profile being located at a predetermined distance from a facial reference landmark in the one or more facial landmarks, and a second 2D profile in the at least one 2D profile of the nose of the user defined as an intersection of a plane parallel to the eyeglass rim plane and the estimated 3D nose profile, the second 2D profile in the at least one 2D profile being located relative to the first 2D profile in the at least one 2D profile based on the predetermined distance of the eyeglass nose pad plane from the eyeglass rim plane; generating (140), for a candidate eyeglass in the subset of eyeglasses, at least one 2D profile of a nose section of a frame of the candidate eyeglass, a first 2D profile in the at least one 2D profile of the nose section of the frame being associated with an eyeglass nose pad of the candidate eyeglass and a second 2D profile in the at least one 2D profile of the nose section of the frame being associated with an eyeglass rim of the candidate eyeglass; virtually docking (150), based on the at least one 2D profile of the nose of the user and the at least one 2D profile of the nose section of the frame of the candidate eyeglass, the candidate eyeglass on the nose of the face of the user; comparing (160) a position of a feature of the eyeglass rim of the virtually docked candidate eyeglass with a subset of the one or more facial landmarks detected in the at least one image of the face of the user; and when the candidate virtually docked eyeglasses satisfy a secondary fitting threshold representative of a distance between a global reference position of the eyeglasses and the face of the user obtained from the at least one image received, calculating (170) a final fitting of the candidate virtually docked eyeglasses on the nose of the user, the final fitting of the candidate virtually docked eyeglasses being a score, the score being a result of a scoring function obtained by: defining (171) a comfort criterion; defining (172) an evaluation function measuring the defined comfort criterion, wherein inputs include one or more morphological features of the face of the user or of the eyeglasses, wherein the one or more morphological features of the face of the user are determined from the at least one image received; defining (173) a target value representative of an ideal result of the evaluation function; and defining (174) a scoring function evaluating a distance between a result of the evaluation function and the ideal result.

Images