Apparatus, method, and computer-readable storage medium for expanding an image database for evaluating eyewear compatibility

Abstract

The present disclosure relates to a method for expanding an image database for evaluating eyewear compatibility. Specifically, the method includes: receiving a user image; receiving a frame image; processing the received frame image by setting pixels of the received frame image other than the frame front face to transparent; defining a left boundary and a right boundary of the frame front face within the processed frame image, the left boundary and the right boundary corresponding to the left eye and the right eye, respectively; receiving a filter image; processing the received filter image by setting pixels of the received filter image other than the frame to transparent based on the left boundary and the right boundary; merging the processed frame image and the processed filter image; and superimposing the merged image onto the received user image.

Description

Technical Field

[0001] The present disclosure relates to eyewear and, in particular, to generating an image database for evaluating eyewear compatibility. Background Art

[0002] When choosing a new visual device or eyewear, users often need to self-examine to determine the aesthetics of the new eyewear on their face. Furthermore, when choosing between multiple eyewear options, users may find it difficult to determine which is most attractive, practical, or best suited to their specific facial bone structure and features. Simultaneously, users may struggle with their own perception of the new eyewear on their face and the perceived suitability of the new eyewear on their face by third parties (e.g., friends, family, professionals, etc.).

[0003] In order to provide an eyewear recommendation tool that, upon receiving an image of a user wearing an eyewear, generates a fit assessment or recommendation based on the unique characteristics of both the user and the eyewear, a large image database must be compiled so that the algorithm can be trained to generate accurate fit assessments even when presented with unknown user and eyewear combinations. Such a large image database may include thousands of images of users wearing eyewear and, understandably, would be laborious to assemble by acquiring photographs of every desired user and eyewear combination. For example, if the eyewear inventory includes twenty-five possible eyewear combinations, the corresponding database would likely include at least a multiple of twenty-five photographs of each eyewear combination worn by a given user. It will be appreciated that, even with the best efforts, acquiring actual photographs of every such combination is impractical, and even more so when considering the diverse range of ages, genders, ethnicities, and so on required to generate a comprehensive database.

[0004] In view of the foregoing, a need exists for a less labor-intensive method for expanding image databases used to assess eyewear compatibility, allowing comprehensive and accurate assessment of different combinations of users and eyewear. This disclosure describes a series of solutions to this end.

[0005] The foregoing "background" description is intended to generally introduce the context of the present disclosure. The work of the inventors (to the extent described in this background section) and aspects of the specification that may not be identified as prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art against the present invention. Summary of the Invention

[0006] The present disclosure relates to an apparatus, method, and computer-readable storage medium for expanding an image database used to assess the compatibility of eyewear.

[0007] According to an embodiment, the present disclosure further relates to a method for expanding an image database for evaluating eyewear compatibility, the method comprising: receiving a user image of a user's face; receiving a frame image of an eyewear frame from a plurality of eyewear frame images; processing, by a processing circuit system, the received frame image by setting pixels of the received frame image other than a frame front of the eyewear to be transparent; defining a left boundary and a right boundary of the frame front of the eyewear within the processed frame image, the defined left boundary and the defined right boundary corresponding to a left eye and a right eye, respectively; receiving a filter image of a filter that models a lens coating of the eyewear; processing, by the processing circuit system, the received filter image by setting pixels of the received filter image other than the frame of the eyewear to be transparent based on the defined left boundary and the defined right boundary of the frame front of the eyewear; merging the processed frame image and the processed filter image; and superimposing, by the processing circuit system, the merged processed frame image and the processed filter image onto the received user image of the user's face.

[0008] According to further embodiments which may be considered individually or in combination:

[0009] - said receiving comprises receiving a frame image of a frame of said eyewear and two or more images of temples of the frame of said eyewear, said two or more images of temples of the frame of said eyewear being utilized during said superposition; and / or

[0010] - the processing comprises processing the received frame image by centering the received frame image based on the received user image and a fitting model defining placement of the frame of the eyewear on the user's face; and / or

[0011] - said centering comprises centering said received frame image according to the orientation of said user's face in said received user image; and / or

[0012] - the processing includes processing the received frame image by scaling the received frame image based on the received user image and one or more scaling factors, the one or more scaling factors including a pupil factor, the pupil factor being a relationship between a real pupil distance and a digital pupil distance, the digital pupil distance being measured on the received frame image; and / or

[0013] - the digital pupillary distance is determined by calculating the distance between the pupils of the user's eyes, the pupils of the user's eyes being defined as the center of gravity of the palpebral fissure of the user's face; and / or

[0014] - the method further comprising processing the received user image of the user's face based on the defined left boundary and the defined right boundary of the front face of the frame of the eyewear, pixels in the received user image outside the defined left boundary and the defined right boundary being set to transparent; and / or

[0015] - the method further comprising applying a correction factor to the processed received user image based on the user's vision prescription; and / or

[0016] - the method further comprising applying a correction factor to the processed received user image based on a material of a frame of the eyewear in the processed frame image and a filter in the processed filter image; and / or

[0017] - Centering and scaling the received filter image according to the processed frame image.

[0018] According to an embodiment, the present disclosure further relates to an apparatus for expanding an image database for evaluating eyewear compatibility, the apparatus comprising a processing circuit system configured to: receive a user image of a user's face; receive a frame image of an eyewear frame from a plurality of eyewear frame images; process the received frame image by setting pixels of the received frame image other than a frame front of the eyewear to be transparent; define a left boundary and a right boundary of the frame front of the eyewear within the processed frame image, the defined left boundary and the defined right boundary corresponding to a left eye and a right eye, respectively; receive a filter image of a filter that models a lens coating of the eyewear; process the received filter image by setting pixels of the received filter image outside the frame of the eyewear to be transparent based on the defined left boundary and the defined right boundary of the frame front of the eyewear; merge the processed frame image and the processed filter image; and overlay the merged processed frame image and the processed filter image onto the received user image of the user's face.

[0019] According to further embodiments which may be considered individually or in combination:

[0020] - the processing circuitry is further configured to process the received frame image by centering the received frame image based on the received user image and a fit model defining placement of the frame of the eyewear on the user's face; and / or

[0021] - the processing circuitry is further configured to process the received frame image by scaling the received frame image based on the received user image and one or more scaling factors, the one or more scaling factors including a pupil factor, the pupil factor being a relationship between a true pupil distance and a digital pupil distance, the digital pupil distance being measured on the received frame image; and / or

[0022] - the processing circuit system is further configured to process the received user image of the user's face based on the defined left boundary and the defined right boundary of the front face of the frame of the eyewear, pixels in the received user image outside the defined left boundary and the defined right boundary being set to transparent; and / or

[0023] - the processing circuitry is further configured to apply a correction factor to the processed received user image based on the user's vision prescription; and / or

[0024] - the processing circuitry is further configured to center and scale the received filter image based on the processed frame image.

[0025] According to an embodiment, the present disclosure further relates to a non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a computer, cause the computer to perform a method for expanding an image database for evaluating eyewear compatibility, the method comprising: receiving an image of a user's face; receiving a frame image of an eyewear frame from a plurality of eyewear frame images; processing the received frame image by setting pixels of the received frame image other than a frame front face of the eyewear to be transparent; defining a frame front face of the eyewear within the processed frame image; a left boundary and a right boundary of a front face of the frame of the eyewear, the defined left boundary and the defined right boundary corresponding to a left eye and a right eye, respectively; receiving a filter image of a filter that models a lens coating of the eyewear; processing the received filter image by setting pixels in the received filter image that are outside the frame of the eyewear to be transparent based on the defined left boundary and the defined right boundary of the front face of the frame of the eyewear; merging the processed frame image and the processed filter image; and superimposing the merged processed frame image and processed filter image on the received user image of the user's face.

[0026] The foregoing paragraphs are provided as a general introduction and are not intended to limit the scope of the following claims.The described embodiments, together with further advantages, will be best understood by reference to the following detailed description taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] A more complete appreciation of the present disclosure and its many attendant advantages will be readily obtained as the same become better understood by reference to the following detailed description when considered in conjunction with the accompanying drawings, in which:

[0028] Figure 1 is an illustration of a user wearing eyeglass frames according to an exemplary embodiment of the present disclosure;

[0029] Figure 2 is an exemplary image database according to an exemplary embodiment of the present disclosure;

[0030] Figure 3 is a flow chart of a method for generating an eye care professional-based eye wear compatibility assessment according to an exemplary embodiment of the present disclosure;

[0031] Figure 4 is an illustration of a survey of eye care professionals according to an exemplary embodiment of the present disclosure;

[0032] Figure 5 is a flow chart of a method for expanding an image database for evaluating eyewear compatibility according to an exemplary embodiment of the present disclosure;

[0033] Figure 6A is a flow chart of sub-processes of a method for expanding an image database for evaluating eyewear compatibility according to an exemplary embodiment of the present disclosure;

[0034] Figure 6B is an illustration of a received frame image according to an exemplary embodiment of the present disclosure;

[0035] Figure 6C is an illustration of a processed frame image according to an exemplary embodiment of the present disclosure;

[0036] Figure 7 is a flow chart of sub-processes of a method for expanding an image database for evaluating eyewear compatibility according to an exemplary embodiment of the present disclosure;

[0037] Figure 8A is a flow chart of sub-processes of a method for expanding an image database for evaluating eyewear compatibility according to an exemplary embodiment of the present disclosure;

[0038] Figure 8B is an illustration of an optical filter image according to an exemplary embodiment of the present disclosure;

[0039] Figure 8C is an illustration of a processed frame image according to an exemplary embodiment of the present disclosure;

[0040] Figure 8D is an illustration of a processed filter image according to an exemplary embodiment of the present disclosure;

[0041] Figure 9A is an illustration of a processed frame image according to an exemplary embodiment of the present disclosure;

[0042] Figure 9B is an illustration of a processed filter image according to an exemplary embodiment of the present disclosure;

[0043] Figure 9C is an illustration of a merge of a processed filter image and a processed frame image according to an exemplary embodiment of the present disclosure;

[0044] Figure 10A is an exemplary image of a user according to an embodiment of the present disclosure;

[0045] Figure 10B is an illustration of a merge of a processed filter image and a processed frame image according to an exemplary embodiment of the present disclosure;

[0046] Figure 10C is an illustration of an exemplary image of a user superimposed with a merge of a processed filter image and a processed frame image according to an exemplary embodiment of the present disclosure;

[0047] Figure 11 is a flow chart of an exemplary embodiment of a method for expanding an image database for evaluating eyewear compatibility; and

[0048] Figure 12 is a hardware configuration of a database generating device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] As used herein, the terms "a" or "an" are defined as one or more than one. As used herein, the term "plurality" is defined as two or more than two. As used herein, the term "another" is defined as at least a second or more. As used herein, the terms "including" and / or "having" are defined as "comprising" (i.e., open language). The terms "eyewear," "equipment," "eyeglass frame," "eyeglass frames," and "visual equipment" may be used interchangeably to refer to a device having both a frame and lenses. The term "visual equipment" may be used to refer to a single visual equipment, while the term "visual equipments" may be used to refer to more than one visual equipment. Reference throughout this document to "one embodiment," "certain embodiments," "embodiments," "implementations," "examples," or similar terms means that the specific features, structures, or characteristics described in connection with the embodiment are included in at least one embodiment of the present disclosure. Similarly, the terms "facial image" and "facial image of a person" are corresponding terms that can be used interchangeably. Therefore, the appearance of such phrases throughout the specification or in various places does not necessarily refer to the same embodiment. In addition, the specific features, structures, or characteristics may be combined in any appropriate manner in one or more embodiments without limitation.

[0050] When selecting a new eyewear, a user may desire a third party to provide an opinion on the aesthetic as well as the structural suitability of the eyewear in terms of optical functionality. To provide robust recommendations, recommendation models based on statistical methods or other learning processes can be evaluated and trained on a reference database of 'labeled' images. For example, a Learning Discriminant Analysis (LDA) approach can be applied to a reference database to allow for automatic analysis of new images. Figure 1 and Figure 2, the reference database 202 may include 'tagged' images of users wearing eyewear 103, 203, the tags reflecting an assessment of the relationship between user features 204 and eyewear features 205. LDA may be trained to associate the positional relationship of eyewear features 205 to user features 204 within the 'tagged' images as acceptable or unacceptable. In an example, the 'tagged' images of the reference database may be tagged by a particular third-party group (e.g., eyecare professionals) and may include an assessment of a particular feature of the image or an assessment of the overall fit of the eyewear to the person's face in the image.

[0051] In order to provide accurate recommendations to different user groups, the number of representative 'tagged' images of the reference database needs to be increased and its combination needs to be expanded. To do this, the reference database needs to include a corpus of images encompassing various age groups, genders, ethnicities, etc. The reference database must also be dynamic and be able to include new eyewear designs, new features (e.g., anti-reflective coatings, colors, mirrors, etc.), new placement models / rules, and new adaptation models / rules. Figure 3 , when generating the recommendation model, the thousands of images 303 of the reference database 302 must now be evaluated by, for example, an eye care professional (ECP) 307. The results of the ECP evaluation 309 then provide an expert assessment 308 of the suitability of the eyewear for the user's face for each image 303 of the reference database 302. The exemplary ECP evaluation 408 is shown in FIG. Figure 4 4, wherein an image of a user wearing eyewear 403 can be graded according to specific criteria 410 (e.g., frame width versus face width) and / or overall criteria 411 (e.g., whether the eyewear fits properly). The ECP evaluation 408 can include questions regarding the compatibility between the user's face and the eyewear (e.g., shape, size, color, placement of the eyewear, fit of the eyewear, etc.).

[0052] In one embodiment, ECP evaluation 408 includes a series of images of a user wearing eyewear 403 and, along with each image, a series of questions regarding specific aspects of the eyewear's fit with the person's face. For each question, the ECP can be given a finite number of possible answers. In some examples, this can be a scale between -5 and +5, a scale between 0 and 10, or a choice of one of a set of N items. For example, as an example of questions and answers, the ECP can be asked: "How would you rate the width of the device relative to the width of the user's face?" (a) Too small, (b) Fine, or (c) Too large. In this way, images from the reference database can be 'labeled.'

[0053] However, it is understood that generating such a large reference database of 'labeled' images would be expensive, time-consuming, and largely impractical, requiring the acquisition of thousands of images of real people wearing actual eyewear. Therefore, the present disclosure provides an apparatus, method, and computer-readable storage medium for expanding an image database for evaluating eyewear compatibility. In this manner, the time and expense involved in creating and enriching a reference database can be reduced.

[0054] According to an embodiment, the present disclosure relates to a method for expanding an image database for evaluating eyewear compatibility. For example, a single image of a user's face can be modified to generate multiple images of the user's face wearing multiple eyewear designs, each processed according to one or more eyewear specifications, fit models, placement models, etc. Thus, the resulting multiple images can be uniquely evaluated by an ECP.

[0055] As described above, the method can be applied to a single image of a user's face. The method includes: receiving a user image of the user's face; receiving a frame image of an eyeglass frame from a plurality of eyeglass frame images; processing the received frame image by a processing circuit system; defining a left boundary and a right boundary of a front face of the eyeglass frame of the eyeglass within the processed frame image; receiving a filter image of a filter that models a lens coating of the eyeglass; processing the received filter image by the processing circuit system; merging the processed frame image and the processed filter image; and superimposing the merged processed frame image and the processed filter image onto the received user image of the user's face by the processing circuit system.

[0056] When generating multiple images or 'virtual try-on' images, the methods described herein minimize the number of real photos that need to be acquired. In one example, the methods described herein require only a single image of each user and a single image of each eyewear accessory. The images can be either profile or portrait images, with the single image for each user including reference measurements that enable rescaling of the image. The acquired images can then be processed in combination, in addition to a filtering model that reflects varying lens treatments or coatings, and a placement model and / or fit model that reflects the physical position of the eyewear accessory on the user's face. The placement and fit models can provide for different placement intersection rules, imperfections in height placement rules, imperfections in the position of the device on the user's face, imperfections in branch orientation, and so on. Height placement rules can be defined by the distance between the intersection placement point marked on each lens and the bottom of the bounding box defining the lens in the frame, as measured in the plane of the lens of the frame. When height placement is defined based on frame size and face size, these rules ensure that the frame is correctly positioned on the face. The branched orientation rule allows determining the orientation of the frames on the face in the user's profile image based on facial morphology (e.g., nose, cheeks, ears, etc.) and frame size. By considering combinations and variations of the above parameters (e.g., (1) the user's image, (2) the eyewear image, (3) the filter image, and (4) the placement model and / or the fitting model), the number of images in the reference database can be significantly larger than the number of real images that need to be acquired.

[0057] Referring again to the accompanying drawings, Figure 5 A flow chart is provided for a method for expanding a database for evaluating eyewear compatibility. Although presented as a single example, it is understood that Figure 5 The flowchart can be iteratively executed with countless combinations of the above parameters to generate a large reference database.

[0058] At step 530 of method 500, a user image may be received from an image database. The user image may be one of a plurality of user images within the image database. The user image may be a portrait of the user or a silhouette of the user and may include reference landmarks for scaling during processing.

[0059] At step 535 of method 500, an image of the eyewear or a frame image can be received from an image database. The frame image can be one of a plurality of frame images within the image database. The frame image can be a portrait of the frame or an outline of the frame.

[0060] At subprocess 540 of method 500, the received frame image may be processed. Processing the received frame image may include defining the edges of the frame and making transparent pixels that do not describe the frame. Figure 6A 、 Figure 6B and Figure 6C Sub-process 540 of method 500 will be described in more detail.

[0061] Optionally, at subprocess 545 of method 500, the received user image may be processed. Although not mandatory, processing the received user image may help adjust the appearance of the user's eyes according to the magnification of the lenses of the eyewear. Figure 7 Sub-process 545 of method 500 is described in more detail.

[0062] At step 555 of method 500, an image of the optical filter can be received from an image database. The optical filter image can be one of a plurality of optical filter images within the image database. In an example, the optical filter image can reflect lens coatings or lens treatments, including polarizers, thin films, mirrors, etc. The optical filter image can be a portrait of the optical filter or an outline of the optical filter.

[0063] At subprocess 560 of method 500, the received filter image may be processed. Processing the received filter image may include centering and scaling the filter image, clipping the filter image according to the defined edges of the frame, and making the identified areas of the filter image transparent. Figures 8A to 8D A more detailed description of sub-process 560 of method 500 is described below.

[0064] At step 570 of method 500, the frame image processed at subprocess 540 and the filter image processed at subprocess 560 may be merged into a merged filter frame image. Figures 9A to 9C Step 570 is explained in more detail.

[0065] At step 575 of method 500, the merged filter frame image may be superimposed on the received and optionally processed user image. Figures 10A to 10C Step 575 of method 500 is described in more detail.

[0066] At step 580 of method 500 , the overlay image may be stored in a reference database for subsequent evaluation by an ECP or the like.

[0067] The method 500 can be repeated iteratively until every possible combination of user image, frame image and filter image is considered. In addition, every possible combination can be further expanded by considering the placement model and the adaptation model.

[0068] Thus, the reference database generated at step 580 of method 500 can include multiple images of the user wearing the device, generated from a limited number of real user images, device images, and filter images. In this way, the burden of image acquisition is greatly reduced compared to traditional methods.

[0069] The above method 500 will now be considered in accordance with an embodiment in which the image of the user's face is a single image of a portrait of the user, the image of the eyewear is a portrait image of the eyewear, and the image of the filter is a portrait image of a filter that models a coating that may be applied to a lens of the eyewear. In an embodiment, the above image may include two or more images of the temples of the eyewear, so that when rendering the final image, these images may be interpreted to take perspective into account. Additionally, the above images may be further considered in accordance with a predetermined placement model and / or fit model. In an example, the placement model may define a height placement in millimeters as Where B is the inner vertical dimension of the circular member of the frame. In an example, the fitting model is a law that fixes the distance between the eye, the lens, and the pantoscopic angle (i.e., lens tilt) based on the morphological dimensions (e.g., nose width, nose orientation, ear positioning) and the frame dimensions (e.g., nose bridge width).

[0070] According to the embodiment, and as with respect to Figure 5 As introduced, the present disclosure provides a method for processing a frame image, processing a filter image, optionally processing a user image, optionally merging the processed user image with a user image, merging the processed frame image with a processed filter image, and superimposing the merged filter frame image with the optionally processed user image.

[0071] Reference Figure 6A 、 Figure 6B and Figure 6C , the frame image 613 may be processed according to sub-process 535 of method 500 .

[0072] At step 641 , the frame image 613 may be truncated relative to a bounding box of the frame defining the eyewear in the frame image 613 .

[0073] At step 642, each pixel that does not correspond to a component of the eyewear, including branches, shadows, etc., may be set to transparent. In an example, a branch may be a temple component of a frame. This includes pixels that are inside a 'circle' (which may be other shapes) of the eyewear as well as pixels that are outside of the 'circle' of the eyewear. The 'circle' may correspond to a lens of the eyewear. In an example, pixel association may be determined by image segmentation techniques including thresholding methods, region growing methods for identifying eyewear, etc. Figure 6B As shown, this may include temples 615 of the eyewear and a lens or manufacturer's logo 614 attached to the eyewear.

[0074] At step 643, the edge of the eyewear defining the 'circle' of the eyewear may be detected. The edge may define a hole into which a lens of the eyewear may be inserted. In an embodiment, the edge may be detected as C right and C left , where C right Corresponds to the right circular piece of the eyewear, and C left Corresponding to the left circular portion of the eyewear. Edges can be detected using image segmentation techniques, including active contour-based segmentation techniques, region growing methods, and the like. In one example, edge detection can be performed using a Canny edge detector. As defined above, the circular portion of the eyewear will serve as the processing boundary during subsequent sub-processes.

[0075] At step 644, the frame image 613 may be centered and scaled based on the received user image, taking into account a predetermined placement model and / or fit model. The frame image 613 may be centered and scaled to correct the orientation of the user image and the frame image 613 to account for differences in perspective, and the frames in the frame image 613 may be appropriately sized relative to the user's face in the user image. The placement model and / or fit model may account for the likely position of the frames in the frame image 613 relative to the user's face in the user image. Scaling the frame image 613 may be performed according to one or more of a plurality of scaling factors. Scaling factors may also include using a single ubiquitous reference landmark of known size in the image or using differential measurements between features in the image. For example, where the scaling factor is based on a screw head as a single ubiquitous reference landmark, a real-world size value of the screw head in millimeters may be correlated to a digital size value of the screw head in pixels. In another example where the scaling factor is based on pupillary distance as a differential measurement between features in an image, the true pupillary distance in millimeters can be measured by a pupillometer and correlated to the digital pupillary distance in pixels measured on the user image. In embodiments, facial detection methods and landmark estimation methods can be appropriately applied to the user image, frame image 613, and filter image. In embodiments, with respect to pupillary measurements and the user image, the user's eyelid margin can be detected in the user image and its center of gravity can be calculated. The center of gravity can then be used as an estimate of the pupil center and correlated to the known pupillary distance measured by the pupillometer and used as a scaling factor. The true dimensions determined above can be extrapolated to other images in a similar manner.

[0076] In an example, scaling can be performed to adjust the size of the frame image to the size of the user image. This can be based on the pupil distance in the user image and the frame width in the frame image, each measured as described above. Normalizing one measured distance to the other results in a scaling factor that can be applied.

[0077] like Figure 6C As shown, the result of sub-process 540 of method 500 is a processed frame image 616. Processed frame image 616 may then be used in method 500.

[0078] Reference Figure 7 The user image received at step 530 may optionally be processed according to sub-process 545 of method 500 .

[0079] At step 746 of sub-process 545, the defined segment C′ may be defined based on the placement model and / or the adaptation model. right and C' left and the frame edge C identified in subprocess 540 of method 500 right and C left The user image 746 is located.

[0080] At step 747 of sub-process 545, and similarly to step 642 of sub-process 540, the user image is not defined by the segment C'. right and C' left The described area can be set to transparent. C' right and C' left In an embodiment, the user image can be viewed by evaluating the pixels of the user image and identifying the pixels of the user image. right and C left Defined outside the frame edge and therefore not within the defined section C' right and C' left The transparency can be modified by identifying pixels within the image. These identified pixels can be set to transparent. In an example, pixel identification can be determined by image segmentation techniques including thresholding methods, region growing methods for identifying eyewear, and the like.

[0081] At step 748 of sub-process 545, the correction factor may be applied to the defined region C' that remains in the processed user image. right and C' left By applying the correction factors, the processed image of the user can reflect the visual effects of magnifying or minifying the user's eyes through the lenses used in the eyewear.

[0082] According to an embodiment, the processed user image from step 748 of sub-process 545 can then be merged with the user image received at step 530 to generate a processed user image, which can ultimately be merged with the merged filter frame image, as will be described later. Because sub-process 545 of method 500 is optional, the remainder of method 500 will be described in addition to this sub-process.

[0083] Reference Figure 8A 、 Figure 8B 、 Figure 8C and Figure 8D , the filter image 818 received at step 555 of method 500 may be processed according to sub-process 560 of method 500. In an embodiment, the filter image 818 may represent an image of the environment in which the eyewear will be utilized, the image of the environment being a reflection perceived by an external observer. Such an environment may be, for example, Figure 8B The illustrated indoor environment may alternatively be an outdoor environment. In a non-limiting example, the environment may also be a sunny outdoor location where the reflection of the sun appears on the polarized lens 'filter'.

[0084] At step 861 of subprocess 560, the received filter image 818 may be centered and scaled according to the processed frame image of subprocess 540 of method 500. The processed frame image 816 is shown in FIG. Figure 8C Shown in.

[0085] At step 862 of sub-process 560 , and using the bounding box as defined in sub-process 540 of method 500 , the received filter image 818 may be truncated relative to the bounding box of the frame defining the eyewear in the processed frame image 816 .

[0086] At step 863 of sub-process 560, and based on the frame edge C defined in sub-process 540 of method 500, right and C left , all pixels outside the frame edge can be set to transparent. In other words, the filter image 818 is not defined as C' right and C' left All pixels within the segment of the frame image 816 can be identified and set to transparent. In an example, pixel identification can be determined by image segmentation techniques including thresholding methods, region growing methods for identifying eyewear, and the like. Thus, as Figure 8D As shown, processed filter image 819 represents the effect of filter image 818 on each lens of the eyewear.

[0087] The processed filter image of subprocess 560 may then be provided to step 570 of method 500. Figure 9A and Figure 9B As shown, the processed frame image 916 can be merged with the processed filter image 919 to generate a merged filter frame image 920, as shown in FIG. Figure 9CIn an embodiment, the merging of the processed frame image 916 and the processed filter image 919 may be performed according to methods understood by those skilled in the art.

[0088] To generate an image ready for storage in the reference database at step 580 of method 500, the merged filter frame image of step 570 of method 500 and the frame image received at step 530 of method 500 may be merged. Figure 10A 、 Figure 10B and Figure 10C , the user image 1022 and the merged filter frame image 1020 may be merged to generate Figure 10C In other words, the merged filter frame image 1020 can be superimposed on the user image 1022 or otherwise merged therewith to generate an overlay image 1023. Figure 10C The overlay image 1023 of step 575 shown in FIG. 5 is an exemplary illustration of an image of a user wearing eyewear that may be generated by the method 500 of the present disclosure.

[0089] According to an embodiment, the overlay image generated at step 575 of method 500 may then be stored in a reference database at step 580 for further evaluation by the ECP during development of a recommendation model based on the reference database.

[0090] Now refer to Figure 11 , an exemplary flow chart of the present disclosure will be described. Figure 11 The flowchart is different from Figure 5 The diagram is presented as a linear flow chart to illustrate the non-limiting nature of the flow chart and what relationships are important.

[0091] Initially, the frame image 1113 may be processed 1116 to identify the bounding box, frame edges C right and C left , and prepare to later merge the frame with the processed filter image. The frame edge C is called the frame boundary 1143 right and C leftThis can be used to generate processed filter image 1119 and processed user image 1145. Next, when generating processed filter image 1119, filter image 1118 can be centered and scaled based on the processed frame image, such that the transparent areas of filter image 1118 are defined using the same bounding box and frame boundary 1143. Placement rules and / or fit rules 1125 may also be considered. Processed frame image 1116 and processed filter image 1119 can then be merged to generate a merged filter frame image 1170. Next, processed user image 1145 can be generated from user image 1122. Processed user image 1145 can be generated based on frame boundary 1143 and the user's vision prescription. Again, placement rules and / or fit rules 1125 may also be considered. Processed user image 1145 can then be merged with user image 1122 to generate a merged user image 1149. After generating the merged filter frame image 1170 and the merged user image 1149, the images can be finally merged to generate an overlay image. When merging the images, the merged filter frame image 1170 can be overlaid on the merged user image 1149. The overlay image 1175 can then be stored in a reference database for subsequent evaluation by the ECP during development of the recommendation model.

[0092] According to an embodiment, the method of the present disclosure generates a reference database of 'labeled' images by digitally combining images of a limited number of users, images of eyewear, and images of filters according to a variation of a placement model and / or an adaptation model in an 'artificial' extended reference database. To this end, the method generates N images by merging images of eyewear with images of filters. g *N f Image, where N g is the number of images of eyewear, and N f is the number of filter images. These merged images may contain at least four channels, three for color and one for transparency. Depending on the filter images, the area of ​​the merged image outside the eyewear frame, including the temples of the eyewear, may be made transparent, and the area of ​​the merged image inside the eyewear frame may be made translucent, transparent, or opaque. The method then generates N images that superimpose a set of merged images with the received user image of the user's face. g *N f *N a *N w Image, where N w is the number of images of the user, and N aIndicates consideration of placement and / or fit models. By considering more than one placement and / or fit model, a continuous model of the structure, position, and placement suitability of the eyewear on the user's face can be generated based on expert advice, wherein the structural positioning of the eyewear on the user's face simulates different types of placement or fit errors. It can be appreciated that the methods of the present disclosure allow for a multiplied impact by providing a single image of the user, thereby eliminating the need to acquire thousands of images of users wearing different eyewear.

[0093] Now refer to Figure 12 , Figure 12 is a hardware description of a database generation device according to an exemplary embodiment of the present disclosure.

[0094] exist Figure 12 In the embodiment of the present invention, the database generation device includes a CPU 1285 that performs the above-described processes. The database generation device can be a general-purpose computer or a specific, dedicated machine. In one embodiment, when the processor 1285 is programmed to perform visual device selection (and in particular, any of the processes discussed with reference to the above disclosure), the database generation device becomes a specific, dedicated machine.

[0095] Alternatively or additionally, as will be appreciated by one of ordinary skill in the art, the CPU 1285 may be implemented on an FPGA, ASIC, PLD, or using discrete logic circuits. Further, the CPU 1285 may be implemented as multiple processors working in parallel to execute the instructions of the above-described inventive process.

[0096] The database generation device also includes a network controller 1288, such as an Intel Ethernet PRO network interface card, for interfacing with a network 1299. As will be appreciated, the network 1299 can be a public network such as the Internet, or a private network such as a LAN or WAN network, or any combination thereof, and can also include a PSTN or ISDN subnetwork. The network 1299 can also be wired (such as an Ethernet network) or wireless (such as a cellular network, including EDGE, 3G, and 4G wireless cellular systems). The wireless network can also be WiFi, Bluetooth, or any other known form of wireless communication.

[0097] The database generation device further includes a display controller 1289, such as a graphics card or graphics adapter, for interfacing with a display 1290 (e.g., a monitor). A general-purpose I / O interface 1291 interfaces with a keyboard and / or mouse 1292, and a touchscreen panel 1293, which may be on or separate from the display 1290. The general-purpose I / O interface 1291 is also connected to various peripheral devices 1294, including printers and scanners.

[0098] A sound controller 1295 is also provided in the database generating device to interface with a speaker / microphone 1296 to provide sound and / or music.

[0099] The general storage controller 1297 connects the storage medium disk 1287 to the communication bus 1298 to interconnect all components of the database generation device. The communication bus may be ISA, EISA, VESA, PCI, etc. For the sake of brevity, descriptions of the general features and functions of the display 1290, keyboard and / or mouse 1292, display controller 1289, storage controller 1297, network controller 1288, sound controller 1295, and general I / O interface 1291 are omitted here because these features are already known.

[0100] The exemplary circuit elements described in the context of this disclosure may be replaced with other elements and constructed in a manner different from the examples provided herein. In addition, the circuit systems configured to perform the features described herein may be implemented in multiple circuit units (e.g., chips), or these features may be combined in circuit systems on a single chipset.

[0101] Function and feature described herein can also be performed by the various distributed components of system.For example, one or more processors can perform these system functions, wherein the processor is distributed on a plurality of components with network communication.In addition to including various human-computer interaction devices and communication equipment (for example, display monitors, smart phones, tablet computers, personal digital assistants (PDAs)), distributed components can also include one or more clients and server machines that can share processing.The network can be a dedicated network such as a LAN or WAN, or can be a public network such as the Internet.The input to the system can be received via direct user input, and can be received remotely in real time or as a batch process.Additionally, some embodiments can be performed on modules or hardware different from the described modules or hardware.Therefore, other embodiments are within the scope that can be claimed.

[0102] Obviously, many modifications and variations are possible in light of the above teachings.It is therefore to be understood that within the scope of the appended claims, the invention may be practiced otherwise than as specifically described herein.

[0103] Embodiments of the present disclosure may also be described as follows in brackets.

[0104] (1) A method for expanding an image database for evaluating eyewear compatibility, the method comprising: receiving a user image of a user's face; receiving a frame image of an eyewear frame from a plurality of eyewear frame images; processing, by a processing circuit system, the received frame image by setting pixels of the received frame image other than a frame front of the eyewear to be transparent; defining a left boundary and a right boundary of the frame front of the eyewear within the processed frame image, the defined left boundary and the defined right boundary corresponding to a left eye and a right eye, respectively; receiving a filter image of a filter that models a lens coating of the eyewear, processing, by the processing circuit system, the received filter image by setting pixels of the received filter image other than the frame of the eyewear to be transparent based on the defined left boundary and the defined right boundary of the frame front of the eyewear; merging the processed frame image and the processed filter image; and superimposing, by the processing circuit system, the merged processed frame image and the processed filter image onto the received user image of the user's face.

[0105] (2) The method of (1), wherein the receiving comprises receiving a frame image of the frame of the eyewear and two or more images of temples of the frame of the eyewear, the two or more images of the temples of the frame of the eyewear being utilized during the superposition.

[0106] (3) The method of (1) or (2), wherein the processing includes processing the received frame image by centering the received frame image based on the received user image and a fitting model that defines positioning of the frame of the eyewear on the user's face.

[0107] (4) The method of (1) to (3), wherein the centering includes centering the received frame image according to the orientation of the user's face in the received user image.

[0108] (5) The method of (1) to (4), wherein the processing includes processing the received frame image by scaling the received frame image based on the received user image and one or more scaling factors, the one or more scaling factors including a pupil factor, the pupil factor being a relationship between a true pupil distance and a digital pupil distance, the digital pupil distance being measured on the received frame image.

[0109] (6) The method of (1) to (5), wherein the digital pupillary distance is determined by calculating the distance between the pupils of the user's eyes, the pupils of the user's eyes being defined as the center of gravity of the palpebral fissure of the user's face.

[0110] (7) The method as described in any one of (1) to (6), further includes processing the received user image of the user's face based on the defined left boundary and the defined right boundary in front of the frame of the eyewear, and pixels outside the defined left boundary and the defined right boundary in the received user image are set to transparent.

[0111] (8) The method of any one of (1) to (7), further comprising applying a correction factor to the processed received user image based on the user's vision prescription.

[0112] (9) The method as described in any one of (1) to (8), further comprising applying a correction factor to the processed received user image based on the material of the frame of the eyewear in the processed frame image and the filter in the processed filter image.

[0113] (10) An apparatus for expanding an image database for evaluating eyewear compatibility, the apparatus comprising a processing circuit system configured to: receive a user image of a user's face; receive a frame image of an eyewear frame from a plurality of eyewear frame images; process the received frame image by setting pixels of the received frame image other than a frame front of the eyewear to be transparent; define a left boundary and a right boundary of the frame front of the eyewear within the processed frame image, the defined left boundary and the defined right boundary corresponding to a left eye and a right eye, respectively; receive a filter image of a filter that models a lens coating of the eyewear, process the received filter image by setting pixels of the received filter image other than the frame of the eyewear to be transparent based on the defined left boundary and the defined right boundary of the frame front of the eyewear; merge the processed frame image and the processed filter image; and superimpose the merged processed frame image and the processed filter image on the received user image of the user's face.

[0114] (11) The apparatus of (10), wherein the processing circuit system is further configured to process the received frame image by centering the received frame image based on the received user image and a fitting model that defines positioning of the frame of the eyewear on the user's face.

[0115] (12) An apparatus as described in (10) or (11), wherein the processing circuit system is further configured to process the received frame image by scaling the received frame image based on the received user image and one or more scaling factors, the one or more scaling factors including a pupil factor, the pupil factor being a relationship between a true pupil distance and a digital pupil distance, the digital pupil distance being measured on the received frame image.

[0116] (13) An apparatus as described in any one of (10) to (12), wherein the processing circuit system is further configured to process the received user image of the user's face based on the defined left boundary and the defined right boundary in front of the frame of the eyewear, and pixels in the received user image outside the defined left boundary and the defined right boundary are set to transparent.

[0117] (14) The apparatus of any one of (10) to (13), wherein the processing circuit system is further configured to apply a correction factor to the processed received user image based on the user's vision prescription.

[0118] (15) A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a computer, cause the computer to perform a method for expanding an image database for evaluating eyewear compatibility, the method comprising: receiving an image of a user's face; receiving a frame image of an eyewear frame from a plurality of eyewear frame images; processing the received frame image by setting pixels of the received frame image other than a frame front face of the eyewear to be transparent; defining a left boundary and a right boundary of the frame front face of the eyewear within the processed frame image; The invention also provides a method for manufacturing a user face of a user's face comprising: receiving a filter image of a filter that models a lens coating of the eyewear, processing the received filter image by setting pixels in the received filter image that are outside the frame of the eyewear to be transparent based on the defined left boundary and the defined right boundary on the front of the frame of the eyewear; merging the processed frame image and the processed filter image; and superimposing the merged processed frame image and the processed filter image on the received user image of the user's face.

[0119] (16) The method of any one of (1) to (9), wherein the processing includes processing the received frame image by scaling the received frame image based on the received user image and one or more scaling factors, the one or more scaling factors including a frame factor, the frame factor being the relationship between an actual width of the frame of the eyewear and a corresponding pixel width of the frame of the eyewear in the received frame image.

[0120] Therefore, the foregoing discussion discloses and describes only exemplary embodiments of the present invention. As will be appreciated by those skilled in the art, the present invention may be embodied in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the disclosure of the present invention is intended to be illustrative, rather than limiting, the scope of the present invention and the scope of the other claims. This disclosure (including any readily discernible variations of the teachings herein) partially defines the scope of the aforementioned claim terms so that no creative subject matter is dedicated to the public.

Claims

1. A method for expanding an image database for evaluating eyewear compatibility, the method comprising: receiving a user image of the user's face; receiving a frame image of an eyewear objective lens frame from a plurality of eyewear objective lens frame images; processing, by the processing circuitry, the received frame image by setting pixels of the received frame image other than a front face of the frame of the eyewear to be transparent; defining a left boundary and a right boundary of a front face of a frame of the eyewear within the processed frame image, the defined left boundary and the defined right boundary corresponding to a left eye and a right eye, respectively; receiving a filter image of an optical filter modeling a lens coating of the ophthalmic accessory; processing, by the processing circuitry, the received filtered image by setting pixels in the received filtered image that are outside of the frame of the eyewear to be transparent based on the defined left boundary and the defined right boundary of the front face of the frame of the eyewear; merging the processed frame image and the processed filter image; as well as The processing circuit system superimposes the combined processed frame image and the processed filter image onto the received user image of the user's face, wherein the step of processing the received frame image comprises processing the received frame image by centering the received frame image based on the received user image and a fitting model defining placement of the frame of the eyewear on the user's face; wherein the step of processing the received frame image comprises processing the received frame image by scaling the received frame image based on the received user image and one or more scaling factors, the one or more scaling factors comprising a pupil factor, the pupil factor being a relationship between a true pupil distance and a digital pupil distance measured on the received frame image; and The received filter image is centered and scaled according to the processed frame image.

2. The method according to claim 1, wherein The step of receiving a frame image of an eyewear object frame from a plurality of eyewear object frame images includes receiving a frame image of the eyewear object frame and two or more images of temples of the eyewear object frame, wherein the two or more images of the temples of the eyewear object frame are utilized during the superposition.

3. The method according to claim 1, wherein The centering includes centering the received frame image according to an orientation of the user's face in the received user image.

4. The method according to claim 1, wherein The digital pupillary distance is determined by the following operation: The distance between the pupils of the user's eyes is calculated, the pupils of the user's eyes being defined as the center of gravity of the palpebral fissures of the user's face.

5. The method of claim 1 , further comprising processing the received user image of the user's face based on the defined left boundary and the defined right boundary on the front of the frame of the eyewear, wherein pixels in the received user image outside the defined left boundary and the defined right boundary are set to transparent.

6. The method of claim 5, further comprising applying a correction factor to the processed received user image based on the user's vision prescription.

7. The method of claim 5, further comprising applying a correction factor to the processed received user image based on a material of a frame of the eyewear in the processed frame image and a filter in the processed filter image.

8. An apparatus for expanding an image database for evaluating eyewear compatibility, the apparatus comprising: processing circuitry configured to receiving a user image of the user's face, receiving a frame image of an eyewear objective lens frame from a plurality of eyewear objective lens frame images, processing the received frame image by setting pixels of the received frame image other than a front face of a frame of the eyewear to be transparent, defining a left boundary and a right boundary of a front face of a frame of the eyewear within the processed frame image, the defined left boundary and the defined right boundary corresponding to a left eye and a right eye, respectively, receiving a filter image of an optical filter modeling a lens coating of the ophthalmic accessory, processing the received filter image by setting pixels in the received filter image that are outside of the frame of the eyewear to be transparent based on the defined left boundary and the defined right boundary of the front face of the frame of the eyewear, Merging the processed frame image and the processed filter image, and superimposing the combined processed frame image and the processed filter image onto the received user image of the user's face, wherein the processing circuitry is further configured to process the received frame image by centering the received frame image based on the received user image and a fit model that defines placement of the frame of the eyewear on the user's face; wherein the processing circuitry is further configured to process the received frame image by scaling the received frame image based on the received user image and one or more scaling factors, the one or more scaling factors including a pupil factor, the pupil factor being a relationship between a true pupil distance and a digital pupil distance, the digital pupil distance being measured on the received frame image; and Wherein the processing circuit system is further configured to center and scale the received filter image based on the processed frame image.

9. The device according to claim 8, wherein The processing circuit system is further configured to process the received user image of the user's face based on the defined left boundary and the defined right boundary on the front of the frame of the eyewear, and pixels in the received user image outside the defined left boundary and the defined right boundary are set to transparent.

10. The device according to claim 9, wherein The processing circuitry is further configured to apply a correction factor to the processed received user image based on the user's vision prescription.

11. A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a computer, cause the computer to perform a method for expanding an image database for evaluating eyewear compatibility, the method comprising: receiving a user image of the user's face; receiving a frame image of an eyewear objective lens frame from a plurality of eyewear objective lens frame images; processing the received frame image by setting pixels of the received frame image other than a front face of a frame of the eyewear to be transparent; defining a left boundary and a right boundary of a front face of a frame of the eyewear within the processed frame image, the defined left boundary and the defined right boundary corresponding to a left eye and a right eye, respectively; receiving a filter image of an optical filter modeling a lens coating of the ophthalmic accessory; processing the received filter image by setting pixels in the received filter image that are outside of the frame of the eyewear to be transparent based on the defined left boundary and the defined right boundary of the front face of the frame of the eyewear; merging the processed frame image and the processed filter image; as well as superimposing the combined processed frame image and the processed filter image onto the received user image of the user's face, wherein the step of processing the received frame image comprises processing the received frame image by centering the received frame image based on the received user image and a fitting model defining placement of the frame of the eyewear on the user's face; wherein the step of processing the received frame image comprises processing the received frame image by scaling the received frame image based on the received user image and one or more scaling factors, the one or more scaling factors comprising a pupil factor, the pupil factor being a relationship between a true pupil distance and a digital pupil distance measured on the received frame image; and The received filter image is centered and scaled according to the processed frame image.

Images