System and method for fitting corrective lenses to an eyewear frame

Abstract

Systems and methods of determining fitting parameters for fitting corrective lenses to an eyewear frame, including: receiving a plurality of images of a face of a subject wearing the eyewear frame, with cameras positioned to capture images of the subject from different directions, detecting contours of at least one of lenses and rims of the eyewear frame, and determining a set of points along the contours, based on positions of the cameras in a reference coordinate system, determining three-dimensional coordinates of the set of points in the reference coordinate system, and based on the three-dimensional coordinates of the set of points, determining fitting parameters for fitting corrective lenses to the eyewear frame.

Description

[0001] SYSTEM AND METHOD FOR FITTING CORRECTIVE LENSES TO AN

[0002] EYEWEAR FRAME

[0003] FIELD OF THE INVENTION

[0004]

[0001] The present invention relates to identification or prediction of contours / edges of corrective lenses. More particularly, the present invention relates to systems and methods for determining fitting parameters for fitting corrective lenses to an eyewear frame based on the contours.

[0005] BACKGROUND OF THE INVENTION

[0006]

[0002] Personal fitting of corrective lenses is carried out to ensure that the corrective optics of the lenses are precisely aligned with the visual axis of the user's eyes as they naturally wear the spectacles, thereby delivering optimal visual acuity and comfort.

[0007]

[0003] In order to correctly fit corrective lenses to a particular user, several characteristics can be taken into account, including the user's optical prescription, their unique facial and / or anatomical measurements, and the specific geometry of their chosen eyewear frame. Eye care practitioners (ECPs) typically need to supply the fitting parameters for that user, since eyewear frames can vary in shape and / or size, which can affect the positioning of the corrective lens relative to the eyes of the user.

[0008]

[0004] The fitting parameters can include face parameters, frame parameters, and faceframe relative parameters. The face parameters can include the pupil distance (PD), as the horizontal distance between the centers of the pupils that is measured for both far and near vision to ensure the optical center of each lens is placed directly in the user's line of sight for the intended viewing distance. In case of wrong measurements, misalignment can induce unwanted prismatic effects, leading to eye strain and distorted vision.

[0009]

[0005] The frame parameters can include parameters such as the distance between lenses (DBL) as the shortest distance between the nasal edges of the lenses in a frame, and the frame box to fit the lenses to the frame.

[0010]

[0006] The face-frame relative parameters can include the fitting or optical center (OC) height as the vertical measurement from the bottom of the lens in the frame to the center of the user's pupil. This measurement can determine the correct corridor for transitioning between distance, intermediate, and / or near vision zones.

[0007] The face-frame relative parameters can also include a back vertex distance (BVD) as the distance from the back surface of the lens to the front of the cornea, a mono pupil distance (MPD), a pantoscopic angle / tilt as the angle at which the bottom of the frame is tilted towards the user's cheeks, and / or the face form or a panoramic angle / tilt as the curvature of the frame as it wraps around the user's face.

[0011]

[0008] These fitting parameters are typically measured manually by the ECP, but such manual measurement process can be subjective and susceptible to mistakes.

[0012]

[0009] Existing tools (e.g., with a semi -automated process) can require manually marking points of interest on the image of the wearer wearing the chosen frame or use of a calibration “jig” in order to accurately obtain fitting parameters measurements. The jig is fastened to a frame of the user’s eyewear in order to orient the objects in the image and construct a 3D model, and in addition, to mark important frame elements, e.g. the bottom or top of the frame. These markings enable measurement of face-frame relative parameters, and the frame box.

[0013]

[0010] Some tools that do not require a jig rely on reference points positioned by manually moving and setting digital markers on the image.

[0014]

[0011] It can therefore be advantageous to perform automatic fitting of corrective lenses to an eyewear frame, without the need for an external cumbersome jig that interferes with the natural wearing of the frame by the wearer and with no interventions of the ECP thereby reducing the manual errors and time by the ECP.

[0015] SUMMARY OF THE INVENTION

[0016]

[0012] There is thus provided, in accordance with some embodiments of the invention, a method of determining fitting parameters for fitting corrective lenses to an eyewear frame, the method including, using a computing device operating a processor: receiving a plurality of images of a face of a subject wearing the eyewear frame, the images being taken by a plurality of cameras, wherein each camera is positioned to capture images of the subject from different directions, based on the plurality of images: detecting contours of lenses and / or contours of the inner shape of the rims of the eyewear frame, and determining a set of points along the contours, based on positions of the cameras in a reference coordinate system, determining three-dimensional coordinates of the set of points in the reference coordinate system, and based on the three-dimensional coordinates of the set of points, determining fitting parameters for fitting corrective lenses to the eyewear frame.

[0017]

[0013] In some embodiments, detecting the contours includes detecting masks of the at lenses and / or masks for the inner shape of the rims of the eyewear, and based on the masks, detecting the contours. In some embodiments, detecting the masks includes providing the images as an input to a machine learning model, the machine learning model being trained to detect shapes of the lenses and / or shapes for the inner shape of the rims of eyewear frames in images and output masks corresponding to the detected shapes.

[0018]

[0014] In some embodiments, a plurality of face landmarks of the face of the subject is detected, and one or more points of the set of points is determined based on the face landmarks. In some embodiments, the face landmarks include a left pupil and a right pupil.

[0019]

[0015] In some embodiments, the left pupil and the right pupil are detected based on images taken by cameras capable of capturing an infrared light. In some embodiments, the contours are detected based on images taken by cameras capable of capturing visible light.

[0020]

[0016] In some embodiments, the set of points along the contour include points selected from a group including: a first point corresponding to a topmost point of a left lens or rim, a second point corresponding to a lowermost point of the left lens or rim, a third point corresponding to a leftmost point of the left lens or rim, a fourth point corresponding to a rightmost point of the left lens or rim, a fifth point corresponding to a leftmost point of a right lens or rim, a sixth point corresponding to a rightmost point of the right lens or rim, a seventh point disposed below a pupil of the left eye on the left lens or rim, an eighth point disposed above the pupil of the left eye on the left lens or rim, and a ninth point disposed below a pupil of the right eye on the right lens or rim.

[0021]

[0017] In some embodiments, the fitting parameters are selected from a group including: a distance between a left pupil and a right pupil of the subject, a distance between the lenses, a width of at least one lens, a height of at least one lens, a fitting height, a back vertex distance, a mono pupil distance, a pantoscopic angle, and a panoramic angle.

[0022]

[0018] There is thus provided, in accordance with some embodiments of the invention, a system for determining fitting parameters for fitting corrective lenses to an eyewear frame, the system including: a plurality of cameras, a computing device including: a memory, and a processor to: receive a plurality of images of a face of a subject wearing the eyewear frame, the images being taken by a plurality of cameras, wherein each camera is positioned to capture images of the subject from different directions, based on the plurality of images: detect contours of lenses and / or contours of the inner shape of the rims of the eyewear frame, and determine a set of points along the contours, based on positions of the cameras in a reference coordinate system, determine three-dimensional coordinates of the set of points in the reference coordinate system, and based on the three- dimensional coordinates of the set of points, determine fitting parameters for fitting corrective lenses to the eyewear frame.

[0023]

[0019] In some embodiments, in order to detect the contours, the processor is to: detect masks of the lenses and / or masks for the inner shape of the rims of the eyewear, and based on the masks, detect the contours. In some embodiments, in order to detect the masks, the processor is to provide the images as an input to a machine learning model, the machine learning model being trained to detect shapes of the lenses and / or shapes for the inner shape of the rims of eyewear frames in images and output masks corresponding to the detected shapes.

[0024]

[0020] In some embodiments, the processor is to: detect a plurality of face landmarks of the face of the subject, and determine one or more points of the set of points based on the face landmarks.

[0025]

[0021] In some embodiments, the face landmarks include a left pupil and a right pupil. In some embodiments, the plurality of cameras includes cameras capable of capturing an infrared light, and wherein the processor to detect the left pupil and the right pupil based on images taken by the cameras capable of capturing the infrared light. In some embodiments, the plurality of cameras includes cameras capable of capturing a visible light, and wherein the processor to detect the contours based on images taken by the cameras capable of capturing the visible light.

[0026]

[0022] In some embodiments, the set of points along the contour includes points selected from a group including: a first point corresponding to a topmost point of a left lens or rim, a second point corresponding to a lowermost point of the left lens or rim, a third point corresponding to a leftmost point of the left lens or rim, a fourth point corresponding to a rightmost point of the left lens or rim, a fifth point corresponding to a leftmost point of a right lens or rim, a sixth point corresponding to a rightmost point of the right lens or rim, a seventh point disposed below a pupil of the left eye on the left lens or rim, an eighth point disposed above the pupil of the left eye on the left lens or rim and a ninth point disposed below a pupil of the right eye on the right lens or rim.

[0027]

[0023] In some embodiments, the fitting parameters are selected from a group including: a distance between a left pupil and a right pupil of the subject, a distance between the lenses, a width of at least one lens, a height of at least one lens, a fitting height, a back vertex distance, a mono pupil distance, a pantoscopic angle, and a panoramic angle.

[0028] BRIEF DESCRIPTION OF THE DRAWINGS

[0029]

[0024] The subject matter regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of the specification. The invention, however, both as to organization and method of operation, together with objects, features and advantages thereof, may best be understood by reference to the following detailed description when read with the accompanied drawings. Embodiments of the invention are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like reference numerals indicate corresponding, analogous or similar elements, and in which:

[0030]

[0025] Fig. 1 shows a block diagram of a computing device, according to some embodiments of the invention;

[0031]

[0026] Fig. 2 shows a block diagram of a system for determining fitting parameters for fitting corrective lenses to an eyewear frame, according to some embodiments of the invention;

[0032]

[0027] Fig. 3 shows a flowchart for determining fitting parameters using the ML algorithm, according to some embodiments of the invention;

[0033]

[0028] Figs. 4A-4E show examples of the image processing, according to some embodiments of the invention; and

[0034]

[0029] Fig. 5 shows a flowchart for a method of determining fitting parameters for fitting corrective lenses to an eyewear frame, according to some embodiments of the invention.

[0030] It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements.

[0035] DETAILED DESCRIPTION

[0036]

[0031] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details.

[0037]

[0032] In other instances, well-known methods, procedures, and components, modules, units and / or circuits have not been described in detail so as not to obscure the invention. Some features or elements described with respect to one embodiment may be combined with features or elements described with respect to other embodiments. For the sake of clarity, discussion of same or similar features or elements may not be repeated.

[0038]

[0033] Although embodiments of the invention are not limited in this regard, discussions utilizing terms such as, for example, “processing”, “computing”, “calculating”, “determining”, “establishing”, “analyzing”, “checking”, or the like, may refer to operation(s) and / or process(es) of a computer, a computing platform, a computing system, or other electronic computing device, that manipulates and / or transforms data represented as physical (e.g., electronic) quantities within the computer’s registers and / or memories into other data similarly represented as physical quantities within the computer’s registers and / or memories or other information non-transitory storage medium that may store instructions to perform operations and / or processes.

[0039]

[0034] Although embodiments of the invention are not limited in this regard, the terms “plurality” and “a plurality” as used herein may include, for example, “multiple” or “two or more”. The terms “plurality” or “a plurality” may be used throughout the specification to describe two or more components, devices, elements, units, parameters, or the like. The term set when used herein may include one or more items.

[0040]

[0035] Unless explicitly stated, the method embodiments described herein are not constrained to a particular order or sequence. Additionally, some of the described method embodiments or elements thereof may occur or be performed simultaneously, at the same point in time, or concurrently.

[0036] Reference is made to Fig. 1, which is a block diagram of an example computing device, according to some embodiments of the invention. Computing device 100 may include a controller or processor 105 (e.g., a central processing unit processor (CPU), a chip or any suitable computing or computational device), an operating system 115, memory 120, executable code 125, storage 130, input devices 135 (e.g. a keyboard or touchscreen), and output devices 140 (e.g., a display), a communication unit 145 (e.g., a cellular transmitter or modem, a Wi-Fi communication unit, or the like) for communicating with remote devices via a communication network, such as, for example, the Internet.

[0041]

[0037] Controller 105 may be configured to execute program code to perform operations described herein. The system described herein may include one or more computing device(s) 100, for example, to act as the various devices or the components shown in Fig. 2. For example, communication system 200 may be, or may include computing device 100 or components thereof.

[0042]

[0038] Operating system 115 may be or may include any code segment (e.g., one similar to executable code 125 described herein) designed and / or configured to perform tasks involving coordinating, scheduling, arbitrating, supervising, controlling or otherwise managing operation of computing device 100, for example, scheduling execution of software programs or enabling software programs or other modules or units to communicate.

[0043]

[0039] Memory 120 may be or may include, for example, a Random Access Memory (RAM), a read only memory (ROM), a Dynamic RAM (DRAM), a Synchronous DRAM (SD-RAM), a double data rate (DDR) memory chip, a Flash memory, a volatile memory, a non-volatile memory, a cache memory, a buffer, a short term memory unit, a long term memory unit, or other suitable memory units or storage units. Memory 120 may be or may include a plurality of similar and / or different memory units. Memory 120 may be a computer or processor non-transitory readable medium, or a computer non-transitory storage medium, e.g., a RAM.

[0044]

[0040] Executable code 125 may be any executable code, e.g., an application, a program, a process, task or script. Executable code 125 may be executed by controller 105 possibly under control of operating system 115. For example, executable code 125 may be a software application that performs methods as further described herein.

[0041] Although, for the sake of clarity, a single item of executable code 125 is shown in Fig. 1, a system according to embodiments of the invention may include a plurality of executable code segments similar to executable code 125 that may be stored into memory 120 and cause controller 105 to carry out methods described herein.

[0045]

[0042] Storage 130 may be or may include, for example, a hard disk drive, a universal serial bus (USB) device or other suitable removable and / or fixed storage unit. In some embodiments, some of the components shown in Fig. 1 are omitted. For example, memory 120 may be a non-volatile memory having the storage capacity of storage 130. Accordingly, although shown as a separate component, storage 130 may be embedded or included in memory 120.

[0046]

[0043] Input devices 135 may be or may include a keyboard, a touch screen or pad, one or more sensors or any other or additional suitable input device. Any suitable number of input devices 135 may be operatively connected to computing device 100. Output devices 140 may include one or more displays or monitors and / or any other suitable output devices. Any suitable number of output devices 140 may be operatively connected to computing device 100.

[0047]

[0044] Any applicable input / output (VO) devices may be connected to computing device 100 as shown by blocks 135 and 140. For example, a wired or wireless network interface card (NIC), a universal serial bus (USB) device or external hard drive may be included in input devices 135 and / or output devices 140.

[0048]

[0045] Embodiments of the invention may include an article such as a computer or processor non-transitory readable medium, or a computer or processor non-transitory storage medium, such as for example a memory, a disk drive, or a USB flash memory, encoding, including or storing instructions, e.g., computer-executable instructions, which, when executed by a processor or controller, carry out methods disclosed herein. For example, an article may include a storage medium such as memory 120, computerexecutable instructions such as executable code 125 and a controller such as controller 105.

[0049]

[0046] Such a non-transitory computer readable medium may be for example a memory, a disk drive, or a USB flash memory, encoding, including or storing instructions, e.g., computer-executable instructions, which when executed by a processor or controller, carry out methods disclosed herein.

[0047] The storage medium may include, but is not limited to, any type of disk including, semiconductor devices such as read-only memories (ROMs) and / or random-access memories (RAMs), flash memories, electrically erasable programmable read-only memories (EEPROMs) or any type of media suitable for storing electronic instructions, including programmable storage devices. For example, in some embodiments, memory 120 is a non-transitory machine-readable medium.

[0050]

[0048] A system according to embodiments of the invention may include components such as, but not limited to, a plurality of central processing units (CPUs), a plurality of graphics processing units (GPUs), or any other suitable multi-purpose or specific processors or controllers (e.g., controllers similar to controller 105), a plurality of input units, a plurality of output units, a plurality of memory units, and a plurality of storage units. A system may additionally include other suitable hardware components and / or software components.

[0051]

[0049] In some embodiments, a system may include or may be, for example, a personal computer, a desktop computer, a laptop computer, a workstation, a server computer, a network device, or any other suitable computing device.

[0052]

[0050] For example, a system as described herein may include one or more facility computing device 100 and one or more remote server computers in active communication with one or more facility computing device 100 such as computing device 100, and in active communication with one or more portable or mobile devices such as smartphones, tablets and the like.

[0053]

[0051] According to some embodiments, systems and methods are provided with an automatic measurement tool for both accurate detection of the wearer’s frame and accurate measurement of the required fitting parameters.

[0054]

[0052] To enable accurate positioning for automatic fitting parameters measurements, an accurate depiction of the contour of the lenses may be determined to enable fitting into the chosen frame. By accurately predicting the actual lens contour mounted in the wearer’s eyewear frame, an automatic determination of the required fitting parameters may be achieved.

[0055]

[0053] Reference is now made to Fig. 2, which shows a block diagram of a system 200 for determining fitting parameters 222 for fitting corrective lenses to an eyewear frame, according to some embodiments of the invention. In Fig. 2, hardware elements are indicated with a solid line and the direction of arrows indicate a direction of information flow between the hardware elements.

[0056]

[0054] The system 200 may include a plurality of cameras 201a-201e and a computing device 210 that may be configured to process information from the plurality of cameras 201a-201e to determine the fitting parameters 222 for fitting corrective lenses to an eyewear frame.

[0057]

[0055] The plurality of cameras 201a-201e may be positioned to capture images of a subject 20 wearing the eyewear frame, such that each camera is positioned to capture images from different directions. In some embodiments, the positions 202 of the cameras 201a-201e may be registered and stored, for instance positions in a reference coordinate system, for further analysis later on. For example, the origin of the reference coordinate system may be the point between the pupils.

[0058]

[0056] In some embodiments, the plurality of cameras 201a-201e may capture four RGB images, by cameras capable of capturing visible light, and at least two Infra-Red (IR) images, by cameras capable of capturing an infrared light, of the subject 20 such that a set of six images may be taken simultaneously. Each image provided by one of the plurality of cameras 201a-201e may labeled by the specific camera that captured this image.

[0059]

[0057] For example, the plurality of cameras 201a-201e may include four RGB cameras positioned at top, bottom, left, right of the subject’s face, and two IR cameras and / or illuminators positioned at sides of the subject’s face.

[0060]

[0058] The computing device 210 may include a memory 211 (e.g., such as memory 120 shown in Fig. 1) and a processor 220 (e.g., such as controller 105 shown in Fig. 1). For example, the memory 211 may store the positions 202 of the cameras 201a-201e in the reference coordinate system.

[0061]

[0059] In some embodiments, the processor 220 receives (e.g., wirelessly or via a direct connection) a plurality of images 212 of a face of the subject 20 wearing the eyewear frame, with the images 212 being taken by the plurality of cameras 201a-201e.

[0062]

[0060] The processor 220 may detect contours 213 of contours of the lenses and / or contours of the inner shape of the rims of the eyewear frame, based on the received plurality of images 212 for instance by using at least one image processing algorithm.

[0061] The processor 220 may determine a set of points 214 along the detected contours 213, such that the set of points 214 may define the lenses and / or rims of the eyewear frame. In some embodiments, one or more mask of the contours of the lenses and / or the contours of the inner shape of the rims of the eyewear may be determined such that the contours 213 may be determined based on the one or more mask (e.g., as described in further detail below). The mask may identify at least one region of interest in the image.

[0062] The processor 220 may determine three-dimensional (3D) coordinates 216 of the set of points 214 in the reference coordinate system, based on positions 202 of the cameras.

[0063]

[0063] The processor 220 may determine fitting parameters 222 for fitting corrective lenses to the eyewear frame, based on the determined 3D coordinates 216. Thus, the system 200 allows prediction of an ophthalmic lens contour mounted in a chosen eyewear frame based on images taken by the cameras.

[0064]

[0064] In some embodiments, the fitting parameters may be selected from a group including: a distance between a left pupil and a right pupil of the subject, a distance between the lenses, a width of at least one lens, a height of at least one lens, a fitting height, a back vertex distance, a mono pupil distance, a pantoscopic angle, and a panoramic angle.

[0065]

[0065] In some embodiments, some fitting parameters are transmitted to the machine(s) that fabricate the lens and / or cut it to shape of the frame so it may be inserted to the frame in the desired position according to the fitting parameters.

[0066]

[0066] According to some embodiments, the processor 220 may employ at least one deep machine learning (ML) algorithm for prediction of the fitting parameters 222. For example, the ML algorithm may be a convolutional neural network such as a U-Net model that may be used for image segmentation tasks to accurately predict the boundaries of objects within an image.

[0067]

[0067] Reference is now made to Fig. 3, which shows a flowchart for determining fitting parameters using the ML algorithm, according to some embodiments of the invention.

[0068]

[0068] The ML algorithm may be trained to identify contours or shapes of the lenses and / or the inner rims of eyewear frames in images, and accordingly output masks corresponding to the detected shapes.

[0069] For example, the ML algorithm may be trained to detect lens contours or shapes by feeding the ML algorithm with predefined annotations of various contours. This training may enable the ML algorithm to match a lens mask to an input image and convert it back to and / or inferred as a lens contour that matches the contour of the lens within the frame of the subject’s face in the image.

[0069]

[0070] The plurality of images 212 may be received 30, by the processor 220 (e.g., as shown in Fig. 2) to be fed as input to the ML algorithm. For example, at least nine points of interest may be identified in each image for the required contour in order to accurately determine the fitting parameters for producing corrective ophthalmic lenses fitted to a specific frame.

[0070]

[0071] The ML algorithm may be applied, by the processor, with the received plurality of images as input, to determine at least one mask corresponding to the detected shape.

[0071]

[0072] In order to improve accuracy of the ML algorithm, the training (ore preprocessing) stage may include annotating images of eyewear frames as worn by various subjects, and creating a shape corresponding to the frame contour (e.g., using at least 52 annotation points). For example, the annotation may be carried out for the two lenses and on two separate images, such that two independent shapes are obtained. These shapes may be converted into masks by the ML algorithm.

[0072]

[0073] In some embodiments, the determined masks may be stored in the memory of the computing device and / or stored at a server in communication with the computing device.

[0074] During input of images by the processor 220, one or more of the plurality of cameras 201a-201e (e.g., as shown in Fig. 2) may provide two IR images to be fed to the ML algorithm. For example, the selected IR images may be the ones where the light returned from the pupils is most distinct.

[0073]

[0075] These images may be analyzed (e.g., using image processing or by the ML algorithm) to determine two-dimensional (2D) location of the pupils in the images. In some embodiments, the determined 2D location may be re-sized or calibrated in order to synchronize back with the original image of the subject, for instance to be used again in additional analysis of these images.

[0074]

[0076] The processor 220 may also receive four RGB images for the ML algorithm, such that a 2D lens mask may be predicted for determining the lens contour for each image. The resulting 2D lens masks may undergo re-sizing and / or un-cropping that synchronize back with the original image size of the subject, for instance to be used again in additional analysis of these images. For example, each 2D lens mask may be labeled with the corresponding camera that captured the image from which the mask was determined.

[0075]

[0077] According to some embodiments, at least nine feature-revealing points may be identified on each detected and / or predicted lens contour, where each point or set of points may be related to a different fitting and / or frame parameter for a specific eye. In order to identify these points, a plurality of face landmarks of the subject’s face may be detected and one or more points of the set of points based may be determined on the face landmarks. For example, the face landmarks may include a left pupil and a right pupil.

[0076]

[0078] Reference is now made to Figs. 4A-4E, which show examples of the image processing, according to some embodiments of the invention.

[0077]

[0079] Fig. 4A shows an RGB photo captured (by one of the four RGB cameras) to be fed into the ML model, and Fig. 4B shows a corresponding 2D lens mask predicted to each lens of the subject’s eyewear.

[0078]

[0080] Fig. 4C shows the extracted contour of the frame, and Fig. 4D shows the mask difference validation based on the extracted contour of the frame.

[0079]

[0081] Fig. 4E shows the nine points identified on the detected contours. The set of points along the contour may include points selected from a group including: a first point 401 corresponding to a topmost point of a left lens or rim, a second point 402 corresponding to a lowermost point of the left lens or rim, a third point 403 corresponding to a leftmost point of the left lens or rim, a fourth point 404 corresponding to a rightmost point of the left lens or rim, a fifth point 405 corresponding to a leftmost point of a right lens or rim, a sixth point 406 corresponding to a rightmost point of the right lens or rim, a seventh point 407 disposed below a pupil of the left eye on the left lens or rim, an eighth point 408 disposed above the pupil of the left eye on the left lens or rim, and a ninth point 409 disposed below a pupil of the right eye on the right lens or rim.

[0080]

[0082] For example, the nine points may be identified as the topmost point on subject’s left eye (e.g., relevant for lens shape height), the lowest point on subject’s left eye (e.g., relevant for lens shape width and / or height), the leftmost point on subject’s left eye (e.g., relevant for distance between lenses), the rightmost point on subject’s left eye (e.g., relevant for panoramic tilt), the leftmost point on subject’s right eye (e.g., relevant for panoramic tilt), the rightmost point on subject’s right eye (e.g., relevant for distance between lenses), the point under the left pupil (e.g., defined by crossing the predicted lens shape location with the predicted pupil and relevant for back vertex distance and pupil distance), the point above the left pupil (e.g., relevant for pantoscopic tilt), and the point under the right pupil (e.g., relevant for pupil distance). In some embodiments, based on these definitions the nine points, it may be possible to apply the ML algorithm to predict various contours.

[0081]

[0083] The 3D location of each of the feature-revealing points may be determined using epipolar geometry based stereoscopic deduction (for the relationship between two views of a 3D scene taken by two cameras), with a pair of camera images per each point being selected based on the nature of the specific point for optimal results. For example, the left and right camera images may be used for points relating to features with a horizontal nature (e.g., such as distance between lenses or panoramic tilt) and the top and bottom camera images may be used for points relating to features with vertical nature (e.g., such as fitting height and pantoscopic tilt). Once the 3D locations are identified, the fitting information may be determined based on the coordinated of the points along with those of the pupils.

[0082]

[0084] Reference is now made to Fig. 5, which shows a flowchart for a method of determining fitting parameters for fitting corrective lenses to an eyewear frame, according to some embodiments of the invention.

[0083]

[0085] A plurality of images of a face of a subject wearing the eyewear frame is received 501, the images being taken by a plurality of cameras, wherein each camera is positioned to capture images of the subject from different directions. Based on the plurality of images, contours of lenses and / or contours of the inner shape of the rims of the eyewear frame is detected 502, and a set of points along the contours is determined 503.

[0084]

[0086] Based on positions of the cameras in a reference coordinate system, three- dimensional coordinates of the set of points in the reference coordinate system are determined 504. Based on the three-dimensional coordinates of the set of points, fitting parameters for fitting corrective lenses to the eyewear frame are determined 505.

[0085]

[0087] While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes, and equivalents may occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes.

[0086]

[0088] Various embodiments have been presented. Each of these embodiments may of course include features from other embodiments presented, and embodiments not specifically described may include various features described herein.

Claims

CLAIMS1. A method of determining fitting parameters for fitting corrective lenses to an eyewear frame, the method comprising, using a computing device operating a processor: receiving a plurality of images of a face of a subj ect wearing the eyewear frame, the images being taken by a plurality of cameras, wherein each camera is positioned to capture images of the subject from different directions; based on the plurality of images: detecting contours of lenses or contours of the inner shape of the rims of the eyewear frame; and determining a set of points along the contours; based on positions of the cameras in a reference coordinate system, determining three-dimensional coordinates of the set of points in the reference coordinate system; and based on the three-dimensional coordinates of the set of points, determining fitting parameters for fitting corrective lenses to the eyewear frame.

2. The method of claim 1, wherein detecting the contours comprises: detecting masks of the lenses or masks for the inner shape of the rims of the eyewear; and based on the masks, detecting the contours.

3. The method of claim 2, wherein detecting the masks comprises providing the images as an input to a machine learning model, the machine learning model being trained to detect shapes of the lenses or shapes for the inner shape of the rims of eyewear frames in images and output masks corresponding to the detected shapes.

4. The method of claim 1, comprising: detecting a plurality of face landmarks of the face of the subject; and determining one or more points of the set of points based on the face landmarks.

5. The method of claim 4, wherein the face landmarks comprise a left pupil and a right pupil.

6. The method of claim 5, comprising detecting the left pupil and the right pupil based on images taken by cameras capable of capturing an infrared light.

7. The method of claim 1, comprising detecting the contours based on images taken by cameras capable of capturing visible light.

8. The method of claim 1, wherein the set of points along the contour comprises points selected from a group comprising: a first point corresponding to a topmost point of a left lens or rim; a second point corresponding to a lowermost point of the left lens or rim; a third point corresponding to a leftmost point of the left lens or rim; a fourth point corresponding to a rightmost point of the left lens or rim; a fifth point corresponding to a leftmost point of a right lens or rim; a sixth point corresponding to a rightmost point of the right lens or rim; a seventh point disposed below a pupil of the left eye on the left lens or rim; an eighth point disposed above the pupil of the left eye on the left lens or rim; and a ninth point disposed below a pupil of the right eye on the right lens or rim.

9. The method of claim 1, wherein the fitting parameters are selected from a group comprising: a distance between a left pupil and a right pupil of the subject; a distance between the lenses; a width of at least one lens; a height of at least one lens; a fitting height; a back vertex distance; a mono pupil distance; a pantoscopic angle; and a panoramic angle.

10. A system for determining fitting parameters for fitting corrective lenses to an eyewear frame, the system comprising: a plurality of cameras; a computing device comprising: a memory; and a processor to: receive a plurality of images of a face of a subject wearing the eyewear frame, the images being taken by a plurality of cameras, wherein each camera is positioned to capture images of the subject from different directions; based on the plurality of images: detect contours of lenses or contours of the inner shape of the rims of the eyewear frame; and determine a set of points along the contours; based on positions of the cameras in a reference coordinate system, determine three-dimensional coordinates of the set of points in the reference coordinate system; and based on the three-dimensional coordinates of the set of points, determine fitting parameters for fitting corrective lenses to the eyewear frame.

11. The system of claim 10, wherein in order to detect the contours, the processor is to: detect masks of the lenses and / or masks for the inner shape of the rims of the eyewear; and based on the masks, detect the contours.

12. The system of claim 11, wherein in order to detect the masks, the processor is to provide the images as an input to a machine learning model, the machine learning model being trained to detect shapes of the lenses and / or shapes for the inner shape of the rims of eyewear frames in images and output masks corresponding to the detected shapes.

13. The system of claim 10, wherein the processor is to: detect a plurality of face landmarks of the face of the subject; and determine one or more points of the set of points based on the face landmarks.

14. The system of claim 13, wherein the face landmarks comprise a left pupil and a right pupil.

15. The system of claim 14, wherein the plurality of cameras comprises cameras capable of capturing an infrared light, and wherein the processor to detect the left pupil and the right pupil based on images taken by the cameras capable of capturing the infrared light.

16. The system of claim 10, wherein the plurality of cameras comprises cameras capable of capturing a visible light, and wherein the processor to detect the contours based on images taken by the cameras capable of capturing the visible light.

17. The system of claim 1, wherein the set of points along the contour comprises points selected from a group comprising: a first point corresponding to a topmost point of a left lens or rim; a second point corresponding to a lowermost point of the left lens or rim; a third point corresponding to a leftmost point of the left lens or rim; a fourth point corresponding to a rightmost point of the left lens or rim; a fifth point corresponding to a leftmost point of a right lens or rim; a sixth point corresponding to a rightmost point of the right lens or rim; a seventh point disposed below a pupil of the left eye on the left lens or rim; an eighth point disposed above the pupil of the left eye on the left lens or rim; and a ninth point disposed below a pupil of the right eye on the right lens or rim.

18. The system of claim 1, wherein the fitting parameters are selected from a group comprising: a distance between a left pupil and a right pupil of the subject; a distance between the lenses;a width of at least one lens; a height of at least one lens; a fitting height; a back vertex distance; a mono pupil distance; a pantoscopic angle; and a panoramic angle.

Images