A combined photographic and thermal imaging method for obtaining fitting parameters to fit ophthalmic lenses into spectacle frames
By combining photography and thermal imaging technologies, the outline of the glasses is identified and superimposed onto the photograph, solving the problem of difficulty in identifying the bottom edge of frameless glasses lenses in poor lighting conditions, and achieving accurate acquisition of fitting parameters.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ESSILOR INTERNATIONAL(COMPAGNIE GENERALE D OPTIQUE)
- Filing Date
- 2022-05-25
- Publication Date
- 2026-05-26
AI Technical Summary
Existing technology struggles to accurately identify the bottom edge of rimless glasses lenses in poor lighting conditions or when there is low contrast between skin tone and frame color, making it difficult to determine the appropriate fitting height.
By combining photography and thermal imaging technologies, the system captures photos and thermal images of eyeglass wearers, uses image processing algorithms to identify the outline of the eyeglasses, and overlays them onto the photos to obtain fitting parameters.
It improves the clarity of identifying the shape of glasses in poor lighting conditions and ensures accurate acquisition of fitting parameters such as fitting height.
Smart Images
Figure CN117321484B_ABST
Abstract
Description
Technical Field
[0001] This disclosure pertains generally to the field of ophthalmic optics. More specifically, it relates to the manufacture, ordering, and adjustment of eyeglass frames, as well as the accurate installation of eyeglass lenses into frames. Background Technology
[0002] Document WO 2020 / 064755 A1 discloses a method for determining the fitting height of eyeglass frames for a given wearer, with particular reference to page 16, line 4 and below, and Figure 2 and... Figure 4 In this method, the camera 12 of a smartphone 10 is used to take a photograph of the wearer's head 50 with the selected eyeglass frame. The fitting height is then determined by measuring the distance between the center of the pupil and the bottom edge of the frame or corresponding lens in the photograph.
[0003] The drawback of this method is that it can sometimes be difficult to clearly identify the bottom edge in a photograph. In fact, in poor lighting conditions and / or when the contrast between skin tone and frame color is low, the given eyeglass frame can be barely seen in a photograph. This problem is more pronounced for rimless eyeglasses, as one must be able to discern the bottom edge of the clear lenses.
[0004] Further background information is provided in the list of citations at the end of this document. Summary of the Invention
[0005] Therefore, the purpose of this disclosure is to provide a more robust photographic method for obtaining appropriate parameters.
[0006] According to this disclosure, this objective is achieved through a combined photographic and thermal imaging method that obtains at least one fitting parameter for a selected eyeglass frame model of an eyeglass wearer in order to fit ophthalmic lenses into the eyeglass frame model. The method includes the following steps:
[0007] a) Take a photograph of the wearer's face while wearing glasses with the selected frame model;
[0008] b) Take a thermal image of the wearer's face while wearing the glasses;
[0009] c) Identify the outline of the glasses in the thermal image;
[0010] d) The outline identified in 1) or the box shape derived from the identified outline in 2) is superimposed onto the photograph to obtain a composite image;
[0011] e) Obtain at least one adaptation parameter from the synthesized image.
[0012] In fact, by taking not only photographs but also thermal images, a much better image of the shape of the glasses was obtained. This is because the thermal infrared radiation emitted by the entire face of the glasses wearer is blocked in the area of the face covered by the glasses. Accordingly, the shape of the glasses is clearly visible in the thermal image and can be easily deduced from it.
[0013] The following features may be implemented individually or in combination:
[0014] -At least step c) is performed by a computer-implemented image processing algorithm;
[0015] - When performing step c), the image processing algorithm includes the following steps: i) converting the thermal image into a binary image by classifying each pixel of the thermal image into a first pixel category when the pixel intensity is higher than a predetermined threshold and a second pixel category when the pixel intensity is lower than a predetermined threshold; ii) identifying the outline of glasses in the binary image by edge detection;
[0016] - When performing step c), the image processing algorithm includes another step: after step i) and before step ii), assigning pixel clusters of smaller than a predetermined size in one pixel category of the binary image to another pixel category;
[0017] - Detect facial boundaries in the photo, and wherein, when performing step c), the image processing algorithm includes another step: after step i) and before step ii), assign all pixels in the binary image that belong to the second pixel category and are located outside the facial boundaries to the first pixel category;
[0018] - Identify the eye region in the photograph, wherein, when performing step c), the image processing algorithm applies step i) only to the portion of the thermal image corresponding to the identified eye region;
[0019] - At least a part of the image processing algorithm is based on supervised machine learning;
[0020] - In alternative scheme 1) of step d), step d) involves detecting the same part of the eyeglass frame, such as the bridge of the frame, in the photograph and the identified contour, and using the detected frame part to locate the identified contour on the photograph.
[0021] - At least one fitting parameter obtained is a measurement of the wearer's eye position relative to the glasses, such as fitting height.
[0022] This disclosure also relates to an electronic device for obtaining at least one fitting parameter for a selected eyeglass frame model for an eyeglass wearer in order to fit an ophthalmic lens into the eyeglass frame model, the device comprising:
[0023] - A camera adapted for taking photos of the face of a wearer wearing glasses with a selected eyeglass frame model;
[0024] - A thermal imaging camera adapted to capture thermal images of the face of a wearer wearing the glasses;
[0025] - A memory that stores an image processing algorithm, which includes the following steps:
[0026] i) Identify the outline of the glasses in the thermal image;
[0027] ii) The identified contours or box shapes derived from the identified contours are superimposed onto the photograph to obtain a composite image; and
[0028] iii) Obtain at least one adaptation parameter from the synthesized image; and
[0029] - A processor adapted to execute image processing algorithms stored in memory.
[0030] In particular, the electronic device can be a smartphone, tablet, or a measuring instrument specifically designed for ophthalmic care professionals.
[0031] This disclosure also relates to computer software that includes instructions for implementing the methods described above when a processor executes the software.
[0032] This disclosure also relates to a computer-readable non-transitory storage medium carrying the aforementioned computer software.
[0033] definition
[0034] In the context of this disclosure, the term "photograph" refers to an image taken by a photographic camera in the visible portion of the electromagnetic spectrum. This visible portion covers wavelengths from about 400 to about 800 nm.
[0035] The term "thermal image" refers to an image captured by a thermal imaging camera in the thermal infrared portion of the electromagnetic spectrum. This thermal infrared portion covers wavelengths from approximately 3 to approximately 15 μm.
[0036] The term "frame shape" refers to the frame lens system as defined in ISO standard 8624:2020(en), which is incorporated herein by reference. A frame shape is a pair of rectangular frames that form the shape of the two lenses of an eyeglass.
[0037] The term "fit height" refers to the vertical distance between the center of the pupil of the eyewearer and the bottom edge of the corresponding lens or frame of the eyeglasses worn by the wearer. Attached Figure Description
[0038] Embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, in which:
[0039] Figure 1
[0040] Figure 1 This is a top-down view of the photographic setup based on the content of this disclosure.
[0041] Figures 2a to 2f
[0042] Figures 2a to 2f Different steps of the combined photographic and thermal imaging method according to this disclosure are shown.
[0043] Figure 3
[0044] Figure 3 This is a block diagram of an electronic device based on this disclosure.
[0045] Figure 4
[0046] Figure 4 This demonstrates how, according to this disclosure, the outline of glasses can be identified in a thermal image via thresholding and edge detection.
[0047] Figure 5
[0048] Figure 5 The removal of image artifacts according to embodiments of this disclosure is demonstrated.
[0049] Figure 6
[0050] Figure 6 This demonstrates how facial boundary detection, as described in this disclosure, can help binarize thermal images.
[0051] Figure 7
[0052] Figure 7 An example is shown in which thresholding is applied only to regions identified as eye areas in a thermal image.
[0053] Figure 8
[0054] Figure 8 The image illustrates the positioning of the eyeglasses outline on a photograph using the detected frame portion, according to an embodiment of this disclosure. Detailed Implementation
[0055] Figure 1The basic setup according to this disclosure is illustrated, which can be used to obtain one or more fitting parameters to fit ophthalmic lenses into a given eyeglass frame. This setup can be used in a typical scenario where an eyeglass wearer 1 purchases a new pair of glasses from a store run by an ophthalmic care professional.
[0056] In this typical scenario, the eyeglass wearer 1 selects eyeglass frame model 3 from the frame models displayed in the store. In order to fit the ophthalmic lenses corresponding to the eyeglass wearer's prescription into the selected frame model, the ophthalmologist needs to obtain a set of fitting parameters. These fitting parameters depend on the selected frame model and the individual morphology of the eyeglass wearer.
[0057] In order to have the correct settings for obtaining at least some of these fitting parameters, ophthalmology professionals and eyeglass wearers can perform the following routine:
[0058] First, an ophthalmologist provides a sample of the selected frame model 3 to the eyeglass wearer 1. This sample 3 has two arbitrary lenses 5 fitted within it. The lenses 5 can be dummy lenses without any refractive power.
[0059] -Secondly, the eyeglass wearer 1 puts on the sample 3.
[0060] Finally, after correctly positioning the fake glasses 3 on the head 4 of the glasses wearer 1, she positioned her face 6 in front of the camera device 7. Then, the glasses wearer 1 was in... Figure 1 In the settings shown.
[0061] The camera device 7 is an electronic device for obtaining at least one fitting parameter for a selected eyeglass frame model 3 for eyeglass wearer 1 in order to fit an ophthalmic lens into the eyeglass frame model 3.
[0062] refer to Figure 3 The camera device 7 includes a photographic camera 9, a thermal imaging camera 11, a memory 13, and a processor 15. The memory 13 has an image processing algorithm 17 stored therein. The processor 15 is adapted to execute the image processing algorithm 17 stored in the memory 13. At least a portion of the image processing algorithm 17 may be based on supervised machine learning.
[0063] The camera device 7 can be a smartphone, tablet, or a measuring instrument specifically designed for ophthalmic care professionals. In particular, the camera device 7 can be an electronic measuring column. Such columns are typically installed in opticians' shops and are part of their measuring equipment.
[0064] from Figure 1 The setup shown here, starting with the method for obtaining at least one adaptation parameter according to this disclosure, involves the following steps:
[0065] First, camera device 7 uses its camera 9 to take a photograph 19 of the face 6 of the person wearing glasses 3, which is the wearer of glasses 1. Figure 2a As shown. Camera 9 is marked in black to indicate that it is in operation.
[0066] Secondly, camera device 7 uses its thermal imaging camera 11 to capture a thermal image 21 of the face 6 of the eyeglass wearer 1 wearing glasses 3. For example... Figure 2b As shown. The thermal imaging camera 11 is marked in black to indicate that it is in operation.
[0067] Photograph 19 and thermal image 21 are both stored in the device's memory 13.
[0068] Next, processor 15 executes image processing algorithm 17. This image processing corresponds to the subsequent steps of the adaptation parameter acquisition method. These subsequent steps are as follows:
[0069] like Figure 2c As shown, the thermal image 21 is analyzed in order to identify the outline 23 of the glasses 3. Figure 2c The image on the left highlights the outline 23 identified in thermal image 21. Figure 2c The image on the right shows the identified contour 23 separately. In the example shown, the identified contour 23 consists of two similar closed shapes 23a and 23b. Each shape 23a and 23b corresponds to the shape of one of the two lenses of the eyeglasses.
[0070] exist Figure 2d In the optional step shown, a box shape 25 is derived from the identified contour 23. The box shape 25 consists of two rectangles 25a and 25b. Each rectangle 25a and 25b encloses one of the two closed shapes 23a and 23b. Image processing algorithm 17 determines rectangles 25a and 25b according to the box lens system defined in ISO standard 8624:2020(en).
[0071] Then, as Figure 2e As shown, the image processing algorithm 17 overlays the identified contour 23 or box shape 25 onto the photograph 19 to obtain a composite image 27. Figure 2e The image on the left shows a composite image 27a with outline overlay, and Figure 2e The image on the right shows a composite image 27b with superimposed box shapes.
[0072] exist Figure 2f In the final step shown, image processing algorithm 17 obtains at least one fitting parameter from the synthesized image 27. The obtained one or more fitting parameters are measurements of the wearer's eye position relative to the glasses 3, such as fitting height.
[0073] exist Figure 2f In this process, the determined fitting parameter is the fitting height FH. Therefore, in the synthesized image 27a, the image processing algorithm 17 identifies the centers C1 and C2 of the wearer's pupils, and the bottom edges E1 and E2 of the contour 23. The obtained fitting height can correspond to the vertical distance FH1 between the right center C1 and the corresponding right bottom edge E1, or to the vertical distance FH2 between the left center C2 and the corresponding left bottom edge E2, or to the average of FH1 and FH2.
[0074] The step of obtaining the adaptation parameters can also be performed on an alternative synthetic image 27b, which has an overlaid box shape 25.
[0075] Now for reference Figure 4 The figure provides an example of how image processing algorithm 17 can identify the outline 23 of glasses 3 in thermal image 21.
[0076] In this example, contour 23 was identified through two consecutive image processing steps:
[0077] First, the thermal image 21 is converted into a binary image 29 using a thresholding technique. More specifically, each pixel in the thermal image is classified into a first pixel category when its intensity is above a predetermined threshold, and a second pixel category when its intensity is below a predetermined threshold. This step is performed by... Figure 4 Images A and B are shown in the image.
[0078] Secondly, contour 23 was identified in the binary image 29 through edge detection. This was achieved through... Figure 4 Images C and D are shown in the figure. Images C and D only show the edge detection applied to the details of binary image 29, but the same process is of course applied to the entire binary image 29.
[0079] Edge detection can be divided into two sub-steps: the first sub-step is to extract edges from the binary image 29, and the result is as follows. Figure 4 As shown in image C, and the second sub-step of smoothing the extracted edges, the result is as follows: Figure 4 As shown in picture D.
[0080] The binary image 29 obtained by thresholding can contain small pixel clusters 31, see [link / reference] Figure 5 These small pixel clusters 31 are unwanted artifacts that can distort the results of edge detection. To address this issue, they can be performed after the thresholding step and before the edge detection step. Figure 5The first intermediate step is shown. This first intermediate step removes small pixel clusters 31 from the binary image 29 by assigning pixel clusters smaller than a predetermined size in one pixel category to another pixel category.
[0081] As another way to make binary images 29 more suitable for edge detection, Figure 6 The second intermediate step shown can be performed as a supplement to or alternative to the first intermediate step. Like the first intermediate step, the second intermediate step is performed after the thresholding step and before the edge detection step. The second intermediate step relies on the available photograph 19. More specifically, in the second intermediate step, the boundary 33 of the wearer's face is detected in photograph 19, and all pixels in the binary image 29 belonging to the second pixel category and located outside the facial boundary 33 are assigned to the first pixel category.
[0082] exist Figure 6 In the binary image 29, which is a black and white image, "black" is the first pixel category and "white" is the second pixel category. In this case, the result of the second intermediate step is to convert all white pixels in the binary image 29 located outside the face boundary 33 into black pixels. Figure 6 The image on the right shows the resulting purified binary image 35.
[0083] To simplify thresholding and edge detection, see [link to documentation]. Figure 7 Image processing algorithm 17 can first identify the eye region 37 in photo 19, and then apply thresholding only to the portion of the thermal image 21 corresponding to the identified eye region 37.
[0084] Sometimes, photograph 19 and thermal image 21 may not be perfectly aligned, for example, when there is an offset O between the optical axis X1 of photographic camera 9 and the optical axis X2 of thermal imaging camera 11 (see Figure 1 To compensate for this, the step of overlaying the eyeglasses outline 23 onto the photograph 19 may involve detecting the same portion of the eyeglasses frame in both the photograph 19 and the identified outline 23, and using the detected frame portion to locate the identified outline 23 on the photograph 19. Figure 8 This variant is shown.
[0085] exist Figure 8 In the example, the detected frame portion is the bridge of the frame 39. During the overlay step, the position of the profile 23 on the image 19 is moved until the bridge of the frame 39a of the profile 23 is on top of the bridge of the frame 39b of the image 19.
[0086] In summary, the methods and electronic devices disclosed herein advantageously combine thermal imaging with established photography to significantly improve the determination of ophthalmic fitting parameters.
[0087] Citation List
[0088] Patent documents
[0089] -WO 2020 / 064755 A1.
[0090] Non-patent literature
[0091] -ISO Standard 8624:2020 (en);
[0092] -Yufeng Zheng, “Face detection and eyeglasses detection for thermal face recognition”, Proceedings of SPIE (International Society for Optical Engineering), February 2012;
[0093] -Rolf Rainer Grigat et al., “Robust eye detection under active infrared illumination,” Proceedings of the 18th International Conference on Pattern Recognition, 2006; and
[0094] -George Bebis et al., “Face recognition by fusing thermal infrared and visible imagery”, Image and Visual Computing, 24 (2006), 727-742.
Claims
1. A combined photographic and thermal imaging method for obtaining at least one fitting parameter (FH) for a selected eyeglass frame model of an eyeglass wearer (1) to fit an ophthalmic lens (5) into the eyeglass frame model, the method comprising the following steps: a. Take a photograph (19) of the face (6) of the wearer wearing glasses (3) with the selected eyeglass frame model; b. Take a thermal image (21) of the face (6) of the wearer wearing the glasses (3); c. Identify the outline (23) of the glasses in the thermal image (21); d. will 1) The identified contours (23), or 2) The shape of the box derived from the identified contours (25) The composite image (27) is then superimposed onto the photograph (19). e. Obtain the at least one adaptation parameter (FH) from the synthesized image (27).
2. The method as described in claim 1, wherein, At least step c) is performed by a computer-implemented image processing algorithm (17).
3. The method as described in claim 2, wherein, When step c) is performed, the image processing algorithm (17) includes the following steps: i) The thermal image (21) is converted into a binary image (29) by classifying each pixel of the thermal image into a first pixel category when the pixel intensity is higher than a predetermined threshold, or into a second pixel category when the pixel intensity is lower than a predetermined threshold. ii) Identify the outline (23) of the glasses in the binary image (29) by edge detection.
4. The method of claim 3, wherein, When step c) is performed, the image processing algorithm (17) includes another step: after step i) and before step ii), assigning a pixel cluster (31) of a pixel class in the binary image (29) that is smaller than a predetermined size to another pixel class.
5. The method of claim 3, further comprising the step of detecting facial boundaries (33) in the photograph (19), wherein, When step c) is performed, the image processing algorithm (17) includes another step: after step i) and before step ii), all pixels in the binary image (29) that belong to the second pixel category and are located outside the face boundary (33) are assigned to the first pixel category.
6. The method of claim 3, further comprising the step of identifying the eye region (37) in the photograph (19), wherein, When step c) is performed, the image processing algorithm (17) applies step i) only to the portion of the thermal image (21) corresponding to the identified eye region (37).
7. The method of claim 2, wherein, At least a portion of the image processing algorithm (17) is based on supervised machine learning.
8. The method according to any one of claims 1 to 7, wherein, In an alternative scheme 1) to step d), step d) involves detecting the same part (39) of the eyeglass frame in both the photograph (19) and the identified contour (23) and using the detected same part (39) to locate the identified contour (23) on the photograph (19).
9. The method of claim 8, wherein, The detected portion (39) of the eyeglass frame is the bridge of the nose.
10. The method of claim 1, wherein, At least one fitting parameter (FH) obtained is a measurement of the position of the wearer's eye relative to the glasses (3).
11. The method of claim 10, wherein, The adaptation parameter (FH) is the adaptation height.
12. An electronic device (7) for obtaining at least one fitting parameter (FH) for a selected eyeglass frame model for an eyeglass wearer (1) to fit an ophthalmic lens (5) into the eyeglass frame model, the device (7) comprising: - A camera (9) adapted to take a photograph (19) of the wearer’s face (6) with glasses (3) having a selected eyeglass frame model. - Thermal imaging camera (11), which is adapted to capture thermal images (21) of the wearer's face (6) with the glasses (3) on. - Memory (13), the memory stores image processing algorithm (17), the image processing algorithm includes the following steps: i) Identify the outline (23) of the glasses in the thermal image (21); ii) The identified contour (23) or the box shape (25) derived from the identified contour (23) is superimposed onto the photograph (19) to obtain a composite image (27); and iii) Obtain the at least one adaptation parameter (FH) from the synthesized image (27); and - Processor (15), which is adapted to execute image processing algorithms (17) stored in the memory (13).
13. The electronic device (7) as claimed in claim 12, wherein, The electronic device is a smartphone, tablet, or a measuring instrument used by ophthalmic care professionals.
14. A computer program product comprising instructions which, when executed by a processor (15), implement the method according to any one of claims 1 to 11.
15. A computer-readable non-transitory storage medium carrying a computer program product according to claim 14.