Insertion and removal assistance for digital contact lenses

Abstract

Methods, systems, and methods of user interfaces for contact lens insertion and removal are disclosed herein. An example method may include receiving one or more images of at least a portion of a user. The one or more images may include a representation of the user's eye and a representation of the contact lens on the user's eye. An example method may include analyzing placement of a contact lens on an eye of a user based on the one or more images. An example method may include outputting an indication of proper or improper placement of a contact lens based on analyzing the placement of the contact lens.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. patent application serial number 17 / 891,736 filed on August 19, 2022, which is incorporated herein by reference in its entirety. Background Art

[0003] Patients who are new to soft contact lenses often have difficulty inserting and removing their contact lenses. This frustration can lead to frustration, discomfort, and / or discontinuation of lens wear. Often, when patients undergo contact lens insertion and removal training tasks in an eye care professional's (ECP's) office, the patients may be successful, but once back home, the patients are unable to successfully repeat these tasks. In the office, the patient has the advantage of getting feedback from the office staff, but when attempting to insert or remove a lens at home, the patient does not have that advantage.

[0004] Still needs improvement. Summary of the invention

[0005] Disclosed herein are methods, systems, and user interfaces for contact lens insertion and removal that help assist novice soft contact lens patients with insertion, removal, and reminding them when a contact lens is properly on their eye. Contact lens removal can be reduced if patients have a tool that provides them with feedback on lens insertion and removal outside of an eye care provider's office.

[0006] The present disclosure relates to a digital (e.g., application) solution that can be considered a virtual ophthalmic technician that can help novice soft contact lens patients insert and remove contact lenses and remind patients when contact lenses are correctly on their eyes. Correctly controlling the upper and lower eyelids is key to insertion and removal, so the methods, systems, and user interfaces disclosed herein can utilize a camera in a device (such as a mobile phone or tablet) to project visual cues on the upper and lower eyelids to assist the patient in correctly controlling the eyelids with the patient's fingers during insertion and removal. When the finger is in the correct position, the visual cue can change to a confirmation color, such as green. When the finger is not in the correct position, the visual cue can change to, for example, red. The methods, systems, and user interfaces disclosed herein can provide suggestions to the patient step by step via voice or sound control to successfully insert and remove contact lenses.

[0007] One aspect of successful insertion and removal can be knowing that the contact lens is in the proper position on the eye. Therefore, it may be useful for the methods, systems, and user interfaces disclosed herein to be able to detect that the contact lens is centered on the eye. In order to determine whether the contact lens is correctly inserted, the methods, systems, and user interfaces disclosed herein can utilize edge detection technology to identify the correct placement. The methods, systems, and user interfaces disclosed herein can be used in a home environment to enhance contact lens insertion and removal instruction performed in the office when the office staff is away. In addition, eye care professionals and staff can use the methods, systems, and user interfaces disclosed herein to assist in in-office training.

[0008] A system with one or more computers may be configured to perform specific operations or actions by installing software, firmware, hardware, or a combination thereof on the system, which software, firmware, hardware, or a combination thereof causes the system to perform these actions in operation. One or more computer programs may be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause the device to perform these actions. A general aspect includes a method for contact lens assistance. The method also includes: causing one or more lens type options to be output via a user interface; receiving an indication of a selected lens type in the one or more lens type options via a user interface; causing one or more lens operation options to be output via a user interface; receiving an indication of a selected lens operation in the one or more lens operation options via a user interface; causing one or more eye options to be output via a user interface; receiving an indication of a selected eye in the one or more eye options via a user interface; causing one or more user instructions for inserting or removing a contact lens to be output via a user interface and based on one or more of a selected lens type, a selected lens operation, or a selected eye, wherein the one or more user instructions include user feedback based on an image of the user captured in real time. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of these methods.

[0009] One general aspect includes a method for contact lens assistance. The method also includes: receiving one or more first images of at least a portion of a user, wherein the one or more first images may include a representation of the user's eyes; via a user interface and based on the one or more first images, causing output of a visual cue indicating a user state of the user relative to the user's eyes, wherein the visual cue has a first state indicating a rejected user state and a second state indicating an accepted user state; in response to the accepted user state, causing output of one or more user instructions for inserting a contact lens; receiving one or more second images of at least a portion of the user, wherein the one or more second images may include a representation of the user's eyes and a representation of the contact lens on the user's eyes; based on the one or more second images, analyzing the placement of the contact lens on the user's eyes; and outputting an indication of proper placement or improper placement of the contact lens based on analyzing the placement of the contact lens. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of these methods.

[0010] One general aspect includes a method for contact lens assistance. The method also includes: receiving one or more images of at least a portion of a user, wherein the one or more images may include a representation of the user's eyes and a representation of a contact lens on the user's eyes; based on the one or more images, analyzing the placement of the contact lens on the user's eyes; and based on analyzing the placement of the contact lens, outputting an indication of proper placement or improper placement of the contact lens. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of these methods.

[0011] A general aspect includes a method for contact lens assistance. The method also includes: receiving a real-time video of at least a portion of a user, wherein the video may include a representation of the user's eyes; analyzing the user's user state based at least on the video to determine whether the user state is an accepted user state or a rejected user state; via a user interface and based on the video, causing output of a visual cue indicating the user state of the user relative to the user's eyes, wherein the visual cue has a first state indicating a rejected user state and a second state indicating an accepted user state; via a user interface and based on the video and based on the user state, causing output of one or more real-time prompts to be output to the user; and repeating steps ad until the user state is determined to be an accepted user state. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of these methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The following drawings illustrate various examples discussed in the present disclosure generally by way of example and not by way of limitation. In the drawings:

[0013] Figure 1 An example flow chart of a method according to the present disclosure is depicted.

[0014] Figure 2 Depicted are example user interface screens in accordance with the present disclosure.

[0015] Figure 3 An example flow chart of a method according to the present disclosure is depicted.

[0016] Figure 4 An example flow chart of a method according to the present disclosure is depicted.

[0017] Figure 5 An example flow chart of a method according to the present disclosure is depicted.

[0018] Figure 6 Depicted are example images of a user's eyes in accordance with the present disclosure.

[0019] Figure 7 Depicts the computational boundaries with superposition Figure 6 .

[0020] Figure 8 Describes the present disclosure Figure 6 Example masked image after implementing smart masking techniques to remove image data outside a defined image window.

[0021] Fig. 9 Describes the present disclosure Figure 8 A mask image showing possible boundary arcs.

[0022] Fig.10 Describes the present disclosure Figure 8 A masked image showing representative circles used to identify correct and incorrect representative circles. DETAILED DESCRIPTION

[0023] The present disclosure relates to methods, systems, and user interfaces for contact lens insertion and removal. A user may use a camera of a smart device to record a video of the user attempting to insert and / or remove a contact lens. The screen of the smart device may provide instant feedback to the user. The instant feedback may include approval and / or disapproval of the placement of the user's finger, positioning of the eyelid, positioning of the contact lens, etc. The methods, systems, and user interfaces described herein may reduce the time, effort, and frustration associated with learning to insert and / or remove a contact lens.

[0024] Figure 1An example flow chart of a method for contact lens assistance, such as inserting a contact lens into or removing a contact lens from a user's eye, is depicted. A system having one or more computers may be configured to perform certain operations or actions by installing software, firmware, hardware, or a combination thereof on the system that, in operation, causes the system to perform the actions. As an example, a smartphone may be equipped to capture one or more images (e.g., video, real-time video) of a user. As another example, a user may implement the method on a device configured to display information to a user while capturing a "selfie" view image (e.g., video, real-time video) of the user.

[0025] At step 102, one or more lens type options may be outputted via a user interface. The user interface may be on any device or display. As an example, the user interface may be on a device configured to capture an image of the user's face while viewing the user interface. The lens type options may include a representation of one or more of a reusable or a daily disposable.

[0026] At step 104, an indication of a selected lens type from the one or more lens type options may be received via a user interface. The user interface may be on any input device or touch-enabled display. As an example, the user interface may be on a device configured to display options and receive an indication of a selection of one of the displayed options. Receiving the indication of the selected lens type from the one or more lens type options may include receiving an indication of engagement of a clickable element (such as a button) associated with the selected lens type.

[0027] At step 106, one or more lens operation options may be caused to be output via a user interface. The user interface may be on any device or display. As an example, the user interface may be on a device configured to capture an image of the user's face while viewing the user interface. The one or more lens operation options may include an indication of one or more of insertion or removal of a contact lens.

[0028] At step 108, an indication of a selected lens operation of the one or more lens operation options is received via a user interface. The user interface may be on any input device or touch-enabled display. As an example, the user interface may be on a device configured to display options and receive an indication of a selection of one of the displayed options. Receiving the indication of the selected lens operation of the one or more lens operation options may include receiving an indication of engagement of a clickable element (such as a button) associated with the selected lens operation.

[0029] At step 110, one or more eye options may be outputted via a user interface. The user interface may be on any device or display. As an example, the user interface may be on a device configured to capture an image of the user's face while viewing the user interface. The one or more eye options may include a representation of one or more of a left eye or a right eye.

[0030] At step 112, an indication of a selected eye in one or more eye options is received via a user interface. The user interface may be on any input device or touch-enabled display. As an example, the user interface may be on a device configured to display options and receive an indication of a selection of one of the displayed options. Receiving the indication of the selected eye in one or more eye options may include receiving an indication of engagement of a clickable element (such as a button) associated with the selected eye.

[0031] At step 114, one or more user instructions for inserting or removing a contact lens may be outputted via a user interface, and the one or more user instructions may be based on one or more of a selected lens type, a selected lens operation, or a selected eye. The one or more user instructions may include user feedback based on an image of the user captured in real time. The user interface may be on any device or display. As an example, the user interface may be on a device configured to capture an image of the user's face while viewing the user interface. One or more user instructions for inserting or removing a contact lens may be outputted for a selected eye. One or more user instructions for inserting or removing a contact lens may be outputted for an unselected eye in one or more eye options automatically via the user interface and based on one or more of a selected lens type or a selected lens operation. The one or more user instructions may include washing and drying instructions. The one or more user instructions may include contact lens handling instructions. The one or more user instructions may include contact lens position instructions. The one or more user instructions may include contact lens orientation instructions. The one or more user instructions may include finger position instructions. The one or more user instructions may include camera configuration instructions. The one or more user instructions may include user feedback queries.

[0032] Figure 2An example method flow for contact lens assistance, such as inserting a contact lens into or removing a contact lens from a user's eye, is shown. A system having one or more computers may be configured to perform specific operations or actions by installing software, firmware, hardware, or a combination thereof on the system that, in operation, causes the system to perform the actions. As an example, a smartphone may be equipped to capture one or more images (e.g., video, real-time video) of a user. As another example, a user may implement the method on a device configured to display information to a user while capturing a "selfie" view image (e.g., video, real-time video) of the user.

[0033] The example method flow includes an introductory (e.g., start, login, etc.) page. The introductory page may include a button to initiate one or more methods described herein. The introductory page may include a button to initiate a registration process. Engaging the button to initiate one or more methods described herein may cause a first screen with lens type options to be displayed. The lens type options may include a reusable option and a daily disposable option. Each lens type option may be associated with a button on the first screen. Engagement of the button on the first screen may cause the associated lens type option to be selected and cause a second screen with lens operation options to be displayed. The lens operation options may include an insert option and a remove option. Each lens operation option may be associated with a button on the second screen. Engagement of the button on the second screen may cause the associated lens operation option to be selected and cause a third screen with eye options to be displayed. The eye options may include a left eye option and a right eye option. Each eye option may be associated with a button on the third screen. Engagement of the button on the third screen may cause the associated eye option to be selected and cause an instruction set screen to be initiated based on the selected option. The instruction set screen may include written instructions. The instruction set screen may include a video of the user. The video of the user may include an overlay. The overlay may include an indication of approval or disapproval of the positioning of the finger, eyelid, contact lens, etc. The instruction set screen may include start instructions. The instruction set screen may include end instructions. At the end of the instructions for the selected eye, the instruction set may include instructions related to the unselected eyes.

[0034] Figure 3An example flow chart of a method for contact lens assistance, such as inserting a contact lens into or removing a contact lens from a user's eye, is depicted. A system having one or more computers may be configured to perform certain operations or actions by installing software, firmware, hardware, or a combination thereof on the system that, in operation, causes the system to perform the actions. As an example, a smartphone may be equipped to capture one or more images (e.g., video, real-time video) of a user. As another example, a user may implement the method on a device configured to display information to a user while capturing a "selfie" view image (e.g., video, real-time video) of the user.

[0035] At step 302, one or more first images of at least a portion of a user may be received. The one or more first images may include a representation of an eye of the user. The one or more first images may include a representation of an iris of an eye of the user. The one or more first images may be captured from a video. The one or more first images may be captured from a real-time video.

[0036] At step 304, a visual cue indicating a user state of the user relative to the user's eyes may be output via the user interface and based on the one or more first images. The visual cue may have a first state indicating a rejected user state and a second state indicating an accepted user state. The user state may include one or more of the following: the degree of openness of the user's eyes, the characteristics of the user's other eye, and the placement of the user's fingers relative to the eyes. Output prompts may be provided to the user via the user interface and based on the one or more first images until an accepted user state is detected. The user interface may be on any device or display. As an example, the user interface may be on a device configured to capture an image of the user's face when viewing the user interface.

[0037] At step 306, one or more user instructions for inserting a contact lens may be caused to be output in response to the accepted user status. The one or more user instructions for inserting a contact lens may include audio instructions, visual instructions, or both.

[0038] At step 308, one or more second images of at least a portion of the user may be received. The one or more second images may include a representation of the user's eyes and a representation of the contact lens on the user's eyes. The one or more second images may be captured from a video. The one or more second images may be captured from a real-time video.

[0039] At step 310, the placement of the contact lens on the user's eye may be analyzed based on the one or more second images. Analyzing the placement of the contact lens on the user's eye based on the one or more second images may include detecting an edge of the contact lens using an edge detection model. Analyzing the placement of the contact lens on the user's eye based on the one or more second images may include pre-processing the one or more second images using one or more of cropping, masking, removal, or a combination thereof. Analyzing the placement of the contact lens on the user's eye based on the one or more second images may include pre-processing the one or more second images by digitally removing artifacts from the one or more second images. The artifacts may include the user's eyelashes. Analyzing the placement of the contact lens on the user's eye based on the one or more second images may include detecting an edge of the contact lens using a mask for edge detection. As a non-limiting example, the analysis of the placement of the contact lens on the eye may include one or more of the processes described herein, such as implementing reference Figures 6 to 10 One or more of the processes described. Other processes may be used.

[0040] At step 312, an indication of proper placement or improper placement of the contact lens may be output based on analyzing the placement of the contact lens.

[0041] Figure 4 An example flow chart of a method for contact lens assistance, such as inserting a contact lens into or removing a contact lens from a user's eye, is depicted. A system having one or more computers may be configured to perform certain operations or actions by installing software, firmware, hardware, or a combination thereof on the system that, in operation, causes the system to perform the actions. As an example, a smartphone may be equipped to capture one or more images (e.g., video, real-time video) of a user. As another example, a user may implement the method on a device configured to display information to a user while capturing a "selfie" view image (e.g., video, real-time video) of the user.

[0042] At step 402, one or more images of at least a portion of a user may be received. The one or more images may include a representation of the user's eyes and a representation of a contact lens on the user's eyes. The one or more images may be captured from a video. The one or more images may be captured from a real-time video.

[0043] At step 404, the placement of the contact lens on the user's eye may be analyzed based on the one or more images. Analyzing the placement of the contact lens on the user's eye based on the one or more images may include: detecting the edge of the contact lens using an edge detection model. Analyzing the placement of the contact lens on the user's eye based on the one or more images may include: pre-processing the one or more images using one or more of cropping, masking, removal, or a combination thereof. Analyzing the placement of the contact lens on the user's eye based on the one or more images may include: pre-processing the one or more images by digitally removing artifacts from one or more second images. The artifacts may include the user's eyelashes. Analyzing the placement of the contact lens on the user's eye based on the one or more images may include: detecting the edge of the contact lens using a mask for edge detection. As a non-limiting example, the analysis of the placement of the contact lens on the eye may include one or more of the processes described herein, such as implementing reference Figures 6 to 10 One or more of the processes described. Other processes may be used.

[0044] At step 406, an indication of proper placement or improper placement of the contact lens may be output based on analyzing the placement of the contact lens.

[0045] Figure 5 An example flow chart of a method for contact lens assistance, such as inserting a contact lens into or removing a contact lens from a user's eye, is depicted. A system having one or more computers may be configured to perform certain operations or actions by installing software, firmware, hardware, or a combination thereof on the system that, in operation, causes the system to perform the actions. As an example, a smartphone may be equipped to capture one or more images (e.g., video, real-time video) of a user. As another example, a user may implement the method on a device configured to display information to a user while capturing a "selfie" view image (e.g., video, real-time video) of the user.

[0046] A real-time video of at least a portion of a user may be received at step 502. The video may include a representation of the user's eyes.

[0047] At step 504, a user state of the user may be analyzed based at least on the video to determine whether the user state is an accepted user state or a rejected user state. An accepted user state may include a contact lens being properly inserted into the user's eye. An accepted user state may include a contact lens being properly removed from the user's eye. The user state may include one or more of the following: the degree of openness of the user's eye, the placement of the contact lens on the eye, characteristics of the user's other eye, or the placement of the user's finger relative to the eye, the lens being too high, or the lens being too low. As a non-limiting example, analysis of the placement of the contact lens on the eye may include one or more of the processes described herein, such as implementing reference Figures 6 to 10 One or more of the processes described. Other processes may be used.

[0048] At step 506, a visual cue indicating the user status of the user relative to the user's eyes may be output via the user interface and based on the video. The visual cue may have a first state indicating a rejected user status and a second state indicating an accepted user status. The user interface may be on any device or display. As an example, the user interface may be on a device configured to capture an image of the user's face when viewing the user interface.

[0049] At step 508, one or more real-time prompts may be output to the user via the user interface and based on the video and based on the user's state. The user interface may be on any device or display. As an example, the user interface may be on a device configured to capture an image of the user's face while viewing the user interface. The one or more real-time prompts may include instructions for inserting or removing contact lenses.

[0050] At step 510 , steps 502 to 508 may be repeated until the user status is determined to be an accepted user status.

[0051] As described herein, analysis of the user state may include analyzing one or more of the following: the degree of openness of the user's eyes, the placement of the contact lens on the eye, the characteristics of the user's other eye, or the placement of the user's finger relative to the eye. Such analysis may be based at least in part on an image (or video) capturing a portion of the user (such as the eye). Such images and their analysis may benefit from certain processing techniques, such as those described below. Analyzing one or more images may include: using an edge detection model to detect the edge of the contact lens. Analyzing one or more images may include: pre-processing one or more images using one or more of cropping, masking, removal, or a combination thereof. Analyzing one or more images may include: pre-processing one or more images by digitally removing artifacts from one or more second images. Artifacts may include eyelashes of the user. Analyzing one or more images may include: using a mask for edge detection to detect the edge of the contact lens.

[0052] As a non-limiting example, analysis of a user's state (e.g., placement of a contact lens on an eye) may include one or more pre-processing methods, wherein images or videos captured by the apparatus disclosed herein may be pre-processed. The image processing methods may be in the additional method flows disclosed herein (see Figures 1 to 5 ) before, during or after. Pretreatment methods include, but are not limited to, the various techniques described herein.

[0053] As non-limiting examples, image processing techniques may include: tightly cropping images or videos of the user, highlighting parts of the user's eye structure, adjusting the modeling framework, isolating the blue channel, simplifying analysis methods and models, and other such strategies. Such strategies can be used to achieve or move towards one or more goals. These goals can be selected to improve the overall functionality of the method flow described herein. Goals include, but are not limited to, removing eye recognition obstacles, highlighting the lens ridges of the eye structure, finding multiple rings related to the eye structure, etc. Such goals can assist in improving lens recognition. Improved lens recognition will improve the functionality and output of the additional method flows disclosed herein. The eye structure may include those structures related to the eye, cornea, iris, sclera, lens, and other such visual-related subjects. As a non-limiting example, the processing method may occur after one or more images or videos of at least a portion of the user are captured by the device (e.g., steps 110, 302, 308, 402, 502, etc.) but before the output is provided to the user (e.g., steps 114, 312, 416, 508, etc.). As an additional non-limiting example, processing methods may occur during analysis steps (eg, 310, 404, 504, etc.) As previously described, processing methods improve the reliability and accuracy of the output of the method flows disclosed herein.

[0054] The processing may utilize a variety of methods to improve the output to the user. The processing may improve lens recognition, detection, etc. within one or more images or videos captured by the device. In certain embodiments, the processing may be used to use the local topography of the ocular structure (e.g., the eye) to recognize or detect the lens. In this case, it can be said that there is a certain probability that the lens is found to be within an annular band relative to the iris of the ocular structure. By identifying the characteristic dimensions of the annular band relative to the iris, one or more approximate characteristic dimensions of the lens can be found. These dimensions include, but are not limited to, the minimum radius and the maximum radius of potential lenses (e.g., the edge of the lens). These dimensions may allow for the creation of a window of potential lens locations.

[0055] The window may be defined by two or more concentric circles, but other windows are also possible. As a non-limiting example, the window may be defined by two or more ellipses or other circular structures. As a non-limiting example, an image or video of an eye structure may include parts of a circle (e.g., an incomplete circle) or an arc. An arc may represent more than one eye structure. As a non-limiting example, an arc may represent parts of an eyelid, parts of a lens, etc. The processing method disclosed herein may identify whether an arc is associated with a given eye structure. The correct arc may be identified as an arc associated with an eye structure centered on the iris or within a tolerance of the center. As a non-limiting example, the processing method may distinguish between an arc representing an eyelid and an arc representing a lens. In such an example, the arc representing the eyelid may not be centered on the iris, while the arc representing the lens is centered on the iris. Using an approximate feature size to create an image window improves the ability to accurately and reliably identify one or more lenses in one or more images or videos captured by the device.

[0056] The present disclosure relates to methods for further improving the accuracy and reliability of processing methods. As a non-limiting example, improving the quality of an image or video of a user's ocular structure may improve the accuracy and reliability of the processing methods. The improved quality includes, but is not limited to, more accurate image or video capture of the user's iris. This includes capturing an image or video representing all existing boundaries of the iris. One such method is to increase the exposed area of ​​the user's eye. Increasing the exposed area would allow the device to more accurately capture an image or video of the user's ocular structure. More accurate image capture would allow the processing methods and other method flows disclosed herein to experience improved functionality.

[0057] As an illustrative example, Figure 6 An input image including at least a portion of a user's eye is shown. The input image can be captured using various cameras and across various light spectrums. Based on the captured image, one or more boundaries (e.g., iris boundary, lens edge minimum boundary, lens edge maximum boundary, etc.) can be calculated (e.g., estimated). Various formulas and techniques can be used to calculate the one or more boundaries. As an example, formula set one (1) can be used, as shown below:

[0058] Iris border

[0059] r iris =11.7mm,r lens =14mm,err=2mm

[0060] Estimated minimum lens circumference

[0061] r lensmin =(r lens +r irtis ) / 2≈more than r iris10%,

[0062] Estimated maximum lens circumference

[0063] r lensmax =(r lens +err)≈more than r iris 37%. (1)

[0065] Other formulas, estimates, and representative values ​​(eg, 11.7 mm, 14 mm, 2 mm, etc.) may be used. Figure 7 An illustrative example of a calculated iris boundary 702 (and iris center 703), an estimated lens perimeter minimum 704, and an estimated lens perimeter maximum 706 is shown. Using these boundaries 702, 704, 706, a window between the estimated lens perimeter minimum 704 and the estimated lens perimeter maximum 706 can be calculated. The image can be processed into a masked image 800 by removing (e.g., masking) portions of the image outside the calculated window, such as Figure 8 As shown in . The mask image represents a subset of the image information from the original image.

[0066] The mask image 800 may include one or more arcs or boundary indicators. An arc may represent more than one eye structure. As non-limiting examples, an arc may represent portions of an eyelid, portions of a lens, an edge, a boundary, etc. Such arcs may be analyzed and classified into types such as an eyelid arc 902 or a lens boundary arc 904, such as Fig. 9 As shown. The type can be determined using a formula such as Formula Set 1 (1) or other method to identify the arc within a predetermined confidence level. For example, it can be determined that the located arc forms a portion of a representative circle. By estimating such a circle, the center of the circle can be determined and the center can be compared to the center of the calculated iris boundary. As an illustrative example, Fig.10 A masked image 800 is shown with superimposed representative circles 1002, 1002' and centers 1004, 1004' calculated based on the geometry of the arcs. By using a preset confidence boundary 1006 (which may be derived from a formula such as Formula Set One (1)), it may be determined whether the representative centers 1004, 1004' are within the confidence boundary 1006 or outside the confidence boundary 1006. As another example, the centers 1004 within the confidence boundary 1006 may be identified as a correct circle that fits around a lens. The centers 1004' that are outside the confidence boundary 1006 may be identified as a false circle that fits around something other than the target lens (e.g., an eyelid). The confidence boundary 1006 may be adjusted to control the identification of arcs and artifacts in the masked image 800, which may be used to indicate proper or improper lens placement or lens removal.

[0067] Other lens detection algorithms may be used, for example, the detection equations disclosed in "Fully Automated Soft Contact Lens Detection from NIR Iris Images" by Kumar et al., published in Computer Science, ICPRAM on February 24, 2016. As another example, processing the images may include using some techniques (e.g., pre-processing techniques) such as those described in "Using Iris and Sclera for Detection and Classification of Contact Lenses" by Gragnaniello et al., Pattern Recognition Letters, published by Elsevier Online ISSN: 0167-8655. Additionally and alternatively, the relevant structures of the face discussed herein (including the iris) may be determined by applying one or more combinations of deep learning, neural networks, and training machine learning models, as will be understood by those skilled in the art. As a non-limiting example, irises may be recognized using a convolutional neural network as described in the following paper: “ContlensNet: Robust Iris Contact Lens Detection Using Deep Convolutional Neural Networks” published in the 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[0068] In some examples, lidar data (e.g., available from some The foreground-specific teacher model may be used to train a machine learning model to recognize known failure modes or successes for a user in real time. The foreground-specific teacher model may represent any suitable machine learning model that can be trained to detect and depict objects in images or lidar data, such as a collection of neural network layers. The model can then be deployed in software on a user device (e.g., a mobile phone equipped with one or both of video and lidar sensors) to provide the user with auditory or visual feedback / cues (including but not limited to visual cues as described above) that are known to be used to assist the user in successfully inserting a contact lens into the eye.

[0069] Example

[0070] Embodiment 1: A method for contact lens assistance, the method comprising: causing one or more lens type options to be output via a user interface; receiving an indication of a selected lens type from the one or more lens type options via the user interface; causing one or more lens operation options to be output via the user interface; receiving an indication of a selected lens operation from the one or more lens operation options via the user interface; causing one or more eye options to be output via the user interface; receiving an indication of a selected eye from the one or more eye options via the user interface; causing one or more user instructions for inserting or removing a contact lens to be output via the user interface and based on one or more of the selected lens type, the selected lens operation, or the selected eye, wherein the one or more user instructions include user feedback based on an image of the user captured in real time.

[0071] Embodiment 2: The method of Embodiment 1, wherein the lens type options include an indication of one or more of reusable or daily disposable.

[0072] Embodiment 3: The method of any one of Embodiments 1 to 2, wherein the one or more lens operation options include an indication of one or more of insertion or removal of a contact lens.

[0073] Embodiment 4: The method of any one of Embodiments 1 to 3, wherein the one or more eye options include a representation of one or more of a left eye or a right eye.

[0074] Embodiment 5: A method according to any one of embodiments 1 to 4, wherein causing output of one or more user instructions for inserting or removing a contact lens is performed for the selected eye, and the method further comprises automatically causing output of one or more user instructions for inserting or removing a contact lens via the user interface and based on one or more of the selected lens type or the selected lens operations, and for unselected eyes in the one or more eye options.

[0075] Embodiment 6: The method according to any one of embodiments 1 to 5, wherein the one or more user instructions include washing and drying instructions.

[0076] Embodiment 7: The method of any one of Embodiments 1 to 6, wherein the one or more user instructions include contact lens handling instructions.

[0077] Embodiment 8: The method of any one of Embodiments 1 to 7, wherein the one or more user instructions include contact lens position instructions.

[0078] Embodiment 9: The method of any one of Embodiments 1 to 8, wherein the one or more user instructions include contact lens orientation instructions.

[0079] Embodiment 10: The method according to any one of embodiments 1 to 9, wherein the one or more user instructions include finger position instructions.

[0080] Embodiment 11: The method according to any one of embodiments 1 to 10, wherein the one or more user instructions include camera configuration instructions.

[0081] Embodiment 12: The method according to any one of embodiments 1 to 11, wherein the one or more user instructions include a user feedback query.

[0082] Embodiment 13: A method for contact lens assistance, the method comprising: receiving one or more first images of at least a portion of a user, wherein the one or more first images include a representation of the user's eyes; outputting, via a user interface and based on the one or more first images, a visual cue indicating a user status of the user relative to the user's eyes, wherein the visual cue has a first state indicating a rejected user status and a second state indicating an accepted user status; in response to the accepted user status, outputting one or more user instructions for inserting a contact lens; receiving one or more second images of at least a portion of the user, wherein the one or more second images include a representation of the user's eyes and a representation of a contact lens on the user's eyes; analyzing placement of the contact lens on the user's eyes based on the one or more second images; and outputting an indication of proper or improper placement of the contact lens based on the analysis of the placement of the contact lens.

[0083] Embodiment 14: The method of Embodiment 13, wherein the one or more first images include a representation of an iris of the eye of the user.

[0084] Embodiment 15: A method according to any one of Embodiments 13 to 14, wherein the user state includes one or more of the following: the openness of the eye of the user, the characteristics of the other eye of the user, and the placement of the user's finger relative to the eye.

[0085] Embodiment 16: The method according to any one of embodiments 13 to 15, further comprising providing output prompts to the user via a user interface and based on the one or more first images until an accepted user state is detected.

[0086] Embodiment 17: The method of any one of Embodiments 13 to 16, wherein the one or more user instructions for inserting a contact lens comprise audio instructions, visual instructions, or both.

[0087] Embodiment 18: The method of any one of Embodiments 13 to 17, wherein analyzing the placement of the contact lens on the eye of the user based on the one or more second images comprises: detecting an edge of the contact lens using an edge detection model.

[0088] Embodiment 19: A method according to any one of embodiments 13 to 18, wherein analyzing the placement of the contact lens on the eye of the user based on the one or more second images includes: pre-processing the one or more second images using one or more of cropping, masking, removal, or a combination thereof.

[0089] Embodiment 20: A method according to any one of embodiments 13 to 19, wherein analyzing the placement of the contact lens on the eye of the user based on the one or more second images includes: pre-processing the one or more second images by digitally removing artifacts from the one or more second images.

[0090] Embodiment 21: The method of any one of Embodiments 13 to 20, wherein the artifact comprises eyelashes of the user.

[0091] Embodiment 22: A method according to any one of embodiments 13 to 21, wherein analyzing the placement of the contact lens on the eye of the user based on the one or more second images includes: detecting an edge of the contact lens using a mask for edge detection.

[0092] Embodiment 23: A method according to any one of Embodiments 13 to 22, wherein the one or more first images are captured from a video.

[0093] Embodiment 24: The method according to any one of Embodiments 13 to 23, wherein the one or more second images are captured from a video.

[0094] Embodiment 25: A method according to any one of Embodiments 13 to 24, wherein the one or more first images are captured from real-time video.

[0095] Embodiment 26: A method according to any one of Embodiments 13 to 25, wherein the one or more second images are captured from real-time video.

[0096] Embodiment 27: A method for contact lens assistance, the method comprising: receiving one or more images of at least a portion of a user, wherein the one or more images include a representation of the user's eyes and a representation of a contact lens on the user's eyes; based on the one or more images, analyzing the placement of the contact lens on the user's eyes; and based on the analysis of the placement of the contact lens, outputting an indication of proper placement or improper placement of the contact lens.

[0097] Embodiment 28: The method of Embodiment 27, wherein analyzing the placement of the contact lens on the eye of the user based on the one or more images comprises: detecting an edge of the contact lens using an edge detection model.

[0098] Embodiment 29: A method according to any one of Embodiments 27 to 28, wherein analyzing the placement of the contact lens on the eye of the user based on the one or more images includes: pre-processing the one or more images using one or more of cropping, masking, removal, or a combination thereof.

[0099] Embodiment 30: A method according to any one of Embodiments 27 to 29, wherein analyzing the placement of the contact lens on the eye of the user based on the one or more images includes: pre-processing the one or more images by digitally removing artifacts from the one or more second images.

[0100] Embodiment 31: A method according to any one of Embodiments 27 to 30, wherein the artifact includes eyelashes of the user.

[0101] Embodiment 32: A method according to any one of Embodiments 27 to 31, wherein analyzing the placement of the contact lens on the eye of the user based on the one or more images includes: detecting an edge of the contact lens using a mask for edge detection.

[0102] Embodiment 33: A method according to any one of Embodiments 27 to 32, wherein the one or more images are captured from a video.

[0103] Embodiment 34: A method according to any one of Embodiments 27 to 33, wherein the one or more images are captured from real-time video.

[0104] Embodiment 35: A method for contact lens assistance, the method comprising: a) receiving a real-time video of at least a portion of a user, wherein the video includes a representation of the user's eyes; b) analyzing the user's user state based at least on the video to determine whether the user state is an accepted user state or a rejected user state; c) outputting, via a user interface and based on the video, visual cues indicating the user state of the user relative to the user's eyes, wherein the visual cues have a first state indicating a rejected user state and a second state indicating an accepted user state; d) outputting, via a user interface and based on the video and based on the user state, one or more real-time prompts to be output to the user; and repeating steps ad until the user state is determined to be an accepted user state.

[0105] Embodiment 36: The method of Embodiment 35, wherein the accepted user state comprises a contact lens being properly inserted onto the eye of the user.

[0106] Embodiment 37: The method of any one of Embodiments 35 to 36, wherein the accepted user state comprises a contact lens being properly removed from the eye of the user.

[0107] Embodiment 38: The method of any one of Embodiments 35 to 37, wherein the one or more real-time prompts include instructions for inserting or removing a contact lens.

[0108] Embodiment 39: A method for contact lens assistance, the method comprising:

[0109] receiving a representation of at least a portion of a user's face during an attempt by the user to insert a contact lens into an eye;

[0110] analyzing the representations to determine whether the user is in a known failure mode, wherein the analyzing comprises applying a machine learning model configured to identify the known failure mode based on an aggregate representation of the user during the known failure mode;

[0111] Real-time prompts known to assist the user in successfully inserting the contact lens into the eye are provided via a user interface.

[0112] Embodiment 40: A method according to Embodiment 39, wherein the representation includes lidar data.

[0113] Embodiment 40: A method according to Embodiment 39, wherein the representation includes video data.

[0114] Embodiment 41: A method according to embodiment 39, wherein the failure mode includes one of the following: i) insufficient opening of the eye of the user; ii) insufficient opening of the other eye of the user; iii) improper finger placement of the user relative to the eye; and iv) the lens is too high or too low.

[0115] Embodiment 42: The method of embodiment 39, wherein analyzing the machine learning model is further configured to identify success patterns and provide positive feedback to the user via the user interface.

[0116] Many operating systems (including Linux, and and ) is able to run many tasks simultaneously and is called a multitasking operating system. Multitasking is the ability of an operating system to execute more than one executable task at the same time. Each executable task runs with its own address space, which means that the executable tasks cannot share their memory. Therefore, it is impossible for any program to damage the execution of any other program running on the system. However, programs cannot exchange any information through the operating system (or by reading files stored on the file system).

[0117] Multi-process computing is similar to multi-tasking computing in that the terms task and process are often used interchangeably, although some operating systems distinguish between the two. The present invention may be or include a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon, which are used to cause a processor to perform various aspects of the present invention. A computer-readable storage medium may be a tangible device that can save and store instructions for use by an instruction execution device.

[0118] The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or a raised structure in a groove with instructions recorded thereon, and any suitable combination of the foregoing.

[0119] As used herein, computer-readable storage media should not be understood as transient signals themselves, such as radio waves or other free propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber-optic cables), or electrical signals transmitted by wires. Computer-readable program instructions as described herein can be downloaded from computer-readable storage media to corresponding computing / processing devices, or downloaded to external computers or external storage devices via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). A network may include copper transmission cables, transmission optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from a network and forwards computer-readable program instructions to be stored in a computer-readable storage medium in a corresponding computing / processing device.

[0120] The computer-readable program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages, such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., by using the Internet of an Internet service provider).

[0121] In some embodiments, to perform various aspects of the present invention, electronic circuits including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions to personalize the electronic circuits by utilizing state information of the computer-readable program instructions.

[0122] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box in the flowchart illustration and / or block diagram and the combination of boxes in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to generate a mechanism so that instructions executed by the processor of the computer or other programmable data processing device create instructions for implementing the flowchart and / or block diagram. Figure 1 A tool that specifies functions / actions in one or more boxes.

[0123] These computer-readable program instructions may also be stored in a computer-readable storage medium, which may guide a computer, a programmable data processing device, and / or other apparatus to work in a specific manner, so that the computer-readable storage medium having instructions stored therein includes an implementation flowchart and / or block diagram. Figure 1The computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other apparatus, so that a series of operation steps are executed on the computer, other programmable device, or other apparatus to produce a computer-implemented process, so that the instructions executed on the computer, other programmable device, or other apparatus implement the flowchart and / or block diagram. Figure 1 The functions / actions specified in one or more boxes. The flowcharts and block diagrams in the accompanying drawings illustrate the possible specific implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, fragment or portion of an instruction, which includes one or more executable instructions for implementing a specified logical function.

[0124] In some alternative implementations, the functions indicated in the frame may not occur in the order indicated in the accompanying drawings. For example, depending on the functions involved, the two frames shown in succession may actually be performed substantially simultaneously, or in reverse order. It should also be noted that each frame in the block diagram and / or the flow chart diagram and the combination of frames in the block diagram and / or the flow chart diagram can be implemented by a dedicated hardware-based system that performs a specified function or action or by a combination of dedicated hardware and computer instructions. Although specific embodiments of the present invention have been described, it will be appreciated by those skilled in the art that there are other embodiments that are equivalent to the described embodiments. Therefore, it should be understood that the present invention is not limited by the specific exemplified embodiments, but is only limited by the scope of the appended claims.

[0125] As can be seen from the above description, the present invention provides systems, computer program products and methods for effectively executing the technology. Unless explicitly stated otherwise, references to singular elements in the claims are not intended to mean "one and only one", but "one or more". All structural and functional equivalents of the elements of the above-described example embodiments currently known or later known to those of ordinary skill in the art are intended to be covered by the claims of the present invention. Unless explicitly stated using the phrase "device for..." or "step for...", the claim elements herein should not be interpreted according to the provisions of the sixth paragraph of Section 112 of 35 U.S.C.

[0126] Although the foregoing written description of the present invention enables a person of ordinary skill to make and use what is currently considered to be the best mode of the present invention, a person of ordinary skill will understand and appreciate the existence of alternatives, adaptations, variations, combinations and equivalents of the specific embodiments, methods and examples herein. It will be understood by those skilled in the art that the present disclosure is merely exemplary and that various modifications may be made within the scope of the present invention. In addition, although specific features of the teachings have been disclosed with respect to only one of several specific implementations, such features may be combined with one or more other features of other specific implementations, as may be desired and advantageous for any given or specific function. In addition, with respect to the terms "containing", "including", "having", "with" or variations thereof used in the specific embodiments and claims, such terms are intended to be inclusive in a manner similar to the term "comprising".

[0127] Other embodiments of the present teachings will be apparent to those skilled in the art in view of the description and practice of the present teachings disclosed herein. Therefore, the present invention should not be limited to the embodiments, methods and examples described, but rather encompasses all embodiments and methods within the scope and spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments shown herein, but is only limited by the following claims.

Claims

1. A method for contact lens assistance, the method comprising: causing outputting, via a user interface, one or more lens type options; receiving, via the user interface, an indication of a selected lens type from among the one or more lens type options; causing output of one or more lens operation options via the user interface; receiving, via the user interface, an indication of a selected lens operation of the one or more lens operation options; causing output of one or more eye options via the user interface; receiving, via the user interface, an indication of a selected eye from among the one or more eye options; causing output, via the user interface and based on one or more of the selected lens type, the selected lens operation, or the selected eye, one or more user instructions for inserting or removing a contact lens, The one or more user instructions include user feedback based on images of the user captured in real time.

2. The method according to claim 1, wherein: The lens type options include an indication of one or more of reusable or daily disposable.

3. The method according to claim 1, wherein: The one or more lens operation options include an indication of one or more of insertion or removal of a contact lens.

4. The method according to claim 1, wherein: The one or more eye options include a representation of one or more of a left eye or a right eye.

5. The method according to claim 1, wherein: One or more user instructions for inserting or removing a contact lens are caused to be output for the selected eye, and the method also includes automatically causing, via the user interface and based on one or more of the selected lens type or the selected lens operation, and for unselected eyes in the one or more eye options, the output of one or more user instructions for inserting or removing a contact lens.

6. The method according to claim 1, wherein: The one or more user instructions are selected from the group consisting of: wash and dry instructions, contact lens handling instructions, contact lens position instructions, contact lens orientation instructions, finger position instructions, camera configuration instructions, user feedback queries, and combinations thereof.

7. A method for contact lens assistance, the method comprising: receiving one or more first images of at least a portion of a user, wherein the one or more first images include a representation of an eye of the user; causing, via a user interface and based on the one or more first images, outputting a visual cue indicating a user status of the user relative to the eyes of the user, wherein the visual cue has a first state indicating a rejected user status and a second state indicating an accepted user status; causing output of one or more user instructions for inserting a contact lens in response to the accepted user status; receiving one or more second images of at least a portion of a user, wherein the one or more second images include a representation of the eye of the user and a representation of a contact lens on the eye of the user; analyzing placement of the contact lens on the eye of the user based on the one or more second images; as well as Based on analyzing the placement of the contact lens, an indication of proper placement or improper placement of the contact lens is output.

8. The method of claim 7, wherein the one or more first images include a representation of an iris of the eye of the user.

9. The method according to claim 7, wherein: The user state includes one or more of: an openness of the eye of the user, a characteristic of the other eye of the user, and a finger placement of the user relative to the eye.

10. The method of claim 7, further comprising providing output prompts to the user via a user interface and based on the one or more first images until an accepted user state is detected.

11. The method according to claim 7, wherein: The one or more user instructions for inserting a contact lens include audio instructions, visual instructions, or both.

12. The method according to claim 7, wherein: The analysis includes one or more of the following: i) detecting an edge of the contact lens using an edge detection model; ii) pre-processing the one or more second images using one or more of cropping, masking, removing, or a combination thereof; iii) pre-processing the one or more second images by digitally removing artifacts from the one or more second images; as well as iv) detecting the edge of the contact lens using a mask for edge detection.

13. The method according to claim 7, wherein: The one or more first images or the one or more second images are captured from a video and optionally from a real-time video.

14. A method for contact lens assistance, the method comprising: receiving one or more images of at least a portion of a user, wherein the one or more images include a representation of an eye of the user and a representation of a contact lens on the eye of the user; analyzing placement of the contact lens on the eye of the user based on the one or more images; as well as Based on analyzing the placement of the contact lens, an indication of proper placement or improper placement of the contact lens is output.

15. The method according to claim 14, wherein: The analysis includes one or more of the following: i) detecting an edge of the contact lens using an edge detection model; ii) pre-processing the one or more images using one or more of cropping, masking, removal, or a combination thereof; iii) pre-processing the one or more images by digitally removing artifacts from the one or more second images; as well as iv) detecting the edge of the contact lens using a mask for edge detection.

16. The method according to claim 14, wherein: The one or more images are captured from video and optionally from real-time video.

17. A method for contact lens assistance, the method comprising: a. receiving a real-time video of at least a portion of a user, wherein the video includes a representation of the user's eyes; b. analyzing the user status of the user based at least on the video to determine whether the user status is an accepted user status or a rejected user status; c. causing, via a user interface and based on the video, outputting a visual cue indicating the user status of the user relative to the eyes of the user, wherein the visual cue has a first state indicating a rejected user status and a second state indicating an accepted user status; d. causing, via the user interface and based on the video and based on the user status, one or more real-time prompts to be output to the user; and e. Repeat steps ad until the user status is determined to be an accepted user status.

18. The method according to claim 17, wherein: The accepted user states include one or more of: a contact lens being properly inserted onto the eye of the user, and a contact lens being properly removed from the eye of the user.

19. The method according to claim 17, wherein: The one or more real-time prompts include instructions for inserting or removing a contact lens.

20. A method for contact lens assistance, the method comprising: a. receiving a representation of at least a portion of the user's face during an attempt by the user to insert a contact lens into the eye; b. analyzing the representation to determine whether the user is in a known failure mode, wherein the analyzing comprises applying a machine learning model configured to identify the known failure mode based on the aggregate representation of the user during the known failure mode; c. Providing real-time prompts via a user interface that are known to assist the user in successfully inserting the contact lens into the eye.

21. The method according to claim 20, wherein: The representation includes one or more of lidar data and video data.

22. The method according to claim 20, wherein: The failure modes include one or more of the following: i) insufficient opening of the eye of the user; ii) insufficient opening of the other eye of the user; iii) improper finger placement of the user relative to the eye; and iv) a lens that is too high or too low.

23. The method according to claim 20, wherein: Analyzing the machine learning model is further configured to identify successful patterns and provide positive feedback to the user via the user interface.

Images