Real-time detection of artifacts in ophthalmic images

Abstract

Certain aspects of the present disclosure provide a system for processing image data from an intraoperative diagnostic device in real time during an ophthalmic procedure. The system includes an image capture element that captures a grayscale image of a first size and an image processing element that scales the grayscale image from the first size to a second size. The system also includes a two-stage classification model including a feature extraction stage that processes the scaled grayscale image and generates a feature vector based on the scaled grayscale image, and a classification stage that processes the feature vector and generates an output vector. The image processing element is further configured to determine an image quality of the captured grayscale image based on the output vector for display to an operator, the image quality of the captured grayscale image being indicative of a probability that the captured grayscale image includes an artifact.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit and priority of U.S. Provisional Patent Application No. 63 / 135,125, filed on January 8, 2021, entitled "Real - Time Detection of Multilabel Image Artifacts in an Ophthalmic Instrument Using a Convolutional Neural Network / Deep Neural Network Model", and U.S. Patent Application No. 17 / 236,908, filed on April 21, 2021, entitled "Real - Time Detection of Artifacts in Ophthalmic Images", which are hereby incorporated by reference in their entirety.

[0002] Introduction Aspects of the present disclosure relate to systems and methods for detecting artifacts in image data used during surgical procedures, such as cataract surgery, that enable improved surgical outcomes for patients.

Background Art

[0003] Cataract surgery generally involves replacing the natural lens of a patient's eye with an intraocular lens (IOL). During cataract surgery, a physician may utilize various image - based measurement systems to analyze a patient's eye in real - time, such as to ensure proper selection, placement, and orientation of the IOL for cataract intervention, and to assist in performing the cataract procedure. However, artifacts present in the imaging data of a patient's eye can lead to measurement errors that are unknown or unnoticed by medical practitioners, and as a result, can reduce the effectiveness of such procedures and lead to poor patient outcomes. In many cases, such results require additional surgical intervention.

Summary of the Invention

Problems to be Solved by the Invention

[0004] Therefore, improved techniques are needed to perform image data processing and analysis during procedures such as cataract surgery, which will lead to improved surgical outcomes for patients. [Means for solving the problem]

[0005] A particular embodiment provides a system for processing image data from an intraoperative diagnostic device in real time during an ophthalmic procedure. The system comprises an image acquisition element configured to acquire a grayscale image of a patient's eye from an intraoperative diagnostic device, the grayscale image having a first size. The system further comprises an image processing element configured to acquire the grayscale image from the image acquisition element, scale the grayscale image from the first size to a second size, and preprocess the scaled grayscale image in preparation for classification. The system also comprises a two-stage classification model including a feature extraction stage configured to process the scaled grayscale image and generate a feature vector based on the scaled grayscale image, and a classification stage configured to process the feature vector and generate an output vector based on the feature vector. The image processing element is further configured to determine the image quality of the acquired grayscale image based on the output vector for display to an operator, the image quality of the acquired grayscale image indicating the probability that the acquired grayscale image contains artifacts.

[0006] Another embodiment provides a method for processing image data acquired from an intraoperative diagnostic device in real time during an ophthalmic procedure. The method includes acquiring a grayscale image of a patient's eye from an intraoperative diagnostic device, wherein the grayscale image has a first size; obtaining the grayscale image from an image acquisition element; and preprocessing the grayscale image in preparation for classification by a two-stage machine learning model. The method further includes generating a feature vector based on the preprocessed grayscale image by the feature extraction stage of the two-stage machine learning model; and generating an output vector based on the feature vector by the classification stage of the two-stage machine learning model. The method also includes determining the image quality of the acquired grayscale image based on the output vector for display to an operator. The image quality of the acquired grayscale image indicates the probability that the acquired grayscale image contains artifacts that interfere with measurements by the intraoperative diagnostic device.

[0007] Another embodiment provides a method for training a two-stage machine learning model to identify artifacts in images acquired from an intraoperative aberration analyzer during an ophthalmic procedure. The method includes acquiring images, generating feature vectors for each image by a feature extraction stage of a two-stage machine learning model, generating a feature matrix based on the stacking of the generated feature vectors, and training a classification stage based on the feature matrix. The trained classification stage generates an output for the processed image indicating the probability that the image contains artifacts.

[0008] Other embodiments provide a processing system configured to perform the methods described above and the methods described herein, a processing system comprising, when executed by one or more processors of the processing system, a non-temporary computer-readable medium containing the methods and instructions causing the processing system to perform the methods described herein, a computer program product embodied on a computer-readable storage medium containing code for performing the methods described above and the methods further described herein, and means for performing the methods described above and the methods further described herein.

[0009] The following description and related drawings detail specific exemplary features of one or more embodiments.

[0010] The accompanying drawings illustrate specific aspects of one or more embodiments and should therefore not be considered to limit the scope of this disclosure. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 shows a block diagram of an imaging system for capturing digital images of a patient's eye during surgery, diagnosis, or other procedures, according to a specific embodiment. [Figure 2A-2B] Figure 2A shows a data flow for processing individual images by a machine learning model implemented by the system in Figure 1, according to a particular embodiment, and Figure 2B shows a set of data flows for processing multiple images by different machine learning models, according to a particular embodiment. [Figure 3A-3C] Figure 3A shows the architecture of a convolutional neural network (CNN) applied as the feature extraction stage of a machine learning model implemented by the system in Figure 1, according to a particular embodiment. Figure 3B shows a representative diagram of the CNN architecture applied to the first stage (i.e., the feature extraction stage) of the machine learning model in Figures 2A and 2B, according to a particular embodiment. Figure 3C shows the architecture of the second stage (i.e., the classification stage) of the machine learning model of the system in Figure 1, according to a particular embodiment, which generates an output vector based on the feature vectors generated by the first stage for each individual image of the captured digital image. [Figure 4A-4G] Figures 4A-4G show images that may be present in an image dataset according to a particular embodiment, and / or captured by one or more cameras of the system in Figure 1. [Figure 5]Figure 5 illustrates a method, according to a specific embodiment, for identifying digital images containing one or more artifacts detrimental to image processing and analysis using a machine learning model. [Figure 6] Figure 6 illustrates a method, according to a specific embodiment, for training a machine learning model to identify digital images containing one or more artifacts detrimental to image processing and analysis. [Figure 7] Figure 7 is a diagram illustrating an embodiment of a processing system that implements or embodies an aspect described herein, according to a specific embodiment. [Figures 8A-8B] Figures 8A-8B show a display concept for providing a user with a digital image of a patient's eye along with details of artifacts detected via a graphical user interface, according to a particular embodiment. [Figure 8C-8D] Figures 8C-8D show a display concept for providing a user with a digital image of a patient's eye along with details of artifacts detected via a graphical user interface, according to a particular embodiment. [Figure 8E-8F] Figures 8E-8F are display concepts for providing a user with a digital image of a patient's eye along with details of artifacts detected via a graphical user interface, according to a particular embodiment. [Figure 8G-8H] Figures 8G-8H show a display concept for providing a user with a digital image of a patient's eye along with details of artifacts detected via a graphical user interface, according to a particular embodiment. [Figure 8I-8J] Figures 8I-8J are display concepts for providing a user with a digital image of a patient's eye along with details of artifacts detected via a graphical user interface, according to a particular embodiment. [Figure 8K-8L] Figures 8K-8L are display concepts for providing a user with a digital image of a patient's eye along with details of artifacts detected via a graphical user interface, according to a particular embodiment. [Figure 8M-8N]Figures 8M-8N show a display concept for providing a user with a digital image of a patient's eye along with details of artifacts detected via a graphical user interface, according to a particular embodiment. [Figure 8O] Figure 8O is a display concept for providing a user with a digital image of a patient's eye along with details of artifacts detected via a graphical user interface, according to a particular embodiment. [Modes for carrying out the invention]

[0012] To facilitate understanding, the same reference numerals are used whenever possible to indicate identical elements common to each drawing. Elements and features of one embodiment are intended to be usefully incorporated into other embodiments without further explanation.

[0013] Aspects of this disclosure provide apparatus, methods, processing systems, and computer-readable media for performing image data processing and analysis during medical procedures. In various examples described herein, the medical procedures relate to the human eye, such as cataract surgery, and the images may be provided by diagnostic devices such as intraoperative aberration analyzers.

[0014] Intraoperative aberration measurement is generally a process that enables a surgeon to perform refractive measurements in an operating room to assist in the selection and placement of an intraocular lens (IOL) power. In some cases, an intraoperative aberration measurement system can measure a "wavefront" that represents the propagation of light waves through a patient's eye. In particular, an intraoperative aberrometer can be configured to identify the aberration (distortion) of light waves caused by irregularities in the patient's eye that converge the light waves irregularly onto the retina. Cataracts are one such irregularity that causes suboptimal operation of the eye. For example, replacement of a patient's natural lens with an IOL requires extreme precision to produce the best patient outcomes. Instruments such as intraoperative aberrometers are in principle very effective for this purpose, but in practice, various common conditions can reduce their effectiveness and potentially compromise surgical intervention. For example, visual artifacts in the image data generated and / or processed by an aberrometer, such as illumination glints, motion artifacts, floating objects or bubbles in the eye's fluid, excessive moisture or dryness of the eye, debris on the optical device, can lead to refractive measurement errors, and as a result, selection and placement errors and poor patient outcomes. Furthermore, such visual artifacts may be easily overlooked by busy practitioners managing complex procedures, complex instruments, and patients at once.

[0015] To address the drawbacks of conventional systems and enable more reliable refractive measurements, more accurate surgical interventions, and better patient outcomes, the embodiments described herein implement a machine learning model (artificial intelligence) that analyzes image data and identifies artifacts that may degrade the quality of refractive measurements.

[0016] In addition to identifying such artifacts, the embodiments described herein can prophylactically prevent refractive measurement errors, for example, by filtering image data (e.g., an image data frame) that includes the identified artifacts. Filtering such data can advantageously prevent a measurement device from making inaccurate measurements and inaccurate decisions based on those measurements.

[0017] Furthermore, embodiments described herein can actively show a medical practitioner the probability that artifacts in real-time image data are processed by an aberrometer, for example, based on real-time analysis by a machine learning model. In various embodiments, the systems described herein can be configured to generate graphical user interface elements for showing detected artifacts, the potential for measurement errors based on the detected artifacts, and the like. In this way, the embodiments described herein offload this task from the practitioner and enable the practitioner to perform more accurate procedures based on more accurate and complete information, resulting in better patient outcomes. Based on such graphical user interface elements and similar displays, the practitioner can adjust the system (e.g., adjust the camera angle of the aberrometer or the position of the patient's eye, clean the imaging sensor or components, reposition instruments used during the procedure, etc.) to improve the quality of the image data and thereby improve the quality and accuracy of the refractive measurements.

[0018] In particular, often the systems and methods described herein can identify artifacts that are not easily (or at all) distinguishable by the medical practitioners using these systems. For example, small artifacts, scattered artifacts, intermittent artifacts, or transient artifacts, etc. may be significant enough to cause refractive measurement errors but may not be noticed by even the most trained human practitioners. Thus, the systems and methods described herein provide a technical improvement over existing techniques that cannot identify, show, or mitigate the presence of such artifacts.

[0019] Embodiments described herein may utilize multi-stage machine learning models, for example, to identify artifacts in image data used by an intraoperative aberration analyzer. In one example, a two-stage machine learning model includes a first or front-end stage configured to extract features from image data. Features may be created, for example, in the form of feature vectors. The two-stage machine learning model further includes a second or back-end stage configured to perform classification. In some cases, the classification stage may be configured to generate an output vector indicating the probability that the image processed by the first stage contains any artifacts, based on the feature vectors of the image. Combined, the feature extraction (i.e., the first) stage and the classification (i.e., the second) stage can detect one or more artifacts in the processed image data.

[0020] Beneficially, a multi-stage model architecture (e.g., a two-stage architecture) allows for the individual, modular training and implementation of each stage. In this way, different (and improved) classification stages can be implemented without the need to retrain or redesign the feature extraction stage. This modularity, in turn, improves (reduces) training time and resource usage, allowing the entire model to be easily and frequently developed, thereby sequentially improving the intraoperative aberration measurement system described herein, and ultimately the quality of treatment and patient outcomes.

[0021] The image analysis and classification systems, methods, and techniques described herein are described in relation to one or more procedures performed intraoperatively using an intraoperative aberration analyzer, but in certain embodiments, the image analysis and classification systems, methods, and techniques described herein may also be used preoperatively and / or postoperatively. For example, the image analysis and classification systems, methods, and techniques described herein may be used during preoperative procedures to acquire measurements and images for preparing a surgical plan in preparation for the surgical procedure. Similarly, the image analysis and classification systems, methods, and techniques described herein may be used postoperatively, for example, to check and / or verify the results from a procedure. Furthermore, the image analysis and classification systems, methods, and techniques described herein may also be used preoperatively, intraoperatively, and / or postoperatively in conjunction with other optical imaging devices (e.g., other than aberration analyzers).

[0022] Examples of intraoperative imaging systems Figure 1 shows a block diagram of an exemplary imaging system (hereinafter referred to as the system or imaging system) 100 for capturing a digital image of a patient's eye 110 during surgery, diagnosis, or other procedures. The system 100 includes a microscope 102, an aberration meter 104, a controller 106, a user interface 108, and a representation of the patient's eye.

[0023] The microscope 102 may comprise one or more optical mechanisms, observation mechanisms, illumination mechanisms, and / or control mechanisms. The optical mechanisms may include one or more lenses for focusing light reflected by a target object viewed through the microscope 102, such as a patient's eye, during any procedure described herein. Thus, the microscope 102 enables an operator (e.g., a medical practitioner such as a surgeon, nurse, assistant, or specialist) to view the patient's eye (or part thereof) at a greater magnification than what can be seen with the naked eye, and to have additional features such as identifying or marking features. The observation mechanism, which may include one or more eyepieces or computerized interfaces, may include at least one optical channel having at least one optical lens disposed therein. The observation mechanism may be monocular or binocular, enabling the operator to view a target object at magnification. One or more embodiments of the optical mechanisms and / or observation mechanisms may be adjustable by the operator as needed or in an automated manner with respect to the optical mechanisms, focusing of the observation mechanism, positioning of the patient's eye, etc. In some embodiments, the optical mechanisms and observation mechanisms include the optical path of the microscope 102.

[0024] The illumination mechanism may include a light source configured to provide and / or project visible light into the optical path of the microscope 102. The illumination mechanism may be adjustable by the operator as needed, or in an automated manner, with respect to positioning, focusing, or otherwise directing the visible light.

[0025] The control mechanism may allow the operator to manually operate and / or adjust other mechanisms of the microscope 102. For example, the control mechanism may include components that enable adjustment of the illumination mechanism (e.g., controls to turn the illumination mechanism on / off and / or adjust the light level, focus, etc.). Similarly, the control mechanism may include components that enable adjustment of the optical mechanism (e.g., enabling automatic or manual focusing of the optical mechanism, or movement of the optical mechanism so that different targets or parts of a target can be viewed, or the magnification of the target can be changed). For example, the control mechanism may include knobs and similar components that enable adjustment of the optical mechanism (e.g., controls to move optical components horizontally and / or vertically, and to increase and / or decrease magnification). Furthermore, the control mechanism may include components that enable adjustment of the observation mechanism, such as focusing elements and filtering elements. In some embodiments, the control mechanism for the observation mechanism is manually and / or automatically adjustable.

[0026] In some embodiments, the microscope 102 is used in conjunction with or replaces one or more diagnostic devices. The operator may use the microscope 102 during any medical or diagnostic procedure to magnify the patient's eye (or part thereof) for better visibility during the procedure. Furthermore, the operator may use the microscope 102 (or other diagnostic devices) to acquire single and / or multidimensional images and / or other measurements of the patient's eye 110. The microscope 102 may comprise a three-dimensional stereoscopic digital microscope (e.g., NGENUITY® 3D Visualization System (Alcon Inc., Swierland)). One or more diagnostic devices may be any of a number of devices for acquiring and processing single and / or multidimensional camera-based (or similar) images and / or measurements of the anatomical structure of the eye, such as optical coherence tomography (OCT) devices, rotating cameras (e.g., Scheinproof cameras), magnetic resonance imaging (MRI) devices, corneal meters, ophthalmic meters, and / or optical biometers.

[0027] The aberration meter 104 may include a light source that generates a light beam directed into the patient's eye 110 via a combiner mirror or beam splitter. The light beam directed into the patient's eye 110 is reflected from the patient's eye 110 to the aberration meter 104 via the combiner mirror or beam splitter. The reflected light beam is further reflected and refracted by the aberration meter 104 before being diffracted and imaged into an image captured by the aberration meter 104. For example, the aberration meter 104 may include at least one camera, photodetector, and / or similar sensor configured to capture, record, and / or detect an image of the patient's eye and convert it into a computer-readable format. In some embodiments, the at least one camera is not part of the aberration meter 104 but is a standalone component that generates an image of the patient's eye based on information received from the aberration meter 104, as further described below.

[0028] The aberration meter 104 (or other wavefront sensor or diagnostic device) may be positioned between the microscope 102 and the patient's eye 110. For example, the aberration meter 104 may include an optical device for reflecting light, such as a combiner mirror or a beam splitter. The optical device may selectively reflect a portion of the electromagnetic spectrum (e.g., the visible light portion of the electromagnetic spectrum) to the aberration meter 104 for processing, analysis, and / or measurement, while allowing other portions of the electromagnetic spectrum (e.g., the infrared portion of the electromagnetic spectrum) to pass through the optical device and enter the microscope 102 for the operator to view. Alternatively, although not shown in Figure 1, the system 100 may include an optical device positioned between each of the aberration meter 104 and the microscope 102 and the patient's eye 110, thereby directing light to either the aberration meter 104 or the microscope 102 without passing one through the other.

[0029] In some embodiments, system 100 may include one or more cameras and / or imaging systems (not shown in Figure 1) configured to capture images of different fields of view and / or viewpoints of the patient's eye 110. In some cases, one or more camera systems are positioned differently relative to the patient's eye 110, the microscope 102, and / or the aberration meter 104. In some cases, the different camera systems may use one or more different light sources (e.g., light-emitting diodes (LEDs), lasers, etc.) operating at different wavelengths (e.g., in the visible light spectrum, infrared spectrum, etc.).

[0030] In some cases, one or more cameras may provide the controller 106 with multiple types of images or image views. Each type of image or image view may capture different information and / or aspects of the patient's eye 110. For example, multiple image views or types may include a wide-field view illuminated by light having wavelengths in the visible spectrum (of the electromagnetic spectrum), a focus view illuminated by 840 nanometer (nm) wavelength LED light, and an interferogram view illuminated by 740 nm wavelength light.

[0031] The combination of the microscope 102 and the aberration meter 104 allows for observation and measurement of the patient's eye 110 during the planning and execution of various procedures. The microscope 102 and the aberration meter 104 can each be focused on a point occurring, for example, on the surface of the patient's eye 110, such that the field of view of the aberration meter 104 overlaps at least partially with the field of view of the microscope 102, and the patient's eye 110 remains positioned within the overlapping portion of the field of view during the procedure. In some cases, the microscope 102 and the aberration meter 104 are focused on substantially the same point, such that the center of each of their fields of view is located at approximately the same point on the patient's eye 110. Thus, the operator can view the patient's eye 110 through the microscope 102 while the aberration meter 104 (and / or one or more cameras) generates an image of the patient's eye 110.

[0032] More specifically, one or more cameras (either part of the aberration meter 104 or outside of the aberration meter 104) can convert information from the aberration meter 104 into a computer-readable format. The controller 106 can acquire images from the cameras and measure and analyze the images captured by one or more cameras (i.e., the information from the aberration meter 104 is converted into a computer-readable format). The controller 106 can quantify the characteristics of the captured images, and therefore the refractive characteristics of the patient's eye 110 examined during the procedure.

[0033] Different image views may capture different aspects of the patient's eye 110. In some embodiments, a wide-field view may provide the operator with a complete frontal view of the patient's eye 110, allowing for centering of the patient's eye 110 within the field of view for each of the other field types. Such a wide-field image of the patient's eye 110 may contain one or more artifacts that indicate one or more conditions that the operator or system 100 may need to recognize. For example, a wide-field image may contain artifacts caused, for example, by debris on an optical element or combiner mirror / beam splitter, or caused by equipment used during the procedure being too close to the cornea of ​​the patient's eye 110 (e.g., eyelid retractor).

[0034] The focus view provides image acquisition of light produced by one or more light sources as it reflects off the cornea of ​​the patient's eye 110. Such images may enable the calculation of the distance of the system 100 from the patient's eye 110 (e.g., the distance of the camera and / or aberration meter 104). The focus view image may present artifacts resulting from fluid or hydration changes in one or more parts of the patient's eye 110. For example, the focus view image may include artifacts when light from one or more of the light sources spreads or "collapses" due to the drying of the tear film on the anterior surface of the cornea of ​​the patient's eye 110. In some cases, fluid pools (from naturally occurring or supplemented tears) may diffuse or extend the light from the light source in a certain direction, causing "leg" artifacts. Furthermore, excessive motion between even and odd frames captured by the focus view image (e.g., captured by an analog camera) may generate interleaved artifacts of light reflection.

[0035] The interferogram view, once processed, may allow for the capture of an image stream that provides the operator with ocular aberration data, including real-time refractive data. The interferogram view image may include artifacts caused by the presence of bubbles captured in the image, illumination glints (which may correspond to increased reflection of light from the patient's eye 110), floating debris within the patient's eye 110, and general distortions on top of a typical moiré spot pattern.

[0036] Any image type and / or image acquired by the aberration meter 104 or imaging device may contain one or more of the identified artifacts described above, or it may not contain any artifacts.

[0037] The controller 106 may identify whether an image contains one or more artifacts by applying one or more machine learning models, as described, for example, with respect to Figures 2A and 2B. The machine learning model may generally include a feature extraction stage for generating predictive features based on the data received by the controller 106, and a classification stage for predicting whether the received data contains various artifacts (e.g., image data artifacts). One or both of the feature extraction and classification stages may be trained and optimized based on a repository of previously captured images (e.g., images from previous processing). The repository may include images that have been manually classified (e.g., labeled) for whether they contain one or more artifacts, and images that, if an image contains at least one artifact, have been classified (e.g., labeled) for what type of artifact is present in the image. The machine learning model may be trained with several previously labeled images to improve its ability to identify artifacts in images. Further details regarding the machine learning model are provided below with respect to Figures 2A to 3C.

[0038] Images 112 generated by the aberration meter 104 and / or the camera of system 100 are displayed to the operator during the procedure. In some embodiments, the user interface 108 may display the images 112 for observation and / or manipulation by the operator. In some cases, the user interface 108 may also present the operator with information about the images 112 after analysis and processing by the controller 106 using one or more machine learning models.

[0039] The user interface 108 may present one or more image quality indicators, such as a quality bar graph, and the image values ​​displayed on the user interface 108. For example, the user interface 108 may show the operator that a particular image or series of images contains a first artifact (e.g., glint) with a 4% probability, a second artifact (e.g., air bubbles) with a 94% probability, a third artifact (e.g., debris) with a 1% probability, and no artifacts with a 1% probability. Thus, the system 100 enables the operator to quickly and significantly monitor the quality of image data acquired by the system 100, which beneficially improves the quality of the procedures being performed and the final patient outcomes.

[0040] The user interface 108 can identify whether a particular image contains or does not contain various types of artifacts (as described above) by reducing the image quality value when the image contains one or more artifacts, as provided by the system 100 or the central processing system. The operator may use the quality bar graph or quality value and corresponding information to decide whether to exclude the image from processing, such as measurement, and provide such a decision to the controller of the system 100 or the central processing system. In some embodiments, the controller 106 may automatically determine, based on the quality value, when to exclude the image from processing for measurement. For example, if the displayed image also indicates a high probability of containing at least one artifact, the automated processing by the operator or controller 106 may determine that the image should be excluded from measurement generation based on that. On the other hand, if the displayed image indicates a low probability of containing artifacts, the automated processing by the operator and / or controller 106 may determine that measurement should be generated based on the image. Further details regarding the quality bar graph and values ​​are provided below.

[0041] Therefore, the system 100 can be used in surgery to acquire an image 112 of the patient's eye 110, evaluate the quality of the image 112 via a machine learning model, and determine whether the image 112 contains one or more artifacts.

[0042] While the various embodiments described herein are described, for example, in relation to the eye or similar surgeries and procedures, it should be noted that the techniques described herein can be applied to other medical imaging settings such as X-ray images, magnetic resonance imaging (MRI) scans, and computed tomography (CT) scans.

[0043] Exemplary data flow for classifying images during medical procedures Figure 2A shows the data flow 200 for processing individual input images 202 by a machine learning model 203 implemented by the system 100 in Figure 1.

[0044] In short, the data flow 200 includes receiving an image 112 from an aberration meter 104. The image 112 may be preprocessed by a preprocessing module 201 to generate an input image 202 for processing by a machine learning model 203. In this example, the machine learning model 203 includes two stages: a first stage 204 that generates a feature vector 206, and a second stage 208 that generates at least one output vector representing one or more artifact probabilities 210, 212, and 214, for example, the input image 202 containing the corresponding artifact type. The machine learning model 203 may process each input image 202 to classify whether the input image 202, and therefore the corresponding image 112, contains one or more artifacts or does not contain artifacts.

[0045] In some embodiments, the data flow 200 may occur or be executed in the controller 106 or a similar processing component in Figure 1. In some embodiments, the data flow 200 may occur or be executed in a separate computing system (not shown) which may comprise one or more computing devices. In some cases, the separate computing system may apply machine learning models to images from multiple systems 100. For example, an ophthalmic practice may include multiple systems 100 used during surgery, diagnosis, or other procedures. Each of these systems 100 may communicate with a separate computing system which can apply machine learning models to captured images of the patient's eye for each of the systems 100. In some embodiments, the separate computing system may be locally distributed, cloud-based, or a combination thereof.

[0046] In some cases, artifact probabilities 210, 212, and 214 generated by machine learning model 203 or a similar machine learning model may indicate the probability that the image 112 processed according to data flow 200 contains one or more artifacts.

[0047] In some cases, the preprocessing module 201 is configured to preprocess the image 112 for processing by the machine learning model 203. Specifically, the preprocessing module 201 may receive the image 112, identify regions of interest within the image 112, and / or transform one or more aspects of the image 112 when preparing the input image 202. Regions of interest may be generated based on the identification of a specific geographic region, such as the central region of the image 112. In some embodiments, the preprocessing module 201 may use intelligence to identify regions of interest (e.g., one or more aspects of image analysis). Furthermore, the preprocessing module 201 may also transform the pixel format, such as scaling the image 112 and / or converting the pixel format of the image 112 (generated by the aberration meter 104) to a format compatible with the machine learning model 203. Furthermore, the preprocessing module 201 may adjust the number of channels in the image 112.

[0048] Image 112 may be captured by one of the cameras introduced above (for example, based on information from aberration meter 104). Image 112 may be captured with a first color profile and / or size. For example, Image 112 may be a color image with a size of 640 × 480 pixels. If the image is a color image, Image 112 may contain three data channels, such as a red channel, a green channel, and a blue channel, each having corresponding color data. Thus, the preprocessing module 201 may preprocess Image 112 to resize it and ensure that Image 112 contains the expected number of channels. For example, the machine learning model 203 may have an input image parameter size (height in pixels (H) × width in pixels (W) × number of channels (C)) of 480 × 480 pixels × 3 channels (for example, the red channel, green channel, and blue channel of a color image).

[0049] Therefore, for image 112, which is a color image with a size of 640 × 480 pixels, the preprocessing module 201 may resize image 112 to a size of 480 × 480 pixels (for example, by cropping the central region of interest) and maintain the color channels to generate input image 202 as a color image with a size of 480 × 480 pixels. Alternatively, the machine learning model 203 may have input parameter pixel size of any other values ​​and channel requirements, as established by the first step 204, which is described in more detail below. In some cases, the color image may contain a different number of channels of different color components in the color model or color space of the color image. For example, the color image may utilize one or more of the cyan, magenta, yellow, black (CMYK) color model or the luminance / chrominance component color space (e.g., Y-Cb-Cr), thereby changing the number of channels used in the corresponding image.

[0050] If image 112 is a grayscale image (as shown in the figure), image 112 may contain only a single data channel. Therefore, the preprocessing module 201 may duplicate the single data channel across three channels (for example, instead of red, green, and blue channels). Such duplication of a single channel may include bands that duplicate the single channel to create three channels. Furthermore, the preprocessing module 201 may resize image 112 as needed, as described above. Thus, regardless of the size and number of channels in image 112, the preprocessing module 201 may process image 112 to produce an input image 202 in the format expected by the machine learning model 203. In some embodiments, the first stage 204 of the machine learning model 203 may require no multiple channels or more than three channels. In such cases, the preprocessing module 201 may process image 112 to create or truncate the appropriate number of channels for the first stage 204.

[0051] The machine learning model 203 may determine whether each input image 202 processed by the machine learning model 203 contains artifacts. For example, the machine learning model 203 may determine whether an interferogram-type input image 202 contains one or more artifacts caused by one or more of the following: illumination glints, floating debris (in the patient's eye 110), bubbles (in the patient's eye 110), or other distortions introduced above. The machine learning model 203 may generate one or more output vectors representing one or more probabilities that the processed image contains one or more artifacts of one or more artifact types. In embodiments where the machine learning model 203 can determine whether the input image 202 contains multiple artifacts, the machine learning model 203 may generate a separate output vector for each artifact. For example, the machine learning model 203 may generate an output vector that includes artifact probability 210 indicating the probability that the input image 202 contains at least one glint artifact, artifact probability 212 indicating the probability that the input image 202 contains at least one debris artifact, and artifact probability 214 indicating the probability that the input image 202 contains at least one bubble artifact. More specifically, in certain embodiments, the machine learning model 203 may output a single vector of length 3 (i.e., having three elements). The three elements of the output vector may correspond to three artifact probabilities (e.g., artifact probability 210, artifact probability 212, and artifact probability 214, as introduced above). Thus, each element of the output vector may classify the image as containing zero or at least one incident of the corresponding artifact, based on the corresponding probability value.

[0052] As introduced above, the machine learning model 203 may include a first stage 204 that generates feature vectors 206 based on the input image 202, and a second stage 208 that generates artifact probabilities 210, 212, and 214. The first stage 204 may include a feature extraction stage that can be configured to generate a representation of the input image 202. For example, the feature vectors generated by the first stage 204 may represent one or more characteristics of the input image 202. In the image processing described herein, features may correspond to various aspects of the image and the pixels that make up the image.

[0053] A second stage 208 of the machine learning model 203 may process the feature vectors generated by the first stage 204 to generate artifact probabilities 210, 212, and 214. The second stage 208 may correspond to or include a classification stage. The classification stage may take the feature vectors generated by the first stage 204 and, if any, identify which artifacts the processed image contains.

[0054] In an exemplary use case, system 100 may take in an image 112 having an image size of 640 × 480 pixels and having a single channel (for example, being a grayscale image). Image 112 may contain one or more bubble artifacts. Controller 106 (or other processing component) may use a preprocessing module 201 to crop image 112 to have a second image size of 480 × 480 pixels and duplicate the single-channel image 112 across three channels to create an input image 202. Controller 106 may then process the cropped and duplicated input image 202 with a machine learning model 203 to generate artifact probabilities 210, 212, and 214. In an captured image 112 having one or more bubble artifacts, the machine learning model 203 may generate artifact probabilities 210 indicating a 1% probability that image 112 contains a glint artifact, artifact probabilities 212 indicating a 24% probability that image 112 contains a floating object artifact, and artifact probabilities 214 indicating a 75% probability that image 112 contains a bubble artifact. Thus, the artifact probabilities indicate a low probability that image 112 contains glint and floating object artifacts (0.01 and 0.24, respectively) and a high probability that it contains a bubble artifact (0.75).

[0055] In some cases, the processing component may use the artifact probabilities generated by the machine learning model 203 to generate the quality bar graph or quality values ​​introduced above, for example, via a user interface. For example, based on the artifact probabilities 210, 212, and 214 identified above, the processing component may generate quality values ​​for display to the operator. For example, the processing component may generate a quality bar graph and / or values ​​based on Equation 1, where artifact_n_probability is the probability generated in the output vector produced by the second stage 208 for the corresponding artifact type: Quality value = 1.0 - artifact_n_probability (Equation 1)

[0056] Therefore, artifact probabilities 210, 212, and 214 indicate that image 112 has a probability of 0.01 that it contains glint artifacts, a probability of 0.24 that it contains floating artifacts, and a probability of 0.75 that it contains bubble artifacts. In the above example, quality bar graphs, quality values, or other indicators for image 112 can be converted to percentages: In the case of a glint artifact, 1.0 - 0.01 = 0.99, or 99%. • In the case of floating object artifacts, 1.0 - 0.24 = 0.76, or 76%, and • In the case of bubble artifacts, 1.0 - 0.75 = 0.25, or 25%.

[0057] In some cases, the controller 106 generates display quality information for the output vector generated by the machine learning model 203 (i.e., for the artifact probability represented by the output vector). Alternatively or additionally, the controller 106 may generate display quality information based on a comparison of these values ​​with a threshold. For example, the controller 106 may generate quality value data for operator review only if the quality of image 112 is below a threshold such as 50% or above a threshold (making the quality value less than 0.5 and the probability greater than 0.5). In some embodiments, the threshold for generating quality value data may be in the range of 25-50% (e.g., 25%, 30%, 40%, 45%, or 50%, or any value in between) or 50-75% (e.g., 50%, 55%, 60%, 65%, 70%, or 75%, or any value in between). The threshold may also be established and / or adjusted based on one or more of the operator-selectable historical data (e.g., variables based on observed trends). Additionally or alternatively, thresholds may be established by the operator or facility. In some embodiments, the controller 106 generates quality value data for operator review of all images, but applies labels to be displayed with the images based on one or more threshold ranges.

[0058] Such threshold ranges may also be used to determine one or more labels (e.g., "Good," "Poor," "Minimum," etc.) for image 112, referring to Figures 8A to 8O. Thus, each image may be displayed or associated with a label based on a different range of image quality values. For example, an image quality of 0% to 50% may correspond to the "Poor Image" label, 51 to 70% to the "Minimum Image" label, and 71 to 100% to the "Good Image" label, or 0% to 65% to the "Poor Image" label, 66 to 85% to the "Minimum Image" label, and 86 to 100% to the "Good Image" label, and so on.

[0059] Therefore, the controller 106 may restrict the operator's review to only those images 112 that meet the threshold quality level (i.e., are more likely to contain one or more artifacts). In some embodiments, the controller 106 may provide the operator with a simplified or general warning or prompt that the quality level threshold was not met by one or more images, and may provide the operator with the option to view further details regarding the individual images that did not meet the threshold quality level and / or the corresponding artifacts that caused the images to not meet the threshold quality level.

[0060] Similarly, the controller 106 may prompt the operator whether the image 112 should be used to generate measurement data. In some cases, the controller 106 may provide the operator with a recommendation to limit further processing of the image 112 in order to prevent or exclude the processing of the image 112 having a quality value below a threshold quality level from being used to generate measurement data. Alternatively, the controller 106 may automatically exclude processing of the image 112 into measurement data without operator input based on the quality value of the image 112. Furthermore, the controller 106 may also provide the operator with one or more recommendations to improve artifacts that cause the image quality threshold level to be not met. For example, the controller may instruct the operator to reposition one or more of the cameras, cleaning equipment, etc.

[0061] Furthermore, as introduced above, the controller 106 may generate a user interface 108 to identify the location of any artifacts in the image 112. In some cases, the controller 106 may implement an additional machine learning model (not shown) to identify the location of artifacts contained in the image 112. By identifying the location of artifacts in the image 112 on the user interface 108, the controller 106 enables the operator to more easily and quickly decide whether or not to use the image 112 to generate intraoperative measurement data.

[0062] In some embodiments, if the controller 106 identifies that the quality of image 112 is below a desired threshold (i.e., it determines that image 112 contains one or more artifacts that reduce the image quality below the threshold), the controller 106 may display a message to the operator indicating that image 112 was not used to generate measurements because its quality was too low. Such a decision may be made automatically without operator input, as described herein. Alternatively or additionally, if the controller 106 determines that the quality of image 112 is sufficiently high, the controller 106 may enable processing of image 112 for measurement generation and provide those measurements, along with image 112 and quality values, to the operator in real time via a user interface during processing.

[0063] The machine learning model 203 described above may process images of a first type, such as an interferogram. While embodiments of the machine learning model 203 may be general for any image type (e.g., the first or feature extraction stage 204), different image types may contain different artifacts that have different characteristics when incorporated into the image 112, so the second or classification stage 208 may use different designs or architectures for different image types (e.g., different layer configurations). Thus, as further described below with respect to Figure 2B, multiple or different combinations of feature extraction stages (i.e., the first stage 204) and different classification stages (i.e., the second stage 208 and additional stages) of the machine learning model may be used to determine whether different types of images 112 contain different types of artifacts.

[0064] Exemplary data flow for classifying images during medical procedures using machine learning models Figure 2B shows a set of data flows 250a-250c for processing multiple images 112a, 112b, and 112c, each having different machine learning models 203, 217, and 227, respectively. Each of the machine learning models 217 and 227 has a similar structure to machine learning model 203 in Figure 2A. The separate data flows 250a-250c may demonstrate the parallel processing and / or multi-model capabilities of system 100.

[0065] As described above, the aberration meter 104 may provide images having different image types, such as from different image sensors within the aberration meter 104 that simultaneously generate image data. For example, images 112a, 112b, and 112c may be of the wide-field type, focus-view type, and interferogram-view type. Each image type contains different types of artifacts. Thus, each image 112a, 112b, and 112c may be processed by different machine learning models 203, 217, and 227.

[0066] For example, machine learning model 203 in Figure 2B may identify glints, debris, and / or bubbles in an interferogram-view type image 112a. Machine learning model 217 may identify debris (e.g., debris on the combiner mirror or beam splitter introduced above) in a wide-field type image 112b. Machine learning model 227 may identify fluid changes in the patient's eye 110 (e.g., excessive dryness or excessive fluidity in one or more parts of the patient's eye 110) in a focus-view type image 112c. Further details regarding machine learning models 217 and 227 in Figure 2B are provided below.

[0067] Data flow 250a corresponds to data flow 200 in Figure 2A, except that image 112, which is supplied to the preprocessing module 201, is identified as image 112a. The remaining components of data flow 250a correspond to the components of data flow 200 in Figure 2A. As described above with reference to Figure 2A, the second stage 208 obtains the feature vector 206a generated by the first stage 204 and can identify what artifacts, if any, the interferometer-type processed image 112a contains.

[0068] Dataflow 250b includes components similar to those of dataflow 250a, with components of similar number having the properties described with reference to Figure 2A. Dataflow 250b includes image 112b preprocessed by preprocessing module 201 to generate input image 202b. Preprocessing module 201 may preprocess image 112b to generate input image 202b, as described above with respect to preprocessing module 201 in Figure 2A. Input image 202b is processed by machine learning model 217 to generate one or more artifact probabilities, e.g., one or more artifact probabilities 220, 222, and 224. Machine learning model 217 includes a first stage 204 (i.e., a feature extraction stage) that generates a feature vector 206b based on input image 202b, as well as a first stage 204 that generates a feature vector 206 as introduced above with respect to Figure 2A.

[0069] The feature vector 206b generated by the first stage 204 is processed by the second stage 218 (i.e., a classification stage different from the second stage 208) to generate one or more artifact probabilities 220, 222, and 224. The second stage 218 of the machine learning model 217 may process the feature vector 206b generated by the first stage 204 to generate one or more artifact probabilities 220, 222, and 224. The second stage 218 may correspond to or include a classification stage similar to the classification stage of the second stage 208, but trained to classify and / or identify artifact types different from those of the second stage 208. As described above with reference to Figure 2A, the second stage 218 may take the feature vector 206b generated by the first stage 204 and identify which artifacts the wide-field type processed image contains, if any.

[0070] The second stage 218 may start with the same architecture, parameters, weights, etc. as the second stage 208, but may be trained independently and thus evolve to fit the unique features of its input data. For example, the second stage 208 may be trained to generate one or more artifact probabilities 210, 212, and 214 for interferogram image types, while the second stage 218 may be trained to generate one or more artifact probabilities 220, 222, and 224 for wide-field type images 112b.

[0071] Dataflow 250c includes components similar to those of dataflow 250a. Specifically, dataflow 250c includes image 112c, which is preprocessed by preprocessing module 201 to generate input image 202c. Preprocessing module 201 may preprocess image 112c to generate input image 202c, as described above with respect to preprocessing module 201 in Figure 2A. Input image 202c is processed by machine learning model 227 to generate corresponding output vectors, for example, representing one or more artifact probabilities 230, 232, and 234. Machine learning model 227 includes a first stage 204 (i.e., a feature extraction stage) that generates feature vector 206c based on image 112c, similar to the first stage 204 that generates feature vector 206a as introduced above with respect to Figure 2A.

[0072] The feature vector 206c generated by the first stage 204 is processed by the second stage 228 (i.e., a classification stage different from the classification stages of the second stage 208 and the second stage 218) to generate one or more artifact probabilities 230, 232, and 234. The second stage 228 may correspond to or include a classification stage, similar to the classification stage of the second stage 208. As described above with reference to Figure 2A, the classification stage takes the feature vector 206c generated by the first stage 204 and may identify, if any, which artifacts the processed image of the focus view type contains.

[0073] The second stage 228 may start with the same architecture, parameters, weights, etc., as the second stages 208 and 218, but may be trained independently and thus evolve to fit the unique features of its input data. For example, the second stage 208 may be trained to generate one or more artifact probabilities 210, 212, and 214 for interferogram image types, while the second stage 228 may be trained to generate one or more artifact probabilities 230, 232, and 234 for focus view type image 112c.

[0074] The images may be of different types, and each of the images 112a, 112b, and 112c supplied to different machine learning models 203, 217, and 227, respectively, may be processed by the same feature extraction stage (i.e., the first stage 204) but by different classification stages (i.e., the second stages 208, 218, and 228, respectively). The corresponding output vectors then indicate the probability that each image 112a, image 112b, and image 112c contains one or more artifacts corresponding to their respective image types. For example, artifact probabilities 210, 212, and 214 indicate the probability that an interferogram view type image 112a contains (each) one or more of glints, bubbles, or suspended objects, while artifact probabilities 220, 222, and 224 indicate the probability that a wide-field type image 112b contains one or more artifacts caused by debris or equipment placement, and artifact probabilities 230, 232, and 234 indicate the probability that a focus view type image 112c contains one or more artifacts caused by hydration concerns (drying or accumulation of tears) or motion.

[0075] In some embodiments, although not shown in Figure 2B, machine learning models 203, 217, and 227 may be combined or otherwise structured to use a single common first stage 204 that generates feature vectors 206a, 206b, or 206c based on images 112a, 112b, and 112c for processing by three second stages 208, 218, and 228. For example, the first stage 204 may receive all input images 202a, 202b, and 202c generated by the preprocessing module 201. The first stage 204 may generate feature vector 206a based on input image 202a, feature vector 206b based on input image 202b, and feature vector 206c based on input image 202c. Therefore, the first stage 204 can supply the corresponding feature vectors 206a, 206b, and 206c to the corresponding second stages 208, 218, and 228, respectively. Such an architecture can reduce overhead and resource consumption at the cost of requiring additional processing time.

[0076] Exemplary architecture of stages for an image classification machine learning model In some embodiments, the feature extraction step of the machine learning model 203 may include a feature-generating deep neural network, such as a convolutional neural network (CNN), a multilayer perceptron neural network (MLP), or a feature extraction portion of a similar neural network. Figure 3A shows an exemplary architecture of a CNN applied as a feature extraction step that generates feature vectors 206 for individual images 112 provided by a camera and processed and / or analyzed by the CNN. A CNN may differ from other neural networks and deep neural networks because it may apply convolution as opposed to matrix multiplication in the layers of the CNN.

[0077] As shown in Figure 3A, the CNN 300 includes a multilayer neural network configured to process an input image. The CNN 300 includes multiple neurons divided into combinations of an input layer 302, convolutional layers 304 (which may include hidden layers), and pooling layers 306. One of the convolutional layers 304 may be considered a hidden layer when the input and output of the convolutional layer 304 are masked. The CNN 300 applied herein as the first stage 204 may have its fully connected and output layers removed and replaced with the layers of the classification stage, as introduced above and described in more detail below. The CNN 300 may also be applied by a controller 106 or a dedicated processor and may represent a neural network used to implement each of the feature extraction stages of a machine learning model, e.g., one or more of the machine learning models 203, 217, and 227 described above with reference to Figures 2A and 2B.

[0078] Specifically, the architecture of CNN 300 includes an input layer 302 with three channels. For a color image, each channel of the input layer 302 corresponds to a different color: red, green, and blue. The input layer 302 may receive inputs containing multiple images, each having a height, width, and number of channels. CNN 300 may be configured to handle any value for any of these embodiments of CNN 300. In the image processing and classification examples described herein, the feature extraction stage may comprise a CNN 300 having an architecture with several input images, each having a size of approximately 480 × 480 pixels and three channels, although processing of different numbers of images, each having a different size and / or number of channels, is also conceivable.

[0079] The CNN 300 architecture further includes several convolutional layers 304. Each convolutional layer 304 may receive inputs corresponding to the number of images, image size (height and width), and the number of channels in each image. A convolutional layer 304 may abstract an image by convolving its input to produce an output, which is passed to a subsequent layer (e.g., one of another convolutional layer 304 or a pooling layer 306). A convolutional layer 304 may apply a convolutional filter to its input. The filter may have a specific size that is applied horizontally and / or vertically along the image being processed by a specific stride that produces output values ​​for the portion of the image covered by the filter. A controller 106 or dedicated processor may apply the filter to each input image by the corresponding stride to produce an output that is passed to a subsequent layer. In some embodiments, the convolutional filter has a depth corresponding to the depth of the number of channels in the input layer 302.

[0080] As shown in Figure 3A, each convolutional layer 304 is followed by a pooling layer 306. The pooling layers 306 can streamline processing by the controller 106 or dedicated processor applying the first stage 204 of the machine learning model 203. Specifically, each pooling layer 306 can reduce the dimensionality of the output generated by the preceding convolutional layer 304. Effectively, the pooling layer 306 can reduce the number of outputs generated by the previous convolutional layer 304. The pooling layer 306 can apply one or more of several functions to pool the outputs from the preceding convolutional layer 304. For example, the processing component that performs processing on the machine learning model 203 can apply one or more of the following to the pooling layer 306: max pooling (taking the maximum number of clusters of the output portion from the preceding convolutional layer 304), average pooling (taking the average number of clusters of the output portion from the preceding convolutional layer 304), or another pooling calculation. The results from the pooling layer 306 can be provided to a subsequent convolutional layer 304 or mean pooling layer to generate feature vectors, which can then be provided to subsequent stages of the machine learning model 203.

[0081] The feature extraction stage may include any number of convolutional layers 304 and pooling layers 306, depending on the process being performed. In some cases, the applicable CNN is a VGG16 CNN. The VGG16 CNN can utilize a combination of convolutional and pooling layers in the configuration shown below with respect to Figure 3B.

[0082] Figure 3B shows a representative diagram of the CNN architecture 320 applied to the feature extraction stage of a machine learning model, such as machine learning models 203, 217, and / or 227 in Figures 2A and 2B. In some embodiments, the architecture 320 applied by the controller 106 or a dedicated processor may represent a neural network used to implement the feature extraction stage of a machine learning model.

[0083] As introduced above, the feature extraction stage may include the feature extraction stage of a VGG16 CNN, as shown in Figure 3B. In the VGG16 CNN architecture 320 shown in Figure 3B, the architecture 320 includes an input layer 302 and five groups 322 of convolutional layers 304 and pooling layers 306, each group including several convolutional layers 304 and one pooling layer 306. The first group 322a has a first convolutional layer 304a, a second convolutional layer 304b, and a first pooling layer 306a. The second group 322b has a third convolutional layer 304c, a fourth convolutional layer 304d, and a second pooling layer 306b. The third group 322c has a fifth convolutional layer 304e, a sixth convolutional layer 304f, a seventh convolutional layer 304g, and a third pooling layer 306c. The fourth group 322d has an eighth convolutional layer 304h, a ninth convolutional layer 304i, a tenth convolutional layer 304j, and a fourth pooling layer 306d. The fifth group 322e has an eleventh convolutional layer 304k, a twelfth convolutional layer 304l, a thirteenth convolutional layer 304m, and a fifth pooling layer 306e. Following the fifth pooling layer 306e, architecture 320 may include a maximum pooling layer (not shown) that generates feature vectors of the image processed by the VGG16 CNN.

[0084] While architecture 320 represents the VGG16 architecture, it will be understood that the architecture applied to the feature extraction stage may include any combination of input layers 302, convolutional layers 304, pooling layers 306, and / or additional layers as needed to efficiently and accurately generate feature vectors of the input image processed by architecture 320. These layers may be arranged in various configurations, numbers, and / or combinations thereof, or according to different architectures of different CNNs or deep neural networks (DNNs).

[0085] As introduced above, the CNN used in the feature extraction stage of a machine learning model does not have to include fully connected layers. Instead, machine learning model 203 includes fully connected layers in the classification model described later with respect to Figure 3C (i.e., the second stage 208).

[0086] Figure 3C shows an exemplary neural network architecture 350 of the classification model (i.e., second stage 208) of the machine learning model 203 of system 100 in Figure 1, which generates an output vector based on the feature vector generated by the first stage 204 for each individual image of the captured digital image. The neural network architecture 350 represents a multilayer deep neural network according to an exemplary embodiment. In some embodiments, the neural network architecture 350 applied by a processing component (i.e., controller 106 or dedicated processor) may represent a neural network used to implement one or more of the second stages 208, second stage 218, or second stage 228 (i.e., one of the classification stages) of one or more of the machine learning models 203, 217, or 227 described above with reference to Figures 2A and 2B.

[0087] The neural network architecture 350 may process input data 352 (corresponding to feature vectors output by the feature extraction stage) using an input layer 354. The input data 352 may correspond to feature vectors output by the first stage 204. The input layer 354 includes multiple neurons as shown in the figure. The neurons may individually adjust the input data 352 by scaling and / or range limiting, etc. Each neuron in the input layer 354 produces an output that is fed into the input of a subsequent hidden layer 356. Each hidden layer 356 includes multiple neurons that process the output from the previous layer (e.g., either the input layer 354 or another hidden layer 356). In some examples, each neuron in the hidden layer 356 produces an output, which is then propagated through one or more additional hidden layers 356. The neural network architecture 350 may include any number of hidden layers 356. The final hidden layer 356 may include multiple neurons that process the output from the previous hidden layer 356 to produce an output that is fed into the output layer 360. The output layer 360 contains one or more neurons that process the output from the hidden layer 356. It should be understood that the neural network architecture 350 is representative only, and other architectures are possible, such as architectures containing a different number of hidden layers 356 without one or more input layers 354 or output layers 360 containing iterative layers.

[0088] In some examples, each neuron in the various layers of the neural network architecture 350 takes a combination of inputs (e.g., a weighted sum of a trainable weight matrix W) and adds an arbitrary trainable bias b. In some examples, a particular neuron, for example, a neuron in the output layer 360, may have an activation function f. The activation function may generally be a nonlinear activation function such as a sigmoid activation function. However, other activation functions are also possible, such as activation functions with upper and / or lower bounds, log-sigmoid functions, hyperbolic tangent functions, and / or normalized linear unit functions. In some examples, each neuron in the output layer 360 may have the same or different activation function as one or more other neurons in the output layer 360.

[0089] In some embodiments, the number of neurons in the input layer of the classification stage is equal to the number of elements in the feature vector generated by the feature extraction stage.

[0090] The input layer of the classification stage applies trained weights to the received feature vectors and may pass the generated results to the first of several hidden layers. The first hidden layer may contain twice as many neurons as the input layer, with each subsequent hidden layer having half the number of neurons as the previous hidden layer. The neurons in the hidden layers may contain a normalized linear unit activation function, or one or more other activation functions. The output layer of the classification stage may contain a number of neurons equal to the number of artifact types for the type of image being processed, and may have a sigmoid activation function.

[0091] Therefore, in one example, the input layer 354 has a number of neurons equal to the length of the feature vector 206, or 512 neurons for a 512-element feature vector generated by the VGG16 CNN introduced above. After applying the trained weights, the input layer 354 produces an output to a first hidden layer 356 which may have 1024 neurons. Each subsequent hidden layer 356, having neurons with a RELU activation function, has three output neurons, one each for interferometer-type image artifacts (i.e., one neuron for each of glint, floating, and bubble artifacts), and has half the neurons of the preceding hidden layer 356 up to output layer 360, which produces artifact probabilities 362. Artifact probabilities 210, 212, and 214 (for example, for interferogram view-type image 112a) may provide probabilities that image 112 contains each of the corresponding artifact types, as previously stated.

[0092] In some embodiments, systems using machine learning models (and similar machine learning models) offer various improvements in accurately identifying whether an image contains one or more artifacts. For example, the system can accurately identify training images containing one or more artifacts approximately 97% of the time, and can accurately identify whether inspection images contain one or more artifacts approximately 91% of the time, representing an improvement over existing techniques. More specifically, the system using machine learning models accurately identified training images with glint artifacts 99% of the time, and accurately identified whether inspection images contained glint artifacts 91% of the time. Furthermore, the system using machine learning models accurately identified training images with floating object artifacts 97% of the time, and accurately identified whether inspection images contained floating object artifacts 95% of the time. Moreover, the system using machine learning models accurately identified training images with bubble artifacts 97% of the time, and accurately identified whether inspection images contained bubble artifacts 97% of the time. Further training (as described later) can improve the system's artifact detection capabilities beyond existing techniques.

[0093] Training and improving machine learning models In some examples, the machine learning model 203 (and the various stages described above) may be trained using one or more learning methods. The machine learning model 203 may be trained using a set of images labeled for containing one or more artifacts. The images may be images taken from various previous procedures or images of the eye that are not from other procedures. In some cases, the images may be Talbot-Moiré interferometer images, and the dataset may be randomly divided into training, validation, and inspection subsets of images, but various other types of images may be classified using the machine learning model 203. Each image in the set may have been manually reviewed and labeled for artifacts contained therein. For example, each image may be labeled to indicate whether the image contains one or more bubble regions, one or more floating object regions, one or more glint regions, one or more artifacts, etc.

[0094] Images labeled as containing one or more artifacts may include additional labeling information that includes the one or more artifacts the image contains. For example, an image of an eye containing bubbles and debris may have a label indicating that the image contains artifacts and that the artifacts are bubbles and debris. In some cases, labeled images (and general datasets) may include location information about where one or more artifacts are located, and this location information may be associated with the type of artifacts contained. For example, if an image is labeled as containing bubbles and debris, the image may be labeled to indicate where the bubbles and debris are located, so that each location is identified by the type of artifact that contains them.

[0095] In some embodiments, the feature extraction stage is set to weights previously trained using a dataset of images, and only the classification stage needs to be optimized. In such embodiments, the feature vector for each image in the dataset is pre-calculated by the feature extraction stage. These feature vectors can then be formed into a feature matrix by stacking them so that the feature vectors have a width equal to the size of the feature vector and a height equal to the number of images in the dataset, and are processed by the feature extraction stage. The feature matrix is ​​stored in memory. Such storage can improve the speed of training the classification stage. In some cases, the time to train the classification stage can be improved by as much as 100 or 1000 times compared to calculating the image feature vector for each training image in the dataset when training the classification stage.

[0096] Such efficiency improvements are particularly advantageous during hyperparameter optimization of the classification stage, where the architecture of the machine learning model, including the feature extraction and classification stages, is iteratively adjusted and the classification stage is trained based on a memorized feature matrix. Hyperparameter optimization can correspond to the selection of an architecture for the classification stage to improve its classification ability. Such selection (and therefore hyperparameter optimization) may be made by the operator or user of the machine learning model or the system using the machine learning model.

[0097] In some cases, optimizing hyperparameters involves applying an algorithm to select candidate values ​​for hyperparameters from a list of available distributions or available values ​​for each hyperparameter. Other methods for selecting candidate values ​​are understood to be available for use in selecting candidate values. A machine learning model with an architecture generated based on the selected hyperparameters can then be trained and evaluated using at least a portion of the labeled image dataset (e.g., 5x cross-validation of the training set of the dataset). If the selected value for any hyperparameter lies at the edge of the available range for that hyperparameter, the range of that hyperparameter may be extended, and hyperparameter optimization should be repeated until the hyperparameter is no longer at the edge of its corresponding range and the performance of the classification model satisfies the desired threshold and parameter values ​​identified by examining the machine learning model with a set of training images. A selected list of preferred hyperparameters and their corresponding value ranges includes: • Learning rate for the classification stage fitting algorithm ○Values: [0.001,0.0003,0.0001,0.00003,0.00001,0.000003] • Number of epochs to train the classification stage ○Value: Uniform (500, 10000) • Learning Optimizer ○ Value: [Adam,SDG,RMSprop,Adadelta,Adagrad,Adamax,Nadam] Batch size ○Values: [8, 16, 32, 64, 128] • Number of hidden layers ○Values: [1,2,3,4,5,6,7,8,9] • Size of each hiding place ○Value: Uniform (128,2048) • Regularization weights and strategies (e.g., L1, L2, and / or dropout) ○L1 value: [0.0,0.01,0.003,0.001,0.0003,0.0001,0.00003,0.000001,0.0000003] ○L2 value: [0.0,0.01,0.003,0.001,0.0003,0.0001,0.00003,0.000001,0.0000003] ○Dropout % value: [0,0.001,0.003,0.01,0.03,0.1,0.3,1.0,3.0,10.0,20.0,30.0,40.0,50.0] Loss function ○Value: [Binary_CrossEntropy] • Activation function for the layer ○Value: [RELU].

[0098] In some cases, further optimization of the machine learning model may include retraining the feature extraction stage. Such optimization may include retraining the weights of the feature extraction stage and / or hyperparameter optimization of the architecture of the feature extraction stage. In some cases, an additional output layer may be added to the classification stage. The additional output layer may provide applying regression to score image quality.

[0099] In some embodiments, training a machine learning model involves implementing the machine learning model with pre-trained weights for the VGG16 CNN feature extraction stage. Thus, the VGG16 CNN weights can be fully specified, leaving only the weights for the classification stage to be determined by training. Training may involve processing labeled images with three channels and a size of 480x480 pixels from a dataset or repository with the VGG16 CNN stage. The VGG16 CNN stage may output a feature vector (e.g., feature vector 206) having a length of 512 elements or samples. The feature vector may represent a set of image features suitable for the classification task performed by the classification stage. The set of image features in the feature vector may include features from a wide range of image types (e.g., interferogram type images, wide field type images, and focus view type images). Processing the feature vector from the VGG16 CNN stage with the fully connected classification stage generates an output vector representing the probability of the presence of each artifact in the image processed by the machine learning model.

[0100] Exemplary images classified according to the aspects of this specification Figures 4A to 4G show exemplary images that may be present in the image dataset and / or captured by the camera of system 100 in Figure 1.

[0101] Figure 4A shows Image 400, an example of a Talbot-Moiré image of a patient's eye 110 that is free of artifacts. As shown, Image 400 does not contain any very bright areas of light or reflection, or any objects or areas of cloudiness or abstraction within the patient's eye 110.

[0102] Figure 4B shows Image 410, an example of a Talbot-Moiré image of a patient's eye 110 containing a glint artifact 412. Specifically, the glint artifact 412, caused by excessive reflection of the light source from a portion of the patient's eye 112, is shown near the center of Image 410.

[0103] In the exemplary use case introduced above, the user interface 108 may present an image 410 to the operator. More specifically, if the image 410 contains a glint artifact 412, the user interface 108 may optionally or selectively display the image 410 with an identifier 414 to specifically identify the location of the glint artifact 412. In some cases, the identifier 414 may represent any shape or object used to draw the attention of the observer or system to a particular location in the image 410. In some cases, the user interface 108 may also include a message to the operator indicating that the image is believed to contain a glint artifact 412 at the location identified by the identifier 414. In some cases, the user interface 108 displays the identifier 414 only if the probability that the image 410 contains a glint artifact 412 exceeds a specified threshold (i.e., the quality value of the image 410 falls below a specified threshold).

[0104] Figure 4C shows Image 420, an example of a Talbot-Moiré image of a patient's eye 110 containing several bubble artifacts 422. Specifically, the bubble artifacts 422 are shown at multiple locations around the patient's eye in Image 420.

[0105] In the exemplary use case introduced above, the user interface 108 may present an image 420 to the operator. More specifically, if the image 420 contains bubble artifacts 422, the user interface 108 may optionally or selectively display the image 420 having identifiers 424 for specifically identifying one or more locations of the bubble artifacts 422. In some cases, the identifier 424 may represent any shape or object used to draw the attention of the observer or system to a particular location in the image 420. In some cases, the user interface 108 may also include a message to the operator indicating that the image is thought to contain bubble artifacts 422 at the locations identified by the identifier 424. In some cases, the user interface 108 displays the identifier 424 only if the probability that the image 420 contains bubble artifacts 422 exceeds a specified threshold (i.e., the quality value of the image 420 is below a specified threshold).

[0106] Figure 4D shows Image 430, an example of a Talbot-Moiré image of a patient's eye 110 containing a floating artifact 432. Specifically, the floating artifact 432 is shown across a region of the patient's eye in Image 430.

[0107] In the exemplary use case introduced above, the user interface 108 may present an image 430 to the operator. More specifically, if the image 430 contains a floating artifact 432, the user interface 108 may optionally or selectively display the image 430 with an identifier 434 to specifically identify the location of the floating artifact 432. In some cases, the identifier 434 may represent any shape or object used to draw the attention of the observer or system to a particular location in the image 430. In some cases, the user interface 108 may also include a message to the operator indicating that the image is believed to contain a floating artifact 432 at the location identified by the identifier 434. In some cases, the user interface 108 may display the identifier 434 and the message only if the probability that the image 430 contains a floating artifact 432 exceeds a specified threshold (i.e., the quality value of the image 430 is below a specified threshold).

[0108] Figure 4E shows Image 440, an example of a Talbot-Moiré image of a patient's eye 110 that includes both glint artifacts 442 and floating artifacts 443. Specifically, the glint artifact 442 is shown near the center of the patient's eye, and the floating artifact 443 is shown across a region of the patient's eye in Image 440.

[0109] In the exemplary use case introduced above, the user interface 108 may present an image 440 to the operator. More specifically, if the image 440 includes a glint artifact 442 and a floating object artifact 443, the user interface 108 may or may selectively display the image 440 using an identifier 444 for specifically identifying the location of the glint artifact 442 and an identifier 446 for identifying the location or region of the floating object artifact 443.

[0110] In some cases, identifiers 444 and 446 may represent any shape or object used to draw the observer's or system's attention to a specific location in image 440. In some cases, the user interface 108 may also include a message to the operator indicating that the image is thought to contain glint artifacts 442 and floating artifacts 443 at each location identified by identifiers 444 and 446. In some cases, the user interface 108 may display identifiers 444 and 446 only if the probability that image 440 contains glint artifacts 442 and floating artifacts 443 exceeds a corresponding specified threshold (i.e., the quality value of image 440 falls below a specified threshold).

[0111] Figure 4F shows Image 450, an example of a Talbot-Moiré image of a patient's eye 110 that includes both glint artifacts 452 and bubble artifacts 453. Specifically, the glint artifact 452 is shown near the center of the patient's eye, and the bubble artifact 453 is shown along the right edge of the patient's eye in Image 450.

[0112] In the exemplary use case introduced above, the user interface 108 may present the image 450 to the operator. More specifically, if the image 450 contains glint artifacts 452 and bubble artifacts 453, the user interface 108 may optionally or selectively display the image 450 using identifier 454 to specifically identify the location of the glint artifacts 452 and identifier 456 to identify the location or region of the bubble artifacts 453. In some cases, identifiers 454 and 456 may represent any shape or object used to draw the observer's or system's attention to a particular location in the image 450. In some cases, the user interface 108 may also include a message to the operator indicating that the image is thought to contain glint artifacts 452 and bubble artifacts 453 at the respective locations identified by identifiers 454 and 456. In some cases, the user interface 108 displays identifiers 454 and 456 only if the probability that image 450 contains glint artifacts 452 and bubble artifacts 453 exceeds a corresponding specified threshold (i.e., the quality value of image 450 is below the specified threshold).

[0113] Figure 4G shows Image 460, an example of a Talbot-Moiré image of a patient's eye 110 that includes both multiple bubble artifacts 462 and floating object artifacts 463. Specifically, the bubble artifacts 462 are shown along the upper and right edges of the patient's eye, and the floating object artifacts 463 are shown across a region of the patient's eye in Image 460.

[0114] In the exemplary use case introduced above, the user interface 108 may present an image 460 to the operator. More specifically, if the image 460 includes a bubble artifact 462 and a floating object artifact 463, the user interface 108 may optionally or selectively display the image 460 using an identifier 464 to specifically identify the location of the bubble artifact 462 and an identifier 466 to identify the location or region of the floating object artifact 463. In some cases, identifiers 464 and 466 may represent any shape or object used to draw the observer's or system's attention to a particular location in the image 460. In some cases, the user interface 108 may also include a message to the operator indicating that the image is thought to include a bubble artifact 462 and a floating object artifact 463 at the respective locations identified by identifiers 464 and 466. In some cases, the user interface 108 displays identifiers 464 and 466 only if the probability that image 460 contains bubble artifacts 462 and floating object artifacts 463 exceeds a corresponding specified threshold (i.e., the quality value of image 460 is below the specified threshold).

[0115] As introduced above, images of a patient's eye free of artifacts may be processed to generate measurements or corresponding information for use during the procedure. In some cases, the system may determine that one or more images containing artifacts can still be processed for measurements for use during the procedure. For example, if the artifacts in the image are sufficiently small or located in a specific area where their presence has minimal impact on the measurement, the operator and / or system may determine that the image can proceed to processing for measurement based on it.

[0116] In some cases, various factors may be used to determine whether an image 112 containing one or more artifacts can proceed to measurement determination. The operator or system 100 may decide that an image 112 containing one or more artifacts should proceed to measurement generation based on one or more of the size of the artifacts in the image 112, the location of the artifacts in the image 112, and the type of artifacts in the image 112. Specifically, if the decision is made based on the probability that the image 112 contains artifacts, as introduced above with respect to quality bar graphs and / or quality values, the decision may further be made based on an analysis of whether the location, size, and / or type of the artifacts adversely affect the measurements generated based on the image 112. For example, if an image 112 contains a single bubble artifact along the edge of the patient's eye 110 and has a size that covers less than the threshold of the patient's eye 110, the system 100 or operator may decide that the image 112 containing the bubble artifact can still be used to generate measurement data. Alternatively, if image 112 contains multiple bubble artifacts near the center of the patient's eye 110 and has a composite size that includes a size exceeding the threshold of the patient's eye 110, the system 100 or operator may determine that image 112 containing the bubble artifacts cannot be used to generate measurement data.

[0117] The decision to use images containing one or more artifacts to generate measurement data may be image-specific based on a variety of factors. These factors may include the number of available artifact-free images of the patient's eye, the type of artifact in images containing at least one artifact, the size of the artifact in images containing at least one artifact, and the location of the artifact in images containing at least one artifact.

[0118] An exemplary method for real-time processing of image data to identify artifacts within it. Figure 5 shows an exemplary method 500 for identifying a digital image containing one or more artifacts. For example, the controller 106 and / or system 100 in Figure 1 may be configured to perform method 500 based on, for example, the machine learning models in Figures 2A and 2B.

[0119] Method 500 begins in block 502, and in block 504, begins by acquiring an image (e.g., an image 112 of the patient's eye 110) based on the operation of an imaging device (e.g., an aberration meter 104). Optionally, one or more cameras of System 100 may acquire the image. Optionally, the image may include a color image (e.g., with three data channels, one each for the red, green, and blue layers) or a grayscale image. Furthermore, the image may have a first size measured in pixels (e.g., 640 × 480 pixels).

[0120] Method 500 continues in block 506 by acquiring an image from an image acquisition element. As described above, Method 500 can acquire an image from a camera, which corresponds to an image acquisition element.

[0121] In some embodiments, acquiring an image includes receiving an image 112 from a camera or aberration meter 104, as shown in Figure 1.

[0122] Next, method 500 proceeds to block 508, preprocessing the image in preparation for classification by a machine learning model. In some embodiments, the machine learning model is a two-stage machine learning model, as described above with reference to Figures 2A to 3C. In some embodiments, the preprocessing in block 508 is optional. For example, as described herein, preprocessing may only be required if the image needs to be processed in order to present the image as a suitable input to the machine learning model. Preprocessing may include adjusting one or more of the image size, the format of pixels in the image, the number of channels in the image, and / or selecting the region of interest of the image.

[0123] In some cases, image preprocessing involves preprocessing image 112 by preprocessing module 201 to generate input image 202 for input to machine learning model 203, as shown in Figures 2A and 2B.

[0124] Next, Method 500 proceeds to block 510, which generates feature vectors based on the image preprocessed by the feature extraction stage of a two-stage machine learning model. In some embodiments, the feature extraction stage is the first stage of the machine learning model and includes the VGG16 CNN introduced above. Alternatively, the feature extraction stage may include any other neural network, such as VGG19, ResNet50, Inception V3, Xception, etc., with corresponding features or stages. The feature vectors generated by Method 500 may include an output vector.

[0125] In some cases, block 510 corresponds to processing the input image 202, which has been preprocessed by the first stage 204 of the machine learning model 203, to generate feature vectors 206, as shown in Figures 2A and 2B.

[0126] Next, method 500 proceeds to block 512, where a classification stage of a two-stage machine learning model generates an output vector (e.g., one of artifact probabilities 210, 212, or 214) based on the feature vector. In some cases, the output vector generated in block 512 includes combinations of output vectors for all artifact types configured to be identified by the machine learning model. This classification stage may include one or more fully connected layers and an output layer having neurons to which an activation function is applied to generate the output vector. In some embodiments, the activation function of the output layer neurons may include a sigmoid or logistic activation function. In other examples, the activation function may include other functions such as a softmax activation function or another activation function that can provide a probabilistic output.

[0127] In some cases, block 512 corresponds to processing the feature vector 206 by the second stage 208 of the machine learning model 203, as shown in Figures 2A and 2B, to generate artifact probabilities 210, 212, and 214. For example, the generated output vector may include each of the artifact probabilities 210, 212, and 214, or any combination of one or more of the artifact probabilities 210, 212, and 214. In some embodiments, the classification stage corresponds to the second stage 208 of the machine learning model 203.

[0128] Method 500 then proceeds to block 514, where the image quality of the image 112 acquired based on the output vector for display to the operator is determined. In some embodiments, the output vector provides probability information regarding the probability that the image contains one or more artifacts. In some cases, the image quality can be determined based on this probability relating to Equation 1 above. Thus, the image quality may indicate the probability that the image contains artifacts, which, if present, may interfere with the refractive measurements and other data of the patient's eye. Method 500 then terminates in block 516.

[0129] For example, output vectors representing artifact probabilities 210, 212, and 214 may indicate that image 112 has a probability of 0.75 regarding the presence of bubble artifacts, 0.01 regarding the presence of glint artifacts, and 0.24 regarding the presence of floating object artifacts.

[0130] As introduced above, method 500 can generally be performed repeatedly or iteratively for each image generated by the aberration meter 104 or the camera of system 100.

[0131] In particular, Figure 5 is merely illustrative, and other methods having additional, different, and / or fewer steps (or blocks) are possible in accordance with the various embodiments described herein.

[0132] An exemplary method for training a machine learning model to process image data and identify artifacts within it. Figure 6 shows an exemplary method 600 for training a machine learning model to identify a digital image containing one or more artifacts. Training of the machine learning model may be completed by the controller 106 and / or system 100 in Figure 1, or by one or more components or systems outside of system 100. Method 600 may be used, for example, to train the machine learning models in Figures 2A and 2B. In some embodiments, Method 600 trains only the second stages 208, 218, and / or 228 (i.e., the classification stage), and the first stage 204 (i.e., the feature extraction stage) is not trained.

[0133] Method 600 begins in block 602 and in block 604 with acquiring images to be used to train a machine learning model. In some cases, images are acquired in real time from an image acquisition device (e.g., one or more of the aberration meter 104 and / or cameras of system 100). In some cases, images are acquired from a data store, e.g., a database of images for use in training a machine learning model. In some cases, the acquired images are labeled with respect to whether or not they contain artifacts, and if so, what kind of artifacts they contain.

[0134] Method 600 continues in block 606 by generating feature vectors for each image by a feature extraction stage of a two-stage machine learning model. As introduced above, the feature extraction stage generates feature vectors based on applying a feature extraction stage (e.g., the first stage 204 of machine learning model 203) to the image, for example, a feature extraction stage having a VGG16 CNN architecture. In some cases, other feature extraction stages can be implemented for the feature extraction stage of the machine learning model (e.g., VGG19). In some embodiments, the feature vectors for each image have the same dimension (e.g., 1 × 5¹² element or sample dimension).

[0135] Method 600, in block 608, proceeds to generate a feature matrix based on stacking the generated feature vectors. Stacking the generated feature vectors may simply involve creating a matrix from several feature vectors by stacking them on top of each other to create a feature matrix. The feature matrix has dimensions of the length of the feature vectors and the height of the several feature vectors stacked on top of each other. In the exemplary use case herein, the feature matrix may be generated by stacking the feature vectors 206 generated by the first stage 204 for each image processed by the first stage 204 and the machine learning model 203 (e.g., all images in the dataset).

[0136] Method 600 is followed by a block 610 in which a classification stage is trained based on a feature matrix. In some cases, the classification stage (i.e., a second stage 208) includes using a feature matrix generated based on acquired training images to train the classification stage to appropriately identify artifacts in images processed by the classification stage. Appropriately identifying artifacts may include a second stage 208 that produces an output that identifies a high probability of artifacts if the image contains artifacts, and a low probability of artifacts if the image does not contain artifacts. In some embodiments, the activation function of the second stage 208 may be modified as part of training the classification stage. In some embodiments, the trained second stage 208 may then be used to determine whether an image received in real time from a diagnostic imaging device (e.g., aberration meter 104) used during one or more procedures contains one or more artifacts. Method 600 then terminates in a block 612.

[0137] As introduced above, method 600 can generally be performed repeatedly or iteratively.

[0138] In particular, Figure 6 is merely illustrative, and other methods having additional, different, and / or fewer steps (or blocks) are possible in accordance with the various embodiments described herein.

[0139] Exemplary Processing System Figure 7 is a diagram of an embodiment of a processing system or device. According to some embodiments, the processing system in Figure 7 represents a computing system that may be included in one or more intraoperative aberration measurement systems that implement the machine learning model processing described herein, with reference to an aberration meter 104, a controller 106, and / or a user interface 108, etc. Specifically, the processing system in Figure 7 may implement one or more of the machine learning models 203, 217, and 227 according to data flows 200, 250a, 250b, and 250c, which are introduced and described herein for identifying artifacts in image data of a patient's eye, as shown in Figures 4A to 4G, etc.

[0140] Figure 7 shows a computing system 700 in which the components of system 700 communicate electrically with one another. System 700 includes a processor 710 and a system bus 705 that connects various components. For example, the bus 705 connects the processor 710 to various memory components such as read-only memory (ROM) 720 and / or random access memory (RAM) 725 (e.g., PROM, EPROM, FLASH®-EPROM, and / or any other memory chip or cartridge). System 700 may further include a high-speed memory cache 712 that is connected to, adjacent to, or integrated as part of the processor 710 (directly or indirectly). System 700 may access data stored in the ROM 720, RAM 725, and / or one or more storage devices 730 via the cache 712 for high-speed access by the processor 710. In some examples, the cache 712 may provide performance improvements that avoid delays by the processor 710 when accessing data from one or more storage devices 730 previously stored in the ROM 720, RAM 725, and / or cache 712. In some examples, one or more storage devices 730 store one or more software modules (e.g., software modules 732, 734, 736, 738, and / or 739). Software modules 732, 734, 736, 738, and / or 739 may control and / or be configured to control the processor 710 to perform various operations, such as the processes of methods 500 and / or 600. In some embodiments, one or more of the software modules 732, 734, 736, 738, and / or 739 include details of the machine learning models 203, 217, and / or 227 described herein. Some examples may include additional or fewer software modules and / or code to program the processor 710 to perform other functions.

[0141] In some embodiments, the software module 732 includes instructions for programming the processor 710 to preprocess an image. The code for preprocessing the image may cause the processor 710 (or any other component for the computing system 700 or any other computing system) to preprocess the image 112 generated by the aberration meter 104 and / or the camera of system 100. The processor 710 may perform one or more of the following actions: adjust the size of the image 112, adjust the pixel format of the image 112, identify a region of interest within the image 112, or change the number of channels in the image 112, thereby generating the input image 202.

[0142] In some embodiments, the software module 734 includes instructions for programming the processor 710 to generate feature vectors. The code for generating feature vectors may cause the processor 710 (or any other component for the computing system 700 or any other computing system) to apply a first stage 204 of the machine learning model 203 (or a corresponding machine learning model 203) to process and analyze the input image 202 to generate feature vectors. The first stage 204 of the machine learning model 203 may include any feature-generating neural network component, such as a VGG16 CNN.

[0143] In some embodiments, the software module 736 includes instructions for programming the processor 710 to generate an output vector. The code for generating the output vector may cause the processor 710 (or any other component for the computing system 700 or any other computing system) to apply a second stage 208 of the machine learning model 203 (or the corresponding machine learning model 203) to process and analyze the feature vectors to generate the output vector. The second stage 208 of the machine learning model 203 may include any classification neural network component, such as a fully connected layer having an output layer that uses a sigmoid activation function to generate an output vector that identifies the probability that the processed image contains a corresponding artifact.

[0144] In some embodiments, the software module 738 includes instructions for programming the processor 710 to determine image quality based on the generated output vector. The code for determining image quality may cause the processor 710 (or any other component for the computing system 700 or any other computing system) to analyze the probabilities identified in the output vector and determine the corresponding image quality based on the probabilities in the output vector.

[0145] In some embodiments, the software module 739 includes instructions for programming the processor 710 to train the machine learning model 203. The code for training the machine learning model may cause the processor 710 (or any other component for the computing system 700 or any other computing system) to train one or more of the first stages 204 or second stages 208 of the machine learning model 203 (or the corresponding machine learning model 203). In some cases, training the stages may include using a dataset of labeled images to identify and train the parameters, weights, and / or biases of the corresponding stages based on the labeled images, enabling the stages to classify images according to the labels.

[0146] Although System 700 is represented by only one processor 710, it is understood that the processor 710 may represent one or more central processing units (CPUs), multicore processors, microprocessors, microcontrollers, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), and / or tensor processing units (TPUs), etc. In some examples, System 700 may be implemented as a standalone subsystem and / or as a board added to a computing device, or as a virtual machine, or as a cloud-based processing machine.

[0147] To enable user interaction with the system 700, the system 700 includes one or more communication interfaces 740 and / or one or more input / output (I / O) devices 745. In some examples, one or more communication interfaces 740 may include one or more network interfaces and / or network interface cards, etc., to provide communication in accordance with one or more network and / or communication bus standards. In some examples, one or more communication interfaces 740 may include interfaces for communicating with the system 700 over a network. In some examples, one or more I / O devices 745 may include one or more user interface devices (e.g., keyboard, pointing / selection device (e.g., mouse, touchpad, scroll wheel, trackball, and / or touchscreen, etc.), audio device (e.g., microphone and / or speaker), sensor, actuator, and / or display device, etc.).

[0148] Each of the one or more storage devices 730 may include non-temporary and non-volatile storage devices, such as those provided by hard disks, optical media, and / or solid-state drives. In some examples, each of the one or more storage devices 730 may be located in the same location as the system 700 (e.g., local storage devices) and / or may be located away from the system 700 (e.g., cloud storage devices).

[0149] According to some embodiments, the system 700 may provide a graphical user interface (GUI) suitable for assisting a user (e.g., a surgeon and / or other medical professional or operator) in the execution of the processes of methods 500 and / or 600. For example, the GUI may provide the user interface 108 shown in Figure 1. The GUI may include depictions of images captured by the system 100, quality bar graphs, and other information generated by machine learning models 203, 217, and / or 227, instructions or recommendations to the operator regarding whether to retain or exclude images for processing, identification of artifacts in images of the patient's eye (e.g., as shown in Figures 4A to 4G), requests for operator input, and / or messages to the operator. In some examples, the GUI may display true-color and / or grayscale images of the patient's eye, etc.

[0150] Exemplary display of classified images Figures 8A to 8O are display concepts for providing a user with a digital image of a patient's eye, including details of image quality and / or any detected artifacts, via a graphical user interface, according to several embodiments. These display concepts illustrate various methods for presenting the operator with information about the patient's eye image, namely whether the image contains artifacts and information about the image quality. Furthermore, the display concepts may also identify the location of any included artifacts, as well as image metrics such as whether the image is a "good" or "bad" image and / or image quality. Note that Figures 8A to 8O are merely examples, and other implementations are possible.

[0151] Figure 8A shows a basic representation of a “good” image 820 and a representational concept 800 of a label 822 that identifies the image quality of the image 820, which here includes the text “Good Image”. In certain embodiments, a “good” image includes an image of a patient’s eye that is free of artifacts or otherwise can be processed for one or more measurements of the patient’s eye. The “good image” classification is for the image 820 as a whole and indicates the absence of significant artifacts. The label 822 may include any text label that uses one or more words to describe the image quality of the image 820, which is evaluated based on whether the imaging system can process the corresponding image for one or more measurements of the patient’s eye. Thus, if the image 820 is a good image, the label 822 may identify the image as such a “good image”. In some embodiments, an image of a patient’s eye containing one or more artifacts may be labeled as a “good image” if the artifacts are below a threshold size or, in particular, in a specific area of ​​the patient’s eye, the one or more artifacts do not impair the ability to process the image for measurements of the patient’s eye.

[0152] Figure 8B shows another representation of the “good” image 820, a label 822 identifying the image quality of the image 820, and a representation concept 801 of a visual or gradient legend or scale (hereinafter referred to as the “visual legend”) 824 indicating the relative image quality of the image 820. The visual legend 824 may include a range such as color, hue, grayscale, and / or pattern using a scheme or scale of image values ​​progressing from a first value (e.g., bottom) to a second value (e.g., top). In some embodiments, the orientation of the visual legend 824 can be adjusted to be horizontal, diagonal, etc. In some embodiments, the first value of the visual legend 824 indicates a low-quality or poor-quality image. The visual legend 824 may show the progression of image quality of the image 820 from a first value (low-quality image) where image quality is increasing to a second value indicating a high-quality or good-quality image. Visual legend 824 may include an arrow 826 indicating where the image quality of image 820 falls in relation to the image quality of image 820 in visual legend 824. In Figure 8B, arrow 826 indicates that image 820 is a good image if arrow 826 identifies that the image quality is close to the second value in visual legend 824. Label 822 contains the text "Good Image," and good images are classified as described above.

[0153] Figure 8C shows another representation of a “good” image 820, a label 822 identifying the image quality of image 820, and a representation concept 802 of a visual signal 828 indicating the relative image quality of image 820. The visual signal 828 may include different colors, hues, grayscales, and / or patterns to indicate different information about image 820, such as a traffic light. Similar to the visual legend 824, the visual signal 828 may include different indicators for different image quality, including poor or low image quality (left), minimum image quality (center), and high or good image quality (right). As shown in Figure 8C, the representation concept 802 shows the rightmost indicator of the visual signal 828 indicating good image quality for image 820. Furthermore, the label 822 includes the text “Good Image,” and good images are classified as described above.

[0154] Figure 8D shows a display concept 803 of another display of image 820, a visual signal 828, and an image quality indicator 830 that shows the relative image quality of image 820. The visual signal 828 may show the rightmost indicator (which may be colored green, for example) representing good image quality. The image quality indicator 830 may provide an image quality percentage between 0 and 100%. As shown in display concept 803, the image quality indicator 830 may show an image quality of 87% for image 820. Just as the visual legend 824 and visual signal 828 can show different image quality levels, the image quality indicator 830 may have different thresholds for good image quality, minimum image quality, and low image quality. As shown in Figure 8D, the display concept 803 shows that image 820 has an image quality of 87% via the image quality indicator 830. The rightmost indicator of the visual signal 828 indicates that image 820 has good image quality, and good image quality is classified as described above.

[0155] Figure 8E shows another representation of the “good” image 820, quality-related values ​​834 for the sequence of images up to and including image 820, and a conceptual representation 804 of a graph 832 of the quality of the sequence of images up to and including image 820. The quality-related values ​​834 represent the standard deviation (STD) of the quality values ​​22 and the 63% mean quality. The mean quality may include the average of the image quality values ​​830 for the images in the sequence of images up to and including image 820. Furthermore, graph 832 also shows, for example, how at least a subset of the sequence of images are compared to each other with respect to the quality-related values ​​834. In some embodiments, graph 832 is a real-time graph scaled until a predefined number of images are reached, and can be scrolled as additional images are added to the sequence of images.

[0156] Figure 8F shows a conceptual representation 805 of a “poor” image 840, a label 822, and an image quality indicator 830 indicating the relative image quality of image 840. Label 822 indicates that image 840 is a “poor” image and further indicates the reason why image 840 has poor image quality, which in this case is because a bubble artifact was detected in image 840. The “poor image” classification is for image 840 as a whole and indicates the presence of one or more artifacts in image 840. As introduced above, label 822 may include a text label that uses one or more words to describe the image quality of image 840, which is evaluated based on whether the imaging system can process the corresponding image for one or more measurements of the patient’s eye. Thus, if image 840 is a poor image, label 822 may identify image 840 as a “poor image” with details about the reason why image 840 has poor image quality. The image quality indicator 830 in this example provides image quality 32% for image 840.

[0157] Figure 8G shows a display concept 806 of another representation of the “bad” image 840, in which label 822 identifies the image quality of image 840, and visual legend 824 indicates the relative image quality of image 840. Visual legend 824 may include ranges such as color, hue, grayscale, and / or pattern, as introduced above. Visual legend 824 may include an arrow 826 indicating where the image quality of image 840 lies within the visual legend 824. In Figure 8G, arrow 826 identifies the first level of the visual legend 824, thereby indicating that image 840 is a bad image. Label 822 includes the text “bad image” to which the bad image is classified as described above, and additional text identifying the type of artifact identified in image 840.

[0158] Figure 8H shows a display concept 807 of another representation of the “bad” image 840, in which label 822 identifies the image quality of image 840, and visual signal 828 indicates the relative image quality of image 840. Visual signal 828 may include different colors, hues, grayscales, and / or patterns, etc., indicating different information about image 840, as introduced above. As shown in Figure 8H, display concept 807 indicates that the leftmost indicator of visual signal 828 indicates a bad image quality for image 840. Furthermore, label 822 also includes the text “bad image” to which the bad image is classified as described above, and additional text that identifies the type of artifact identified in image 840.

[0159] Figure 8I shows a display concept 808 of another representation of the “poor” image 840, in which the visual signal 828 and image quality indicator 830 indicate the relative image quality of image 840. For example, the visual signal 828 may indicate the leftmost indicator representing the poor image quality of image 840. The image quality indicator 830 provides an image quality percentage of 17% for image 840.

[0160] Figure 8J shows another representation of the “bad” image 840, quality-related values ​​834 for the sequence of images up to and including image 840, and a conceptual representation 809 of a graph 832 of the quality of the sequence of images up to and including image 840. The quality-related values ​​834 represent 22 STD quality values ​​and 14% average quality. Average quality may include the average of the image quality values ​​830 for the images in the sequence of images up to and including image 840. Furthermore, graph 832 also shows how at least a subset of the sequence of images compare to each other, for example, with respect to quality scores and / or average quality 834. In some embodiments, graph 832 is a real-time graph scaled until a predefined number of images are reached, and can be scrolled as additional images are added to the sequence of images.

[0161] Figure 8K shows a representation of the “bad” image 840, a label 822, an image quality indicator 830 showing the relative image quality of image 840, and a representation concept 810 of one or more circles 837 identifying the approximate location of bubble artifacts in image 840. The circles 837 may include one or more colors, hues, grayscales, and / or patterns, etc., to identify the differences between different artifacts and / or the acceptableness of the artifacts. For example, following the above description, the different colors, hues, grayscales, and / or patterns, etc., of the circles 837 may correspond to different ranges of artifacts, from unacceptable or bad artifacts to minimal artifacts and acceptable artifacts. The label 822 indicates that image 840 is a “bad” image and further indicates the reason why image 840 was identified as bad (e.g., the detection of bubble artifacts). The image quality indicator 830 may provide an image quality percentage of 32% for image 840.

[0162] Figure 8L shows a representation concept 811 of one or more circles having a “poor” image 840, a label 822, an image quality indicator 830 showing the relative image quality of image 840, and a transparent fill 838 identifying the approximate location of bubble artifacts in image 840. As introduced above, the circles with the transparent fill 838 may include one or more colors, hues, grayscales, and / or patterns, etc., to identify between different artifacts and / or the acceptableness of artifacts. For example, following the above description, different colors, hues, grayscales, and / or patterns, etc., of the circles with the transparent fill 838 may correspond to different ranges of artifacts, from unacceptable or poor artifacts to minimal artifacts and acceptable artifacts. The label 822 indicates that image 840 has “poor” image quality and further indicates why image 840 is poor (e.g., the detection of bubble artifacts). Image quality indicator 830 may provide a 32% image quality percentage compared to image 840.

[0163] Figure 8M shows a representation concept 812 of another “bad” image 840, a label 822 identifying the image quality of image 840, a visual legend 824 showing the relative image quality of image 840, and individual text descriptions 839 about the called-out artifacts. The visual legend 824 may include ranges such as color, hue, grayscale, and / or pattern, as introduced above. The visual legend 824 may include an arrow 826 indicating where image 840 is located on the visual legend 824. In Figure 8M, the arrow 826 indicates that image 840 is a bad image if the arrow 826 identifies a low value in the visual legend 824. The label 822 includes the text “bad image” and the description “bubble artifact detection,” classifying the bad image as described above and identifying the type of artifact identified in image 840. The individual text descriptions 839 may include various details of the identified artifact, including size, relative location, impact on image quality, and tolerance. As shown in the representation concept 812, individual text descriptions 839 are associated with artifacts identified using leader lines, etc.

[0164] Figure 8N shows a representation concept 813 of another representation of the “bad” image 840, a label 822 identifying the image quality of image 840, a visual legend 824 showing the relative image quality of image 840, and individual text descriptions 839 about the called-out artifacts. The visual legend 824 may include ranges such as color, hue, grayscale, and / or pattern, as introduced above. The visual legend 824 may include an arrow 826 indicating where image 840 is located on the visual legend 824. In Figure 8N, the arrow 826 indicates that image 840 has poor image quality if the arrow 826 identifies a low level on the visual legend 824. The label 822 includes the text “bad image” and the description “bubble artifact detection,” classifying the bad image as described above and identifying the type of artifact identified in image 840. The individual text descriptions 839 may include various details of the identified artifact, including size, relative location, impact on image quality, and tolerance. As shown in the representation concept 813, each text description 839 is associated with an artifact identified using an alphanumeric identifier or the like.

[0165] Figure 8O shows another representation of the “bad” image 840, quality-related values ​​834 for the sequence of images up to and including image 840, a graph 832 of the quality of the sequence of images up to and including image 840, and a representation concept 814 of one or more circles 837 that identify the approximate location of bubble artifacts in image 840. The quality-related values ​​834 represent 22 STD quality values ​​and 14% average quality. Average quality may include the average of the image quality 830 values ​​for the images in the sequence of images up to and including image 840. Furthermore, graph 832 shows, for example, how at least a subset of the sequence of images are compared to each other with respect to the quality-related values ​​834. In some embodiments, graph 832 is a real-time graph scaled until a predefined number of images are reached, and can scroll as additional images are added to the sequence of images. Furthermore, circles 837 may include one or more colors, hues, grayscales, and / or patterns, etc., to identify the tolerance between and / or artifacts. For example, following the above explanation, different colors, hues, grayscales, and / or patterns in circle 837 may correspond to different ranges of artifacts, from unacceptable or poor artifacts to minimal artifacts and to acceptable artifacts.

[0166] In some embodiments, the graphical identification of artifact locations within an image sequence is based on hysteresis. That is, for each image in the sequence, a given artifact, such as a bubble, can generally be placed in the same or substantially the same position compared to the previous and / or subsequent image. These positions may be affected by numerical quantization and other noise factors that slightly change or alter the positional values ​​from one image to the other. Due to hysteresis, the artifact position graphic on the image display, such as the color-coded circle overlay introduced above, is not updated unless the difference between the previous position and the current position exceeds some threshold. The use of hysteresis can prevent distracting, visible jitter in the position of image artifacts during real-time display of the image sequence. This hysteresis can also be applied to numerical data and text descriptions such as "good image" and "bad image".

[0167] In some embodiments, the location of a particular artifact within a displayed image is indicated in various ways. Artifacts may be displayed in real time or only for still images, for example, based on selection by the operator. In some embodiments, a set of options may be specified, for example, set by the individual operator, so that the display options are retained between examinations and / or procedures and the operator does not need to select them for each patient. Furthermore, various operating modes (e.g., novice mode and / or expert mode) may exist, resulting in different levels of assistance and / or identification being provided to the operator based on the procedure or the operating mode selected or activated for the operator. In some embodiments, identified artifacts may be displayed numerically for both individual artifacts and the overall artifact, along with a text display of an artifact description such as "bubble." Such text displays of artifact descriptions may include artifact severity (e.g., quality score) and area values. In some embodiments, the overall artifact value may include the total image quality score, the total number of artifacts in the image, the number of each type of artifact, and the total area affected by the artifact.

[0168] Exemplary items Implementation examples are described in the following numbered sections.

[0169] Item 1: A system for processing image data from an intraoperative diagnostic device in real time during an ophthalmic procedure, comprising: an image acquisition element configured to acquire a grayscale image of a patient's eye from an intraoperative diagnostic device, wherein the grayscale image has a first size; an image processing element configured to acquire a grayscale image from the image acquisition element, scale the grayscale image from a first size to a second size, and preprocess the scaled grayscale image in preparation for classification; and a two-stage classification model comprising: a feature extraction stage configured to process the scaled grayscale image and generate a feature vector based on the scaled grayscale image; and a classification stage configured to process the feature vector and generate an output vector based on the feature vector, wherein the image processing element is further configured to determine the image quality of the acquired grayscale image based on an output vector of artifact probabilities for display to an operator, wherein the image quality of the acquired grayscale image indicates the probability that the acquired grayscale image contains artifacts.

[0170] Item 2: The system according to Item 1, further comprising a data display element configured to generate a display to an operator of the image quality of an acquired grayscale image.

[0171] Item 3: The system described in Item 2, wherein the data display element is further configured to display an acquired grayscale image based on the determination that the grayscale image contains artifacts and that the artifacts reduce the image quality of the acquired grayscale image below a threshold.

[0172] Item 4: The system according to Item 3, wherein the data display element is further configured to generate a display of an acquired grayscale image having an indicator that identifies the location of artifacts in the acquired grayscale image, based on the determination that the grayscale image contains artifacts that degrade the image quality below a threshold.

[0173] Item 5: The system according to Item 4, further comprising at least one of an image segmentation model or an object detection model configured to identify the location of artifacts for display to an operator based on an acquired grayscale image.

[0174] Item 6: A data display element further configured to display a first indicator showing an overall quality representation relative to the average image quality of a subset of images being processed, and a second indicator showing the image quality for the acquired grayscale image, wherein the image quality for the acquired grayscale image is based on the probability that artifacts are present in the scaled and preprocessed grayscale image, as described in any one of Items 2 to 5.

[0175] Item 7: The system according to any one of Items 1 to 6, wherein an image processing element is further configured to create a set of three scaled grayscale images for preprocessing scaled grayscale images, and a feature extraction step is further configured to generate feature vectors based on the set of three scaled grayscale images for processing the scaled grayscale images and generating feature vectors based on the scaled grayscale images, with each scaled grayscale image in the set of three grayscale images being used as an input data channel to the feature extraction step.

[0176] Item 8: The system described in any one of items 1 to 7, wherein the feature extraction stage includes a convolutional neural network and the classification stage includes one or more fully connected layers and a sigmoid activation function.

[0177] Item 9: The system according to any one of items 1 to 8, wherein the acquired grayscale image includes a wide-field image type and the artifact is caused by one or more of debris or equipment; the acquired grayscale image includes a focus-view image type and the artifact is caused by one or more of dryness or excess fluid at the patient's eye position; or the acquired grayscale image includes an interferogram-view image type and the artifact includes one or more of glints, bubbles, or suspended matter in the acquired grayscale image.

[0178] Item 10: A second two-stage classification model comprising: a second feature extraction stage configured to process a second scaled grayscale image and generate a second feature vector based on the second grayscale image; and a second classification stage configured to process the second feature vector and generate a second output vector based on the second feature vector; and a third two-stage classification model comprising: a third feature extraction stage configured to process a third scaled grayscale image and generate a third feature vector based on the third grayscale image; and a third classification stage configured to process the third feature vector and generate a third output vector based on the third feature vector. The system according to item 9, further comprising a third two-stage classification model, wherein the scaled grayscale image processed by the feature extraction stage includes a first of the wide-field image type, focus-view image type, and interferogram-view image type; the second grayscale image processed by the second feature extraction stage includes another of the wide-field image type, focus-view image type, and interferogram-view image type; and the third grayscale image processed by the third feature extraction stage includes a third of the wide-field image type, focus-view image type, and interferogram-view image type that are not processed by the feature extraction stage and the second feature extraction stage.

[0179] Item 11: The system according to Item 10, wherein a scaled grayscale image is processed by a feature extraction step in parallel with a second grayscale image being processed by a second feature extraction step and a third grayscale image being processed by a third feature extraction step.

[0180] Item 12: The system according to either Item 10 or Item 11, wherein a scaled grayscale image is processed sequentially with a feature extraction step, the second grayscale image being processed sequentially with a feature extraction step, the third grayscale image being processed with a third feature extraction step.

[0181] Item 13: The system described in any one of items 1 to 13, wherein the intraoperative diagnostic device is configured to perform refractive analysis on an image.

[0182] Item 14: The system described in any one of items 1 to 14, wherein the image processing element is further configured to exclude grayscale images from further processing if the image quality falls below a first threshold based on artifacts.

[0183] Item 15: A method for processing image data acquired from an intraoperative diagnostic device in real time during an ophthalmic procedure, comprising: acquiring a grayscale image of a patient's eye from the intraoperative diagnostic device, wherein the grayscale image has a first size; acquiring the grayscale image from an image acquisition element; preprocessing the grayscale image in preparation for classification by a two-stage machine learning model; generating a feature vector based on the preprocessed grayscale image by the feature extraction stage of the two-stage machine learning model; generating an output vector based on the feature vector by the classification stage of the two-stage machine learning model; and determining the image quality of the acquired grayscale image based on the output vector for display to an operator, wherein the image quality of the acquired grayscale image indicates the probability that the acquired grayscale image contains artifacts that interfere with measurements by the intraoperative diagnostic device.

[0184] The method of the

[0185] Item 17: The method according to any one of items 15 and 16, wherein preprocessing a grayscale image includes scaling the grayscale image from a first size to a second size.

[0186] The method according to paragraph 17, wherein preprocessing a scaled grayscale image further comprises duplicating a scaled grayscale image to create a set of three scaled grayscale images, and generating feature vectors based on the scaled grayscale images comprises generating feature vectors based on the set of three scaled grayscale images, wherein each scaled grayscale image in the set of three grayscale images is used as an input data channel to the feature extraction stage.

[0187] Item 19: The method according to any one of items 15 to 18, wherein the feature extraction step comprises a convolutional neural network and the classification step comprises one or more fully connected layers and a sigmoid activation function.

[0188] Item 20: The method according to any one of items 15 to 19, wherein the acquired grayscale image includes a wide-field image type and the artifact is caused by one or more of debris or equipment; the acquired grayscale image includes a focus-view image type and the artifact is caused by one or more of dryness or excess fluid at the patient's eye position; or the acquired grayscale image includes an interferogram-view image type and the artifact includes one or more of glints, bubbles, or suspended objects in the acquired grayscale image.

[0189] Item 21: The method according to Item 20, further comprising: generating a second feature vector based on a second grayscale image by a second feature extraction step of a second two-stage machine learning model; generating a second output vector based on the second feature vector by a second classification step of a second two-stage machine learning model; generating a third feature vector based on a third grayscale image by a third feature extraction step of a third two-stage machine learning model; and generating a third output vector based on the third feature vector by a third classification step of a third two-stage machine learning model, wherein the grayscale image includes a first of a wide-field image type, a focus-view image type, and an interferogram-view image type; the second grayscale image includes another of a wide-field image type, a focus-view image type, and an interferogram-view image type; and the third grayscale image includes a third of a wide-field image type, a focus-view image type, and an interferogram-view image type that is not processed by the feature extraction step and the second feature extraction step.

[0190] Item 22: The method according to Item 21, wherein the feature vector is generated in parallel with the second and third feature vectors.

[0191] Item 23: The method according to either item 21 or 22, wherein a feature vector is generated in series with a second feature vector, and a third feature vector is generated in series with a third feature vector.

[0192] Item 24: A method for training a two-stage machine learning model to identify artifacts in images acquired from an intraoperative aberration analyzer during an ophthalmic procedure, comprising: acquiring images; generating feature vectors for each of the images by a feature extraction stage of a two-stage machine learning model; generating a feature matrix based on the stacking of the generated feature vectors; and training a classification stage based on the feature matrix, wherein the trained classification stage generates an output of a processed image indicating the probability that the image contains artifacts.

[0193] Item 25: The method according to item 24, further comprising training the weights for the feature extraction stage.

[0194] Item 26: A processing system comprising: a memory containing computer executable instructions; and one or more processors configured to execute computer executable instructions and cause the processing system to perform the method described in any one of claims 1 to 25.

[0195] Item 27: A processing system comprising means for performing the method described in any one of claims 1 to 25.

[0196] Item 28: A non-temporary computer-readable medium comprising computer-executable instructions, which, when executed by one or more processors of a processing system, cause the processing system to perform the method described in any one of claims 1 to 25.

[0197] Item 29: A computer program product, embodied on a computer-readable storage medium, which includes code for performing any method described in any one of items 1 through 25.

[0198] Additional considerations The foregoing description is provided so that those skilled in the art can practice the various embodiments described herein. The examples described herein do not limit the scope, applicability, or embodiments described in the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. For example, changes may be made to the function and arrangement of the elements discussed without departing from the scope of this disclosure. Various procedures or components may be omitted, replaced, or added in various examples as needed. For example, the described methods may be performed in an order different from the order described herein, and various steps may be added, omitted, or combined. Also, features described in some examples may be combined in some other examples. For example, an apparatus may be implemented or a method may be carried out using any number of embodiments described herein. Furthermore, the scope of this disclosure is intended to cover such apparatus or methods carried out using, in addition to, or other structures, functions, or structures and functions, in addition to the various embodiments of this disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the claims.

[0199] As used herein, the term “exemplary” means “to serve as an example, case, or illustration.” Any embodiment described herein as “exemplary” should not necessarily be construed as being preferable or more advantageous than any other embodiment.

[0200] Where used herein, the phrase “at least one of” the list of items refers to any combination of those items that contains a single element. For example, “at least one of a, b, or c” is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination of multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc or any other order of a, b, and c).

[0201] As used herein, the term “determining” encompasses a wide range of actions. For example, “determining” may include calculating, calculating, processing, deriving, investigating, searching (e.g., searching in a table, database, or other data structure), confirming, etc. It may also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. It may also include clarifying, selecting, choosing, and determining.

[0202] The methods disclosed herein include one or more steps or actions to achieve the method. The steps and / or actions of the method may be interchangeable with one another without departing from the claims. In other words, unless a specific order of steps or actions is specified, the specific order and / or use of steps and / or actions may be changed without departing from the claims. Furthermore, various operations of the above methods may be performed by any suitable means capable of performing the corresponding functions. These means may include, but are not limited to, circuits, application-specific integrated circuits (ASICs), or processors, but include various hardware and / or software components and / or modules. Generally, where there are operations shown in the figures, those operations may include corresponding equivalent means and functional components that are similarly numbered.

[0203] The following claims are not intended to be limited to the embodiments described herein, and the full scope consistent with the language of the claims should be recognized. In the claims, a singular reference to an element is not intended to mean "one and only one," but rather "one or more," unless specifically provided otherwise. Unless specifically provided otherwise, the term "some" refers to one or more. No element of the claims should be construed under Section 112(f) of the U.S. Patent Act unless the element is expressly enumerated using the phrase "means for," or, in the case of a method claim, the element is enumerated using the phrase "step for." All structural and functional equivalents of elements in various forms described throughout this disclosure, known or to those skilled in the art, are expressly incorporated by reference herein and are intended to be included in the claims. Furthermore, nothing disclosed herein is intended to be dedicated to the public, whether or not such disclosure is explicitly enumerated in the claims. According to embodiment (1), a system for processing image data from an intraoperative diagnostic device in real time during an ophthalmic procedure, An image acquisition element configured to acquire a grayscale image of a patient's eye from the intraoperative diagnostic device, wherein the grayscale image has a first size; Image processing element, The grayscale image is obtained from the image acquisition element. The grayscale image is scaled from the first size to generate a scaled grayscale image of the second size. An image processing element configured to preprocess the scaled grayscale image in preparation for classification, It is a two-stage classification model, A feature extraction step configured to process the scaled grayscale image and generate a feature vector based on the scaled grayscale image, A two-stage classification model comprising: a classification stage configured to process the feature vectors and generate an output vector based on the feature vectors, The image processing element is further configured to determine the image quality of the grayscale image based on the output vector for display to the operator. The system is such that the image quality of the grayscale image indicates the probability that the grayscale image contains artifacts. According to embodiment (2), the system further comprises a data display element configured to generate a display of the image quality of the grayscale image to the operator. According to embodiment (3), the data display element is further configured to display the grayscale image based on the determination that the grayscale image contains the artifact and the artifact reduces the image quality of the grayscale image to below a threshold. According to embodiment (4), the data display element is further configured to generate the display of the grayscale image having an indicator that identifies the location of the artifact in the grayscale image, based on the determination that the grayscale image contains the artifact which degrades the image quality below a threshold. According to embodiment (5), the system further comprises at least one of an image segmentation model or an object detection model configured to identify the location of the artifact for display to the operator based on the grayscale image. According to embodiment (6), the data display element is A first indicator shows the overall quality relative to the average image quality of a subset of the images being processed, Further configured to display a second indicator showing the image quality for the grayscale image, The image quality of the grayscale image is based on the probability that the artifact is present in the scaled and preprocessed grayscale image. According to embodiment (7), in order to preprocess the scaled grayscale image, the image processing element is further configured to create a set of three scaled grayscale images, To process the scaled grayscale images and generate feature vectors based on the scaled grayscale images, the feature extraction step is further configured to generate the feature vectors based on the set of three scaled grayscale images. Each scaled grayscale image in the set of three scaled grayscale images is used as an input data channel to the feature extraction step. According to embodiment (8), the feature extraction step includes a convolutional neural network, The classification stage includes one or more fully connected layers and a sigmoid activation function. According to embodiment (9), the grayscale image includes a wide-field image type, and the artifact is caused by one or more pieces of debris or equipment. The grayscale image includes a focus view image type, and the artifact is caused by one or more of the following at the patient's eye position: dryness or excess fluid, or The grayscale image includes an interferogram view image type, and the artifact includes one or more glints, bubbles, or suspended particles in the grayscale image. According to embodiment (10), the second two-stage classification model is, A second feature extraction step configured to process a second scaled grayscale image and generate a second feature vector based on the second scaled grayscale image, A second two-stage classification model, comprising: a second classification stage configured to process the second feature vector and generate a second portion of the output vector based on the second feature vector; This is a third two-stage classification model, A third feature extraction step configured to process a third scaled grayscale image and generate a third feature vector based on the third scaled grayscale image, A third two-stage classification model further comprises: a third classification stage configured to process the third feature vector and generate a third portion of the output vector based on the third feature vector, The scaled grayscale image processed by the feature extraction step includes a first of the wide-field image type, the focus-view image type, and the interferogram-view image type. The second scaled grayscale image processed by the second feature extraction step includes another of the wide-field image type, the focus-view image type, and the interferogram-view image type. The third scaled grayscale image processed by the third feature extraction step includes a third of the wide-field image type, the focus-view image type, and the interferogram-view image type that are not processed by the feature extraction step and the second feature extraction step. According to embodiment (11), the scaled grayscale image is processed by the feature extraction step in parallel with the processing of the second scaled grayscale image by the second feature extraction step and the processing of the third scaled grayscale image by the third feature extraction step. According to embodiment (12), the scaled grayscale image is processed in the same order as the third scaled grayscale image is processed in the third feature extraction step, and the second scaled grayscale image is processed in the same order as the second feature extraction step. According to embodiment (13), the intraoperative diagnostic device is configured to perform refractive analysis on the image data. According to embodiment (14), the image processing element is further configured to exclude the grayscale image from further processing if the image quality falls below a first threshold based on the artifacts. According to embodiment (15), a method for processing image data acquired from an intraoperative diagnostic device during an ophthalmic procedure in real time, The process involves acquiring a grayscale image of the patient's eye from the intraoperative diagnostic device, wherein the grayscale image has a first size. Obtaining the grayscale image from the image capture element, The grayscale image is preprocessed in preparation for classification using a two-stage machine learning model, The feature extraction step of the two-stage machine learning model generates feature vectors based on the preprocessed grayscale image, The classification stage of the two-stage machine learning model generates an output vector based on the feature vector, This includes determining the image quality of the grayscale image based on the output vector for display to the operator, The method is such that the image quality of the grayscale image indicates the probability that the grayscale image contains artifacts that interfere with measurements by the intraoperative diagnostic device. According to embodiment (16), the method generates a display of the image quality of the grayscale image to the operator, The further includes identifying the location of the artifact in the grayscale image based on the determination that the grayscale image contains the artifact which reduces the image quality to below a first threshold. According to embodiment (17), preprocessing the grayscale image includes scaling the grayscale image of the first size to generate a scaled grayscale image of a second size. According to embodiment (18), preprocessing the scaled grayscale image further includes duplicating the scaled grayscale image to create a set of three scaled grayscale images, Generating a feature vector based on the scaled grayscale image includes generating the feature vector based on the set of three scaled grayscale images, Each scaled grayscale image in the set of three scaled grayscale images is used as an input data channel to the feature extraction step. According to embodiment (19), the feature extraction step includes a convolutional neural network, The classification stage includes one or more fully connected layers and a sigmoid activation function. According to embodiment (20), the grayscale image includes a wide-field image type, and the artifact is caused by one or more pieces of debris or equipment. The grayscale image includes a focus view image type, and the artifact is caused by one or more of the following at the patient's eye position: dryness or excess fluid, or The grayscale image includes an interferogram view image type, and the artifact includes one or more glints, bubbles, or suspended particles in the grayscale image. According to embodiment (21), the second feature extraction step of the second two-stage machine learning model generates a second feature vector based on the second grayscale image, The second classification stage of the second two-stage machine learning model generates a second output vector based on the second feature vector, The third feature extraction step of the third two-stage machine learning model generates a third feature vector based on the third grayscale image, The third two-stage machine learning model further includes generating a third output vector based on the third feature vector by the third classification stage of the third two-stage machine learning model, The grayscale image includes a first of the wide-field image type, the focus-view image type, and the interferogram-view image type. The second grayscale image includes another of the wide-field image type, the focus-view image type, and the interferogram-view image type. The third grayscale image includes a third of the wide-field image type, the focus-view image type, and the interferogram-view image type that are not processed by the feature extraction step and the second feature extraction step. According to embodiment (22), the feature vector is generated in parallel with the second feature vector and the third feature vector. According to embodiment (23), the feature vector is generated in series with the second feature vector and in series with the third feature vector. According to embodiment (24), a method for training a two-stage machine learning model to identify artifacts in images acquired from an intraoperative aberration analyzer during ophthalmic procedures, To obtain the aforementioned image, The aforementioned image is used to generate feature vectors through the feature extraction step of the two-stage machine learning model, The process involves generating a feature matrix based on stacking the aforementioned generated feature vectors, This includes training a classification stage based on the aforementioned feature matrix, The method involves the trained classification stage generating an output for a processed image indicating the probability that the image contains artifacts. According to embodiment (25), the method further includes training the weights of the feature extraction step.

Claims

1. A system for processing image data from intraoperative diagnostic devices in real time during ophthalmic procedures, An image acquisition element configured to acquire a grayscale image of a patient's eye from the intraoperative diagnostic device, wherein the grayscale image has a first size. Image processing element, The grayscale image is obtained from the image acquisition element. The grayscale image is scaled from the first size to generate a scaled grayscale image of the second size. An image processing element configured to preprocess the scaled grayscale image in preparation for classification, It is a two-stage classification model, A feature extraction step configured to process the scaled grayscale image and generate a feature vector based on the scaled grayscale image, A two-stage classification model comprising: a classification stage configured to process the feature vector and generate an output vector based on the feature vector; A data display element configured to generate a display of the image quality of the grayscale image to an operator, Equipped with, The image processing element is further configured to determine the image quality of the grayscale image based on the output vector for display to the operator. The image quality of the grayscale image indicates the probability that the grayscale image contains artifacts. The data display element is further configured to display the grayscale image based on the determination that the grayscale image contains the artifact and that the artifact reduces the image quality of the grayscale image to below a threshold. system.

2. The system according to claim 1, wherein the data display element is further configured to generate a display of the grayscale image having an indicator for identifying the location of the artifact in the grayscale image, based on the determination that the grayscale image contains the artifact which degrades the image quality below a threshold.

3. The system according to claim 2, further comprising at least one of an image segmentation model or an object detection model configured to identify the location of the artifact for display to the operator based on the grayscale image.

4. The aforementioned data display element is A first indicator that shows an overall quality representation of the average image quality of a subset of images being processed, It is further configured to display a second indicator showing the image quality of the grayscale image, The system according to claim 1, wherein the image quality of the grayscale image is based on the probability that the artifact is present in the scaled and preprocessed grayscale image.

5. A system for processing image data from an intraoperative diagnostic device in real time during an ophthalmic procedure, An image acquisition element configured to acquire a grayscale image of a patient's eye from the intraoperative diagnostic device, wherein the grayscale image has a first size. Image processing element, The grayscale image is obtained from the image acquisition element. The grayscale image is scaled from the first size to generate a scaled grayscale image of the second size. The image processing element is configured to preprocess the scaled grayscale image in preparation for classification, It is a two-stage classification model, A feature extraction step configured to process the scaled grayscale image and generate a feature vector based on the scaled grayscale image, The two-stage classification model includes a classification stage configured to process the feature vectors and generate an output vector based on the feature vectors, The image processing element is further configured to determine the image quality of the grayscale image based on the output vector for display to the operator. The image quality of the grayscale image indicates the probability that the grayscale image contains artifacts. To preprocess the scaled grayscale image, the image processing element is further configured to create a set of three scaled grayscale images. To process the scaled grayscale images and generate the feature vectors based on the scaled grayscale images, the feature extraction step is further configured to generate the feature vectors based on the set of three scaled grayscale images. A system in which each of the three scaled grayscale images in the set of scaled grayscale images is used as an input data channel to the feature extraction stage.

6. A system for processing image data from an intraoperative diagnostic device in real time during an ophthalmic procedure, An image acquisition element configured to acquire a grayscale image of a patient's eye from the intraoperative diagnostic device, wherein the grayscale image has a first size. Image processing element, The grayscale image is obtained from the image acquisition element. The grayscale image is scaled from the first size to generate a scaled grayscale image of the second size. The image processing element is configured to preprocess the scaled grayscale image in preparation for classification, It is a two-stage classification model, A feature extraction step configured to process the scaled grayscale image and generate a feature vector based on the scaled grayscale image, The two-stage classification model includes a classification stage configured to process the feature vectors and generate an output vector based on the feature vectors, The image processing element is further configured to determine the image quality of the grayscale image based on the output vector for display to the operator. The image quality of the grayscale image indicates the probability that the grayscale image contains artifacts. The feature extraction step includes a convolutional neural network, A system in which the classification step comprises one or more fully connected layers and a sigmoid activation function.

7. A system for processing image data from an intraoperative diagnostic device in real time during an ophthalmic procedure, An image acquisition element configured to acquire a grayscale image of a patient's eye from the intraoperative diagnostic device, wherein the grayscale image has a first size. Image processing element, The grayscale image is obtained from the image acquisition element. The grayscale image is scaled from the first size to generate a scaled grayscale image of the second size. The image processing element is configured to preprocess the scaled grayscale image in preparation for classification, It is a two-stage classification model, A feature extraction step configured to process the scaled grayscale image and generate a feature vector based on the scaled grayscale image, The two-stage classification model includes a classification stage configured to process the feature vectors and generate an output vector based on the feature vectors, The image processing element is further configured to determine the image quality of the grayscale image based on the output vector for display to the operator. The image quality of the grayscale image indicates the probability that the grayscale image contains artifacts. The grayscale image includes a wide-field image type, and the artifact is caused by one or more pieces of debris or equipment. The grayscale image includes a focus view image type, and the artifact is caused by one or more of the following at the patient's eye location: dryness or excess fluid, or A system wherein the grayscale image includes an interferogram view image type, and the artifact includes one or more glints, bubbles, or suspended particles in the grayscale image.

8. The second two-stage classification model is, A second feature extraction step configured to process a second scaled grayscale image and generate a second feature vector based on the second scaled grayscale image, A second two-stage classification model, comprising: a second classification stage configured to process the second feature vector and generate a second portion of the output vector based on the second feature vector; This is a third two-stage classification model, A third feature extraction step configured to process a third scaled grayscale image and generate a third feature vector based on the third scaled grayscale image, A third two-stage classification model further includes: a third classification step configured to process the third feature vector and generate a third portion of the output vector based on the third feature vector; The scaled grayscale image processed by the feature extraction step includes a first of the wide-field image type, the focus-view image type, and the interferogram-view image type. The second scaled grayscale image processed by the second feature extraction step includes another of the wide-field image type, the focus-view image type, and the interferogram-view image type. The system according to claim 7, wherein the third scaled grayscale image processed by the third feature extraction step includes a third of the wide-field image type, the focus-view image type, and the interferogram-view image type that are not processed by the feature extraction step and the second feature extraction step.

9. The system according to claim 8, wherein the scaled grayscale image is processed by the feature extraction step in parallel with the processing of the second scaled grayscale image by the second feature extraction step and the processing of the third scaled grayscale image by the third feature extraction step.

10. The system according to claim 8, wherein the scaled grayscale image is processed in series with the third scaled grayscale image being processed by the third feature extraction step, and the second scaled grayscale image is processed in series with the second feature extraction step.

11. A system for processing image data from an intraoperative diagnostic device in real time during an ophthalmic procedure, An image acquisition element configured to acquire a grayscale image of a patient's eye from the intraoperative diagnostic device, wherein the grayscale image has a first size. Image processing element, The grayscale image is obtained from the image acquisition element. The grayscale image is scaled from the first size to generate a scaled grayscale image of the second size. The image processing element is configured to preprocess the scaled grayscale image in preparation for classification, It is a two-stage classification model, A feature extraction step configured to process the scaled grayscale image and generate a feature vector based on the scaled grayscale image, The two-stage classification model includes a classification stage configured to process the feature vectors and generate an output vector based on the feature vectors, The image processing element is further configured to determine the image quality of the grayscale image based on the output vector for display to the operator. The image quality of the grayscale image indicates the probability that the grayscale image contains artifacts. A system in which the intraoperative diagnostic device is configured to perform refractive analysis on the image data.

12. A system for processing image data from an intraoperative diagnostic device in real time during an ophthalmic procedure, An image acquisition element configured to acquire a grayscale image of a patient's eye from the intraoperative diagnostic device, wherein the grayscale image has a first size. Image processing element, The grayscale image is obtained from the image acquisition element. The grayscale image is scaled from the first size to generate a scaled grayscale image of the second size. The image processing element is configured to preprocess the scaled grayscale image in preparation for classification, It is a two-stage classification model, A feature extraction step configured to process the scaled grayscale image and generate a feature vector based on the scaled grayscale image, The two-stage classification model includes a classification stage configured to process the feature vectors and generate an output vector based on the feature vectors, The image processing element is further configured to determine the image quality of the grayscale image based on the output vector for display to the operator. The image quality of the grayscale image indicates the probability that the grayscale image contains artifacts. A system wherein the image processing element is further configured to exclude the grayscale image from further processing if the image quality falls below a first threshold based on the artifacts.

13. A method for training a two-stage machine learning model to identify artifacts in images acquired in the system according to claim 1 for processing image data from the intraoperative diagnostic device in real time, To obtain the aforementioned image, The feature vector is generated for the aforementioned image by the feature extraction step of the two-stage machine learning model, A feature matrix is ​​generated by stacking the generated feature vectors, This includes training a classification stage based on the aforementioned feature matrix, A method wherein the trained classification stage generates an output for a processed image indicating the probability that the image contains the artifact.

14. The method according to claim 13, further comprising training the weights of the feature extraction step.

Images