Digital Image Optimization for Ophthalmic Surgery

Abstract

The present disclosure provides a digital image optimization system including a camera including at least one sensor that detects light reflected from an eye and sends a signal corresponding to the detected light to a processor. The system further includes an image processing system including the processor and executing instructions to generate a digital image of the eye. The image processing system also executes instructions to apply a digital image optimization algorithm including a trained machine learning model to the digital image of the eye to generate an optimized digital image of the eye. The system further includes a digital image processor that displays the optimized digital image of the eye.

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates to ophthalmic surgery and surgical equipment, and more particularly to digital image optimization systems and related methods for improving digital visualization in ophthalmic surgery. [Background technology]

[0002] Ophthalmic surgery is surgery performed on the eye or any part of the eye. Ophthalmic surgery protects and improves the vision of tens of thousands of patients each year. However, because vision is sensitive to even slight changes in the eye and the characteristics of many ocular structures are delicate and intricate, ophthalmic surgery can be difficult to perform, and reducing minor or rare surgical errors or even slight improvements in precision of surgical skill can make a noticeable difference in a patient's vision after surgery.

[0003] Vitreoretinal surgery, a type of ophthalmic surgery, encompasses a variety of delicate procedures involving the interior of the eye, such as the vitreous humor, retina, and vitreoretinal membranes. Different vitreoretinal surgical procedures, sometimes in conjunction with lasers, are used to improve visual sensation performance in the treatment of many eye diseases, including outer membranes, diabetic retinopathy, vitreous hemorrhage, macular hole, retinal detachment, and complications of cataract surgery, among others.

[0004] During ophthalmic procedures, such as vitreoretinal surgery, ophthalmologists typically use non-electronic optical surgical microscopes with eyepieces to view magnified images of the eye undergoing surgery. More recently, vitreoretinal surgeons have used digital imaging systems without eyepieces to aid in visualization during vitreoretinal surgery. These systems may include three-dimensional (3D) high-dynamic-range ("HDR") camera systems with 2D complementary metal-oxide semiconductor (CMOS) single-chip paired or triple-chip sensors that allow surgeons to view the retina on a display screen using polarized glasses, digital eyepieces, or a head-mounted display. The display screen eliminates the need to view the procedure using eyepieces, allowing others in the operating room to see exactly as the surgeon does. This system also allows for improved digital imaging and increased depth of field at high magnifications compared to traditional optical analog surgical microscopes, thereby improving visualization of the eye. Summary of the Invention [Means for solving the problem]

[0005] The present disclosure provides a digital image optimization system and associated methods for improving digital visualization in ophthalmic surgery. The digital image optimization system includes a camera including at least one sensor that detects light reflected from the eye and sends a signal corresponding to the detected light to a processor. The digital image optimization system also includes an image processing system including a processor. The image processing system executes instructions to generate a digital image of the eye and executes instructions to apply a digital image optimization algorithm including a trained machine learning model to the digital image of the eye to generate an optimized digital image of the eye. The digital image optimization system also includes a digital display that displays the optimized digital image of the eye.

[0006] The digital image optimization system and method of use may include the following additional features. i) the digital image optimization algorithm may at least partially reduce instrument glare, optical aberrations, vitreous opacity, or any combination thereof in the digital image of the eye to generate an optimized digital image of the eye; ii) the optimized digital image of the eye may have equal or greater contrast, sharpness, clarity, dynamic range, or any combination thereof than the digital image of the eye; iii) the optimized digital image of the eye may have less noise, distortion, vignetting, or any combination thereof than the digital image of the eye; iv) the digital image optimization algorithm may utilize interpolation, multi-exposure image noise reduction, or a combination thereof to generate the optimized digital image of the eye; v) the trained machine learning model may have been trained using a plurality of training images from a red boost library; vi) the trained machine learning model may have been trained using a plurality of digital images of the eye captured at the start of surgery and a plurality of corresponding digital images of the eye captured at the end of surgery; and vii) the digital image optimization system may be a component of the NGENUITY® 3D visualization system (Novartis AG Corp., Switzerland).

[0007] The present disclosure further provides a digital image optimization system including a camera including at least one sensor that detects light reflected from the fellow eye and sends a signal corresponding to the detected light to a processor. The at least one sensor also detects light reflected from the surgical eye and sends a signal corresponding to the detected light to the processor. The digital image optimization system also includes an image processing system including a processor. The image processing system executes instructions to generate a template digital image of the fellow eye, executes instructions to generate a digital mirror image of the template digital image, executes instructions to generate a working digital image of the surgical eye, and executes instructions to align the digital mirror image with the working digital image to generate a retinal arcade alignment template. The digital image optimization system also includes a digital display that displays the working digital image and the retinal arcade alignment template.

[0008] The digital image optimization system and method of use may include the following additional features. i) the retinal arcade alignment template may be displayed as an overlay on the working digital image; ii) the template digital image may include ocular structures that are the retina of the fellow eye, the central retinal vein of the fellow eye, the retinal arcade of the fellow eye, the optic nerve disc of the fellow eye, or any combination thereof; iii) the working digital image may include the retina of the surgical eye, the central retinal vein of the surgical eye, the retinal arcade of the surgical eye, the optic nerve disc of the surgical eye, or any combination thereof; iv) the working digital image may include a field of view of the surgical eye that is equivalent to the field of view of the template digital image of the fellow eye; v) aligning the digital mirror image to the working digital image may include aligning ocular structures that are the optic nerve disc of the fellow eye and the optic nerve disc of the surgical eye, the central retinal vein of the fellow eye and the central retinal vein of the surgical eye, the arterial-vein intersection of the fellow eye and the arterial-vein intersection of the surgical eye, or any combination thereof; and vi) the digital image optimization system may be a component of the NGENUITY® 3D visualization system (Novartis AG Corp., Switzerland).

[0009] The present disclosure further provides a digital image optimization system including an endoscope including an optical fiber. The digital image optimization system also includes a camera including at least one sensor that detects light reflected from the interior of the eye and propagating through the optical fiber. The sensor sends a signal corresponding to the detected light to a processor. The digital image optimization system also includes an image processing system including the processor. The image processing system receives instructions to generate an endoscopic digital image of the eye and executes the instructions to apply an endoscopic digital image optimization algorithm including a trained machine learning model to the endoscopic digital image of the eye to generate an optimized endoscopic digital image of the eye. The digital image optimization system also includes a digital display that displays the optimized endoscopic digital image of the eye.

[0010] The digital image optimization system and method of use may include the following additional features: i) the endoscopic digital image optimization algorithm may utilize interpolation, multi-exposure image noise reduction, or a combination thereof to generate optimized endoscopic digital images of the eye, ii) the machine learning model may be pre-trained using a plurality of training images including a plurality of endoscopic digital images of the eye that have been successfully optimized using image processing methods, iii) the endoscopic digital image optimization algorithm may utilize single-image super-resolution to generate optimized endoscopic digital images of the eye, iv) the single-image super-resolution may use a very deep super-resolution neural network, v) the endoscopic digital image optimization algorithm may at least in part reduce noise, increase resolution, improve digital image quality, or any combination thereof, of the endoscopic digital images of the eye, and vi) the digital image optimization system may be a component of the NGENUITY® 3D visualization system (Novartis AG Corp., Switzerland).

[0011] The present disclosure also provides an optimization network including a digital image optimization system. The digital image optimization system includes a camera including at least one sensor that detects light reflected from the eye and transmits a signal corresponding to the detected light to a processor. The digital image optimization system further includes a first image processing system including a first processor and configured to execute instructions to generate a digital image of the eye. The digital image optimization system further includes a digital display. The optimization network also includes a communications network communicatively coupled to the digital image optimization system and the digital image management system. The optimization network transmits the digital image of the eye to the digital image management system. The optimization network also includes the digital image management system. The digital image management system includes a second image processing system including a second processor and configured to include a trained machine learning model. The second image processing system executes instructions to process the digital image of the eye using the trained machine learning model to calculate an optimized digital image of the eye and transmit the optimized digital image of the eye to a digital display. The digital display displays the optimized digital image of the eye.

[0012] The optimization network and its method of use may include the following additional features. i) the second image processing system may be stored locally as part of the first image processing system; ii) the second image processing system may operate on a server; iii) the second image processing system may provide a cloud-based digital image optimization service; iv) the trained machine learning model may have been trained using a plurality of training images; v) the plurality of training images may include a plurality of digital images of the eye captured at the start of surgery and a plurality of paired digital images of the same eye captured at the end of surgery; vi) the plurality of training images may include a plurality of endoscopic digital images of the eye that have been successfully optimized using the image processing method, and the plurality of training images may have been captured using an NGENUITY® 3D visualization system (Novartis AG Corp., Switzerland); vii) the plurality of training images may be uploaded to a digital image management system using a communication network; viii) the trained machine learning model may have been trained using an integrated architecture including a convolutional neural network and at least one long-term short-term memory unit; ix) the digital image optimization system may be integrated into an NGENUITY® 3D visualization system (Novartis AG Corp., Switzerland). Corp., Switzerland).

[0013] The present disclosure further provides a digital time-out system including an identification marker secured to a patient that includes machine-readable information. The digital time-out system also includes a camera including at least one sensor that acquires the machine-readable information and transmits a signal corresponding to the machine-readable information to a processor. The digital time-out system also includes an image processing system including a processor that processes the machine-readable information to identify patient data. The image processing system also determines discrepancies, matches, or a combination thereof between the patient data and information provided to a surgeon, operating room staff, or a combination thereof.

[0014] The digital time-out system and method of use may include the following additional features: i) the digital time-out system may further include a digital display that displays discrepancies, matches, or a combination thereof between the patient data and information provided to the surgeon, operating room staff, or a combination thereof; ii) the identification marker may be secured over the patient's eye; iii) the identification marker may be a removable transfer tattoo, a patch, a sticker, tape, or any combination thereof; iv) the identification marker may further include a locator; v) the machine-readable information may be a one-dimensional barcode, a multi-dimensional barcode, a quick response code, a symbol, or any combination thereof; and vi) the digital time-out system may be integrated into the NGENUITY® 3D Visualization System (Novartis AG). Corp., Switzerland); vii) the patient data may include information associated with the patient, which may be identification information associated with the patient, medical information associated with the patient, indexing information, or any combination thereof; viiii) the camera may detect light reflected from the eye and send a signal corresponding to the detected light to a first processor, and the image processing system may execute instructions to generate a digital image of the eye; ix) the system may include a communications network communicatively coupled to the digital timeout system and the digital image management system. The communications network may send the digital image of the eye to the digital image management system. The digital image management system may include a second image processing system including a second processor and a trained machine learning model. The machine-readable data may include indexing information associated with the digital image of the eye and stored in the digital image management system; and x) the trained machine learning model may have been trained using a plurality of training images including the digital image of the eye. The plurality of training images may have been selected based on the indexing information associated with the digital image of the eye.

[0015] The present disclosure further provides a method of optimizing a digital image of an eye by capturing a digital image of the eye with a camera, applying a digital image optimization algorithm using an image processing system to generate an optimized digital image of the eye, and displaying the optimized digital image of the eye on a digital display. The digital image optimization algorithm may at least partially reduce instrument glare, optical aberrations, vitreous opacities, or any combination thereof, in the digital image of the eye to generate the optimized digital image of the eye. The digital image optimization algorithm may be a trained machine learning model. The trained machine learning model may have been trained using multiple digital images of the eye captured at the beginning of surgery and multiple corresponding digital images of the eye captured at the end of surgery. The camera may be a component of the NGENUITY® 3D visualization system (Novartis AG Corp., Switzerland).

[0016] The present disclosure further provides a method for generating a digital image optimization algorithm by capturing a plurality of input digital images of an eye at the beginning of surgery and a plurality of output digital images of the eye at the end of surgery, using the plurality of input digital images of the eye and the plurality of output digital images of the eye as training images to train a machine learning model, and using the trained machine learning model as the digital image optimization algorithm.

[0017] The present disclosure further provides a method for optimizing a digital image of an eye by capturing a digital image of a fellow eye, capturing a digital image of a surgical eye, generating a mirror image of the digital image of the fellow eye, registering the mirror image of the digital image of the fellow eye with the digital image of the surgical eye to generate a retinal arcade alignment template, displaying the retinal arcade alignment template on a digital display as an overlay on the digital image of the surgical eye, and performing ophthalmic surgery on the surgical eye using the retinal arcade alignment template. The mirror image of the digital image of the fellow eye may be registered with the digital image of the surgical eye by registering the optic disc, the central retinal vein, the arteriovenous intersection, the branch veins, or any combination thereof.

[0018] The present disclosure further provides a method of optimizing an endoscopic digital image of an eye by capturing an endoscopic digital image of the eye using an endoscope, applying an endoscopic digital image optimization algorithm to generate an optimized endoscopic digital image of the eye, and displaying the optimized endoscopic digital image of the eye on a digital display. The endoscopic digital image optimization algorithm may use interpolation, multiple exposure noise reduction, machine learning, deep learning, or any combination thereof.

[0019] The present disclosure further provides a method for generating an endoscopic digital image optimization algorithm by capturing a plurality of endoscopic digital images of an eye, optimizing the plurality of endoscopic digital images of the eye using image processing methods to generate a plurality of optimized endoscopic digital images of the eye, training a machine learning model using the plurality of endoscopic digital images of the eye as input images and using the plurality of optimized endoscopic digital images of the eye as output images, and using the trained machine learning model as the endoscopic digital image optimization algorithm.

[0020] The present disclosure further provides a method for training a machine learning model by using an image processing system to input a plurality of training images, including a plurality of input images and a plurality of corresponding output images, into an integrated architecture; generating a feature representation of the plurality of input images using a convolutional neural network; calculating a plurality of hypothetical differences between a plurality of predicted optimized digital images of the eye and a plurality of corresponding output images using the integrated architecture; calculating a loss between the plurality of hypothetical differences and the plurality of differences calculated between the plurality of input images and the plurality of corresponding output images; evaluating whether the overall loss is less than a predefined threshold; updating parameters of the integrated architecture if the overall loss is not less than the predefined threshold; storing the parameters of the integrated architecture if the overall loss is less than the predefined threshold; and terminating training. The loss may be calculated using a loss function. The loss function may be a smooth L1 loss function. The hypothetical differences may be calculated using the feature representations of the plurality of input images. Once the machine learning model is trained, the image processing system may operate on a server. The image processing system may provide a cloud-based digital image optimization service.

[0021] The present disclosure further provides a method of implementing a digital timeout by using a camera to acquire machine-readable information printed on an identification marker secured to a patient, processing the machine-readable information using an image processing system, determining accuracy of the patient's identification information based on the machine-readable information, and stopping the planned medical procedure if the patient's identification information is incorrect, determining that the planned medical procedure is accurate based on the machine-readable information if the patient's identification information is accurate, stopping the planned medical procedure if the planned medical procedure is incorrect, and performing the planned medical procedure if the planned medical procedure is accurate.

[0022] The present disclosure further provides a method of optimizing a digital image of an eye by capturing a digital image of an eye using a digital image optimization system, associating processed machine-readable information with the digital image of the eye, training a machine learning model using the digital image of the eye, processing the digital image of the eye using the trained machine learning model, generating an optimized digital image of the eye using an image processing system, and displaying the optimized digital image of the eye on a digital display. The digital image of the eye may be selected for inclusion in a set of training images based on the processed machine-readable information.

[0023] Aspects of the digital image optimization system and its method of use may be combined with each other unless they are clearly mutually exclusive. In addition, additional features of the digital image optimization system and its associated methods described above may also be combined with each other unless they are clearly mutually exclusive.

[0024] For a more complete understanding of the present disclosure and its features and advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, which are not to scale and in which like numerals refer to like features, and in which: [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a schematic representation of a digital image optimization system including a camera, an image processing system, and a digital display. [Figure 2A-2B] 2A-2B are schematic representations of training images. [Figure 2C-2D] 2C-2D are schematic representations of training images. [Figure 3] FIG. 3 is a flow chart illustrating a method for optimizing a digital image of an eye to improve digital visualization in ophthalmic surgery. [Figure 4] FIG. 4 is a flow diagram illustrating a method for using machine learning models to generate digital image optimization algorithms for improving digital visualization in ophthalmic surgery. [Figure 5]FIG. 5 is a schematic representation of a digital image optimization system including a camera, an image processing system, and a digital display. [Figure 6] FIG. 6 is a flow chart illustrating a method for optimizing a digital image of the eye by using a retinal arcade alignment template to improve digital visualization in ophthalmic surgery. [Figure 7] FIG. 7 is a schematic representation of a digital image optimization system including an endoscope, an optical fiber, a fiber optic light source, a camera, an image processing system, and a digital display. [Figure 8] FIG. 8 is a flow chart illustrating a method for optimizing endoscopic digital images of the eye for improved digital visualization in ophthalmic surgery. [Figure 9] FIG. 9 is a flow diagram illustrating a method for using machine learning models to generate an endoscopic digital image optimization algorithm for improving digital visualization in ophthalmic surgery. [Figure 10] FIG. 10 is a schematic representation of an optimization network including a digital image optimization system, a communication network, and a digital image management system. [Figure 11] FIG. 11 is a diagram of an example solution for identifying an optimized digital image of the eye. [Figure 12] FIG. 12 is a diagram of an example of an integrated architecture for machine learning models. [Figure 13] FIG. 13 is a flow diagram illustrating a method for training a machine learning model. [Figure 14] FIG. 14 is a schematic representation of a digital timeout system, including an identification marker, a camera, an image processing system, and a digital display. [Figure 15] FIG. 15 is a flow diagram illustrating a method for implementing a digital timeout. [Figure 16] FIG. 16 is a flow chart illustrating a method for optimizing a digital image of an eye in ophthalmic surgery. [Figure 17] FIG. 17 is a schematic representation of a computer system that includes a digital image optimization system. [Figures 18A-18C] 18A-18C are schematic representations of a medical system that includes a digital image optimization system. DETAILED DESCRIPTION OF THE INVENTION

[0026] The present disclosure provides systems and related methods involving digital image optimization for improving digital visualization in ophthalmic surgery.

[0027] Ophthalmologists face unique challenges when visualizing the eye. In particular, aberrations during medical procedures can degrade the quality of digital images of the eye. For example, any view obtained through a patient's pupil is subject to optical aberrations, which can distort the digital image of the eye. Optical aberrations can be caused by eye disease or previous surgery, resulting in asphericity of the cornea or intraocular lens implants, which result in aberrated images viewed by the surgeon. Vitreous opacification can also degrade the quality of digital images. Additionally, stainless steel instruments inserted into the eye can create glare that degrades image quality when viewing the eye during surgery. In another example, low-resolution digital images, such as endoscopic digital images with approximately 30,000 pixels, can be displayed on high-resolution digital displays with over 1 million pixels. This can lead to poor image quality of endoscopic digital images.

[0028] Poor image quality can hinder a surgeon's ability to visualize the inside of the eye, making surgery more difficult. In analog systems, methods for correcting the effects of aberrations are very limited, and many of them cannot be easily corrected. Digital visualization systems allow for various correction methods, which may improve the digital image of the eye presented to the surgeon and other personnel assisting the ophthalmic surgery. However, current systems and methods may not utilize optimization algorithms, which may include machine learning models, to improve the quality of the digital image of the eye. Furthermore, current systems and methods may not utilize identifying markers to catalog the digital image of the eye for use in the optimization algorithm. In such cases, the image quality of the digital image of the eye may be degraded. For example, the digital image of the eye may have reduced contrast, sharpness, clarity, and dynamic range, and may exhibit increased noise, distortion, and vignetting. Degraded image quality may increase the risk of complications during ophthalmic surgery.

[0029] The disclosed digital image optimization systems and methods may improve digital visualization in ophthalmic surgery, thereby providing faster, safer, and more efficient medical procedures. The disclosed digital image optimization systems and methods may improve digital visualization in ophthalmic surgery by providing optimized digital images with reduced instrument glare, optical aberrations, vitreous opacities, or any combination thereof, compared to digital images provided by other systems and methods. The disclosed digital image optimization systems and methods may improve digital visualization in ophthalmic surgery, compared to current systems and methods, by providing a retinal arcade alignment template for intraoperative retinal attachment alignment. The disclosed digital image optimization systems and methods may improve digital visualization in ophthalmic surgery, compared to current systems and methods, by providing optimized digital images that are super-resolution digital images. The disclosed systems and methods may improve digital visualization in ophthalmic surgery, compared to current systems and methods, by including a machine learning model to optimize digital images of the eye. The disclosed digital image optimization systems and methods may improve digital visualization in ophthalmic surgery by providing training images to train a machine learning model. The training images may be multiple digital images of multiple eyes from multiple medical procedures. The digital image optimization systems and methods disclosed herein may improve digital visualization in ophthalmic surgery by providing a cloud-based digital image optimization service. The digital image optimization systems and methods disclosed herein may improve digital visualization in ophthalmic surgery by providing a digital timeout. The digital timeout may include a computational check that medical information associated with the patient matches information provided to the surgeon, operating room staff, or a combination thereof.

[0030] The systems and methods disclosed herein may improve digital visualization in ophthalmic surgery by providing a digital image optimization system that can improve the contrast, sharpness, clarity, and dynamic range of digital images of the eye and reduce noise, distortion, and vignetting in digital images of the eye. The digital image optimization system may be a component of a Digitally Assisted Vitreoretinal Surgery ("DAVS") system or the NGENUITY® 3D Visualization System (Novartis AG Corp., Switzerland). For example, digital images of the eye may be captured using a camera or an endoscope. The systems and methods disclosed herein may optimize digital images of the eye using machine learning models. The systems and methods disclosed herein may use a retinal arcade alignment template to align digital images of the same eye captured at different stages during surgery and provide optimized digital images for retinal reattachment following vitrectomy. The systems and methods disclosed herein may also provide an optimization network that enables cloud-based digital image optimization services. The optimization network may be a component of the Digitally Assisted Vitreoretinal Surgery ("DAVS") system or may be a component of the NGENUITY® 3D Visualization System (Novartis AG Corp., Switzerland). The systems and methods disclosed herein may also provide a digital timeout system that includes machine-readable information about digital images of the eye. The machine-readable information may be useful for record-keeping for storing digital images of the eye in the optimization network and for selecting training images for training machine learning models. The digital timeout system may be a component of the Digitally Assisted Vitreoretinal Surgery ("DAVS") system or may be a component of the NGENUITY® 3D Visualization System (Novartis AG Corp., Switzerland).

[0031] 1 , digital image optimization system 100 may include camera 150, image processing system 170, and digital display 190. Image processing system 170 may provide an optimized digital image of eye 101. Camera 150 may capture a digital image of eye 101. The digital image of eye 101 captured by camera 150 may include a field of view 160. Field of view 160 may include a magnified view of eye 101, or may include a high-magnification view of the macula, vitreous humor, or other areas of eye 101. Field of view 160 may also include a surgical instrument 105. Surgical instrument 105 may be a cannula or another surgical tool used in ophthalmic surgery.

[0032] The camera 150 may be a digital camera, an HDR camera, a 3D camera, or any combination thereof. The camera 150 may be a component of the NGENUITY® 3D visualization system (Novartis AG Corp., Switzerland). The camera 150 may also be a camera coupled to a microscope. The camera 150 may replace the eyepiece of a microscope and may be a fifth-generation Image Capture Module (ICM5) 3D surgical camera. The camera 150 may be configured to provide a stereoscopic digital image of the eye 101 (not shown). The camera 150 may include a lens 121. The lens 121 may also be an optomechanical focus lens, a manual focus lens, or any combination thereof. The camera 150 may include at least one image sensor 152, which may be a charge-coupled device (CCD) sensor or a complementary metal-oxide semiconductor (CMOS) sensor. The camera 150 may be a monochrome camera or a color camera, and the at least one image sensor 152 may be a monochrome image sensor or a color image sensor. The at least one image sensor 152 may be an image sensor with a color filter array, such as a Bayer filter, or may be an image sensor without a color filter array.

[0033] The digital image optimization system 100 may include a visible light illumination source 155. The visible light illumination source 155 may be the visible light illumination source of the camera 150. The visible light illumination source 155 may be an endoilluminator. The visible light illumination source 155 may include a xenon source, a white LED light source, or any other suitable visible light source. The visible light illumination source 155 may illuminate the internal structures of the eye 101.

[0034] Light emitted by the visible light illumination source 155 and reflected from the surgical instrument 105 can cause instrument glare 110. Instrument glare 110 can also be caused by light emitted by other illumination sources. Instrument glare 110 can appear as noise or distortion in the digital image of the eye 101 captured by the camera 150.

[0035] Light emitted by visible light illumination source 155 and reflected from the interior of eye 101 may also be affected by optical aberrations 120. Optical aberrations 120 may be caused by corneal opacity, which may be caused by extended medical procedures or ocular defects, or lens opacity, which may be caused by cataracts or intraocular blood. Optical aberrations 120 may also be caused by other characteristics of eye 101, the surgical setup, or a combination thereof. Optical aberrations 120 may reduce the contrast, sharpness, clarity, and dynamic range of the digital image of eye 101 captured by camera 150.

[0036] Light emitted by the visible light illumination source 155 and reflected from the eye 101 may be affected by vitreous opacification 130, which may appear as opacity or fog in a digital image of the eye 101 captured by the camera 150, for example.

[0037] Digital images captured by camera 150 may be processed by image processing system 170. Image processing system 170 may include processor 180. Camera 150 may detect light reflected from within eye 101 into lens 121 using at least one image sensor 152 and may send a signal corresponding to the detected light to processor 180. Processor 180 may execute instructions to generate digital images of eye 101. Image processing system 170 may also include memory medium 181. Digital images of eye 101 may be stored in memory medium 181.

[0038] The processor 180 may execute instructions to apply a digital image optimization algorithm to the digital image of the eye 101 to generate an optimized digital image 191 of the eye. The digital image optimization algorithm may be an algorithm that is a trained machine learning model. The machine learning model, as discussed, may allow the development of a digital image optimization algorithm that learns and evolves from experience without being explicitly programmed. In one example, the machine learning model may be trained using a “red boost” library of digital images from past surgeries as training images. The red boost library may include images of a poor red reflex of the eye and images of a bright red reflex of the eye. By training with the red boost library, the machine learning model may be able to predict an optimized digital image 191 of the eye. The digital image optimization algorithm may alternatively be an algorithm that utilizes another image enhancement technique, such as interpolation or multi-exposure image noise reduction.

[0039] The digital image optimization algorithm may at least partially reduce instrument glare 110, optical aberrations 120, vitreous opacities 130, or any combination thereof, in the digital image of the eye 101 captured by the camera 150 to produce an optimized digital image of the eye 191. The optimized digital image of the eye 191 may have less instrument glare 110, optical aberrations 120, vitreous opacities 130, or any combination thereof, than a digital image of the eye 101 captured by the camera 150 to which the digital optimization algorithm has not been applied.

[0040] Processor 180 may include, for example, a field programmable gate array (FPGA), a microprocessor, a microcontroller, a digital signal processor (DSP), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or any other digital or analog circuit configured to interpret and / or execute program instructions and / or process data.

[0041] Processor 180 may include any physical device capable of storing and / or executing instructions. Processor 180 may execute processor instructions to implement at least a portion of one or more systems, one or more flowcharts, one or more processes, and / or one or more methods described herein. For example, processor 180 may execute instructions to generate a digital image of eye 101. Processor 180 may be configured to receive instructions from memory medium 181. In one example, processor 180 may include memory medium 181. In another example, memory medium 181 may be external to processor 180. The memory medium may store instructions. The instructions stored by memory medium 181 may be executable by processor 180 and may be configured, coded, and / or encoded by instructions according to at least a portion of one or more systems, one or more flowcharts, one or more methods, and / or one or more processes described herein. The memory medium 181 may store instructions that are executable by the processor 180 and that may apply a digital image optimization algorithm to a digital image of the eye 101 captured by the camera 150 .

[0042] The FPGA may be configured, coded, and / or encoded to implement at least a portion of one or more systems, one or more flowcharts, one or more processes, and / or one or more methods described herein. For example, the FPGA may be configured, coded, and / or encoded to generate a digital image of the eye 101. The ASIC may be configured to implement at least a portion of one or more systems, one or more flowcharts, one or more processes, and / or one or more methods described herein. For example, the ASIC may be configured, coded, and / or encoded to generate a digital image of the eye 101. The DSP may be configured, coded, and / or encoded to implement at least a portion of one or more systems, one or more flowcharts, one or more processes, and / or one or more methods described herein. For example, the DSP may be configured, coded, and / or encoded to generate a digital image of the eye 101.

[0043] A single device may include processor 180 and image processing system 170, or processor 180 may be separate from image processing system 170. In one example, a single computer system may include processor 180 and image processing system 170. In another example, a device may include an integrated circuit that may include processor 180 and image processing system 170. Alternatively, processor 180 and image processing system 170 may be incorporated into a surgical console.

[0044] Processor 180 may interpret and / or execute program instructions and / or process data stored on memory medium 181. Memory medium 181 may be configured, in part or in whole, as application memory, system memory, or both. Memory medium 181 may include any system, device, or apparatus configured to hold and / or accommodate one or more memory devices. Each memory device may include any system, module, or apparatus (e.g., computer-readable medium) configured to retain program instructions and / or data for a period of time. One or more of the servers, electronic devices, or other machines described may include one or more similar such processors or memories capable of storing and executing program instructions to perform the functions of the associated machine.

[0045] The digital image optimization system 100 may include a digital display 190. The digital display 190 may include any type of screen or projector capable of displaying a digital image of the eye 101 at sufficient resolution to be usable in ophthalmic surgery. For example, it may include any type of screen or projector used in connection with ophthalmic surgery, including types of displays used in conventional vitreoretinal surgical systems that present digital images. The digital display 190 may display an optimized digital image of the eye 191. This may improve digital visualization in ophthalmic surgery by providing a digital image of the eye that has less instrument glare 110, optical aberrations 120, vitreous opacity 130, or any combination thereof, than a non-optimized digital image of the eye 101. The digital display 190 may display a digital image that is a combined digital image of the digital image of the eye 101 and the optimized digital image of the eye 191. The combined digital image may be generated by the processor 180. The proportion of the digital image of the eye 101 and the optimized digital image of the eye 191 displayed in the combined image may be controlled by the surgeon using, for example, a slider bar controlled by controller 192. In this way, the surgeon can control the extent to which the optimized digital image is used for visualization during surgery.

[0046] The digital display 190 can display a single image or two images for stereoscopic viewing. The digital display 190 can be a digital display, a screen, a head-up display, a head-mounted display, or any combination thereof, and can include multiple displays. The digital display 190 can be a flat-panel display or an ultra-high-definition 3D flat-panel display. The digital display 190 can be a 3D organic light-emitting diode (OLED) surgical display. Images displayed on the digital display 190 can be viewed through passive circularly polarized glasses. The digital display 190 can be a picture-in-picture display. The digital display 190 can be a component of a Digitally Assisted Vitreoretinal Surgery ("DAVS") system or the NGENUITY® 3D Visualization System (Novartis AG Corp., Switzerland).

[0047] Digital display 190 may display optimized digital images 191 of the eye generated by processor 180 or another processor, as well as other information generated by processor 180 or another processor. Such information may include graphical or textual information such as surgical parameters, surgical modes, flow rates, intraocular pressure, endoscopic video, OCT images, warnings, digital images, color coding, augmented reality information, etc. Processor 180 may reformat video produced using camera 150 for display on digital display 190, which may be viewed using circularly polarized glasses, digital eyepieces, or a head-mounted display.

[0048] The digital optimization algorithm may be generated using a machine learning model. The machine learning model, as discussed, may be trained using multiple training images. The training images may include a digital image captured at the beginning of surgery and a digital image captured at the end of surgery, as shown in FIGS. 2A-2D . The digital image captured at the beginning of surgery may provide the worst visualization of the eye. This may be because the digital image includes instrument glare 110, optical aberrations 120, vitreous opacities 130, or any combination thereof. A digital image of the same eye captured at the end of surgery may provide the best visualization of the eye. This may be because many of the sources of noise or distortion, such as instrument glare 110, optical aberrations 120, vitreous opacities 130, or any combination thereof, have disappeared. Thus, by training a machine learning model using multiple digital images captured at the beginning of surgery and multiple corresponding digital images of the eye captured at the end of surgery across multiple eyes, a digital optimization algorithm may be generated that predicts a digital image to be captured at the end of surgery from an input digital image captured at the beginning of surgery. If necessary, the digital image of the eye captured at the beginning of surgery may be registered with the digital image of the eye captured at the end of surgery using the optic disc, the retinal arcade, or a combination thereof, which may provide a reference for aligning and comparing the images.

[0049] For example, the digital image of the eye 201 at the start of surgery may be a digital image of the eye 101 captured by the camera 150 at the start of surgery, as shown in FIG. 2A . The digital image of the eye 201 at the start of surgery may include instrument glare 110 caused by, for example, light emitted by the visible light illumination source 155 and reflected from the surgical instrument 105. The instrument glare 110 may appear as noise or distortion in the digital image of the eye 201 at the start of surgery. The digital image of the eye 201 at the start of surgery may also include optical aberrations 120 caused by light emitted by the visible light illumination source 155 and reflected from the interior of the eye 101. The optical aberrations 120 may be caused by, for example, a cloudy lens 220. The optical aberrations 120 may reduce the contrast, sharpness, clarity, and dynamic range of the digital image of the eye 201 at the start of surgery.

[0050] The digital image of the eye 202 at the end of surgery may be a digital image of the eye 101 captured by the camera 150 at the end of surgery, as shown in FIG. 2B . At the end of surgery, the surgical instruments 105 may have been at least partially removed from the eye 101. Thus, the digital image of the eye 202 at the end of surgery may have less or no instrument glare 110. At the end of surgery, the opacified lens 220 may have been at least partially replaced with an intraocular lens 221. Thus, the digital image of the eye 202 at the end of surgery may have less or no optical aberrations 120. The digital image of the eye 202 at the end of surgery may have improved contrast, sharpness, clarity, and dynamic range, which in turn may result in less noise, distortion, and vignetting, compared to the digital image of the eye at the beginning of surgery.

[0051] In another example, the digital image of the eye at the start of surgery 203 may be a digital image of the eye 101 captured by the camera 150 at the start of surgery, as shown in FIG. 2C . The digital image of the eye at the start of surgery 203 may include vitreous opacities 130 caused by light emitted by the visible light illumination source 155 and reflected from the vitreous body 210. The vitreous opacities 130 may appear as noise or distortion in the digital image of the eye at the start of surgery 203. The digital image of the eye at the start of surgery 203 may also include optical aberrations 120 caused by light emitted by the visible light illumination source 155 and reflected from the interior of the eye 101. The optical aberrations 120 may be caused by blood 211. The optical aberrations 120 may reduce the contrast, sharpness, clarity, and dynamic range of the digital image of the eye at the start of surgery 203.

[0052] Digital image of eye 204 at the end of surgery may be a digital image of eye 101 captured by camera 150 at the end of surgery, as shown in FIG. 2D . At the end of surgery, vitreous body 210 may have been at least partially removed from eye 101. Thus, digital image of eye 204 at the end of surgery may have less or no vitreous opacities 130. At the end of surgery, blood 211 may also have been at least partially removed. Thus, digital image of eye 204 at the end of surgery may have less or no optical aberrations 120. Digital image of eye 204 at the end of surgery may have improved contrast, sharpness, clarity, and dynamic range, resulting in less noise, distortion, and vignetting, compared to digital image of eye 203 at the beginning of surgery.

[0053] Multiple digital images of an eye at the start of surgery and multiple digital images of the same eye at the end of surgery may be used as input and output images, respectively, in a set of training images for training a machine learning model. For example, the input image may be digital image 201 of the eye at the start of surgery, and the output image may be digital image 202 of the eye at the end of surgery. The machine learning model may be trained based on extracting features from the input images and learning relationship functions between the extracted features and differences between the input and output images, as discussed. The trained machine learning model may be used as a digital image optimization algorithm.

[0054] The digital image optimization algorithm may be executed by processor 180 using memory medium 181 to generate processor instructions to optimize a digital image of the eye at the start of surgery, e.g., digital image of the eye at the start of surgery 201 or digital image of the eye at the start of surgery 203. Processor 180 may execute the instructions to generate optimized digital image of the eye 191. Optimized digital image of the eye 191 may have improved image quality comparable to a digital image of the eye at the end of surgery, e.g., digital image of the eye at the end of surgery 202 or digital image of the eye at the end of surgery 204. Optimized digital image of the eye 191 may have improved contrast, improved sharpness, improved clarity, improved dynamic range, reduced noise, reduced distortion, reduced vignetting, or any combination thereof, compared to the digital image of the eye 101.

[0055] FIG. 3 presents a flowchart of a method for optimizing a digital image of an eye to improve digital visualization in ophthalmic surgery. In step 300, a digital image of the eye is captured. The digital image of the eye may be captured by a camera, such as camera 150. The digital image of the eye may be captured at the start of surgery and may be a digital image of the eye at the start of surgery, such as digital image of the eye 201, or the digital image of the eye may be captured intraoperatively. The digital image of the eye may have instrument glare, such as instrument glare 110, optical aberrations, such as optical aberration 120, vitreous opacities, such as vitreous opacities 130, or any combination thereof. In step 310, a digital image optimization algorithm is applied to the digital image of the eye to generate an optimized digital image of the eye, such as optimized digital image of the eye 191. The digital image optimization algorithm may be a trained machine learning model. In step 320, the optimized digital image of the eye is displayed on a digital display, such as digital display 190.

[0056] FIG. 4 presents a flowchart of a method for generating a digital image optimization algorithm to improve digital visualization in ophthalmic surgery using a machine learning model. In step 400, an input digital image of an eye is captured at the beginning of surgery, such as digital image 201 of the eye captured by camera 150 at the beginning of surgery. In step 410, an output digital image of the same eye is captured at the end of surgery, such as digital image 202 of the eye captured by camera 150 at the end of surgery. In step 420, the input digital image of the eye at the beginning of surgery and the output digital image of the eye captured at the end of surgery are used as input and output images, respectively, in a set of training images to train a machine learning model. In step 430, steps 400-420 are repeated for multiple input and output images of multiple patients. These images may be collected from a wide range of surgeons and eye hospitals. Using the multiple input and output images, a machine learning model may be trained to predict an output image, such as a digital image of the eye at the end of surgery, when only an input image, such as a digital image of the eye at the beginning of surgery, is provided. In step 440, the trained machine learning model may be used as a digital image optimization algorithm to improve the digital image of the eye.

[0057] In an alternative example, the digital image optimization system 500 may improve digital visualization in ophthalmic surgery by providing a retinal arcade alignment template, as shown in FIG. 5 . The retinal arcade may include the retinal vasculature. The retinal arcade may include the temporal arcade canal. Aligning the retinal arcade may align digital images of the same eye captured at different stages during surgery. Alternatively, registering a digital mirror image of the retinal arcade of a fellow eye 502 (the unoperated, healthy eye) to a digital image of the operative eye 501 of the same patient may provide a retinal arcade alignment template during surgery. This may provide an optimized digital image of the operative eye. The fellow eye may be the right eye, and the operative eye may be the left eye of the same patient. Alternatively, the fellow eye may be the right eye, and the operative eye may be the left eye of the same patient.

[0058] For example, during vitrectomy, digital visualization can be improved by providing a retinal arcade alignment template for retinal reattachment using a registered digital image of the fellow eye's retinal arcade superimposed on a digital image of the surgical eye. This can, at least in part, prevent cyclo-distortion following vitrectomy and correct retinal detachment. The retinal arcade pattern of the fellow eye can best resemble the retinal arcade pattern of the surgical eye compared to any other eye. Alignment can be performed intraoperatively using a digital visualization system, as shown in FIG. 5. The visualization system can be a 2D or 3D visualization system.

[0059] The digital image optimization system 500 may include a camera 550, an image processing system 570, and a digital display 590. The surgical camera 550 may be a digital camera, an HDR camera, a 3D camera, or any combination thereof. The camera 550 may be a component of the NGENUITY® 3D visualization system (Novartis AG Corp., Switzerland). The camera 550 may also be a camera coupled to a microscope. The camera 550 may replace the eyepiece of a microscope and may be a fifth-generation Image Capture Module (ICM5) 3D surgical camera. The camera 550 may be configured to provide a stereoscopic digital image of the surgical eye 501 or the fellow eye 502 (not shown). The camera 550 may include a lens 521. The lens 521 may also be an optomechanical focus lens, a manual focus lens, or any combination thereof. The camera 550 may include at least one image sensor 552, which may be a charge-coupled device (CCD) sensor or a complementary metal-oxide semiconductor (CMOS) sensor. The camera 550 may be a monochrome camera or a color camera, and the at least one image sensor 552 may be a monochrome image sensor or a color image sensor. The at least one image sensor 552 may be an image sensor with a color filter array, such as a Bayer filter, or an image sensor without a color filter array. Alternatively, the image captured by the camera 550 may be an optical coherence tomography (OCT) image, a multi / hyperspectral image, an ultrasound image, or any combination thereof.

[0060] The digital image of the surgical eye 501 or fellow eye 502 captured by the camera 550 may include a field of view 560. The field of view 560 may include a magnified view of the surgical eye 501 or fellow eye 502, and may include a high-magnification view of the retinal arcade or other areas of the surgical eye 501 or fellow eye 502.

[0061] Preoperatively, the camera 550 may capture a template digital image 592 of the fundus of the fellow eye 502. The template digital image 592 may include a digital image 510 of the retina of the fellow eye, a digital image 512 of the central retinal vein of the fellow eye, a digital image 515 of the retinal arcade of the fellow eye, a digital image 517 of the optic nerve disc of the fellow eye, or any combination thereof. Intraoperatively, the camera 550 may capture a working digital image 591 of the fundus of the surgical eye 501. The working digital image 591 may have a field of view 560 of the surgical eye 501 equivalent to the field of view 560 of the template digital image 592 of the fellow eye 502. The working digital image 591 may include a digital image 509 of the retina of the surgical eye, a digital image 511 of the central retinal vein of the surgical eye, a digital image 514 of the retinal arcade of the surgical eye, a digital image 516 of the optic nerve disc of the surgical eye, or any combination thereof.

[0062] Digital images captured by the camera 550 may be processed by an image processing system 570. The image processing system 570 may include a processor 580. The camera 550 may use at least one image sensor 552 to detect light reflected into the lens 521 from within the surgical eye 501 or the fellow eye 502, and the image sensor 552 may send a signal corresponding to the detected light to the processor 580. The processor 580 may execute instructions to generate a working digital image 591 of the surgical eye 501, a template digital image 592 of the fellow eye 502, or a combination thereof. The image processing system 570 may also include a memory medium 581. The working digital image 591, the template digital image 592, or a combination thereof may be stored in the memory medium 581.

[0063] Processor 580 may execute instructions to create a digital mirror image 593 of template digital image 592. Processor 580 may execute instructions to align digital mirror image 593 with working digital image 591 to generate a retinal arcade alignment template 594. Processor 580 may execute instructions to align digital mirror image 593 with working digital image 591 by aligning the optic disc 517 of the fellow eye with the optic disc 516 of the patient eye, aligning the central retinal vein of the fellow eye with the central retinal vein of the patient eye, or a combination thereof. Processor 580 may also execute instructions to align digital mirror image 593 with working digital image 591 by aligning arterial-venous intersections (not shown), branch veins (not shown), or a combination thereof. Processor 580 may be similar to processor 180.

[0064] Digital display 590 may include any type of screen or projector capable of displaying a digital image of surgical eye 501 at sufficient resolution for use in ophthalmic surgery. For example, it may include any type of screen or projector used in connection with ophthalmic surgery, including displays of the types used in conventional vitreoretinal surgical systems that present digital images. Digital display 590 may be similar to digital display 190.

[0065] Digital display 590 may display working digital image 591, template digital image 592, digital mirror image 593, retinal arcade alignment template 594, or any combination thereof. Digital display 590 may display an overlay of working digital image 591 and retinal arcade alignment template 594. This may improve digital visualization in ophthalmic surgery by providing retinal arcade alignment template 594 for retinal reattachment. Working digital image 591 with the overlay of retinal arcade alignment template 594 may represent an optimized digital image of the eye compared to working digital image 591 alone. Digital display 590 may display a digital image that is a combined digital image of retinal arcade alignment template 594 and working digital image 591. The combined digital image may be generated by processor 580. The proportion of working digital image 591 and retinal arcade alignment template 594 displayed in the combined image may be controlled by the surgeon using, for example, a slider bar controlled by controller 595. In this way, the surgeon can control the extent to which the retinal arcade alignment template 594 is used for visualization during surgery.

[0066] The digital display 590 can display a single image or two images for stereoscopic viewing. The digital display 590 can be a digital display, a screen, a head-up display, a head-mounted display, or any combination thereof, and can include multiple displays. The digital display 590 can be a flat-panel display or an ultra-high-definition 3D flat-panel display. The digital display 590 can be a 3D organic light-emitting diode (OLED) surgical display. Images displayed on the digital display 590 can be viewed through passive circularly polarized glasses. The digital display 590 can be a component of a Digitally Assisted Vitreoretinal Surgery ("DAVS") system or the NGENUITY® 3D Visualization System (Novartis AG Corp., Switzerland).

[0067] Alternatively, digital display 590 may be a picture-in-picture display and may simultaneously display at least two digital images. For example, digital display 590 may simultaneously display working digital image 591 and retinal arcade alignment template 594. Working digital image 591 may be displayed as the main image, and retinal arcade alignment template 594 may be displayed in an inset position. Alternatively, working digital image 591 may be displayed as the main image, and working digital image 591 with an overlay of retinal arcade alignment template 594 may be displayed in an inset position.

[0068] Digital display 590 may display working digital image 591, template digital image 592, digital mirror image 593, retinal arcade alignment template 594, or any combination thereof, generated by processor 580 or another processor, as well as other information generated by processor 580 or another processor. Such information may include graphical or textual information such as surgical parameters, surgical mode, flow rate, intraocular pressure, endoscopic video, OCT images, warnings, digital images, color coding, augmented reality information, etc. Processor 580 may reformat video produced using camera 550 for display on digital display 590, viewable using circularly polarized glasses, digital eyepieces, or a head-mounted display.

[0069] The digital image optimization system 500 may be used to optimize a digital image of the eye after vitrectomy by providing a retinal arcade alignment template 594 for retinal reattachment. The image processing system 570 may detect any retinal misalignment as the retina is being reattached using the retinal arcade alignment template 594. The retinal misalignment may be adjusted using, for example, a soft-tip cannula. The image processing system 570 may also be used to quantify the degree of misalignment. The degree of misalignment may be quantified mathematically, graphically, or a combination thereof to inform the surgeon's assessment of retinal reattachment. The mathematical information, graphical information, or a combination thereof may be further used by the digital image optimization system 500 to suggest surgical alignment movements to the surgeon.

[0070] FIG. 6 presents a flowchart of a method for optimizing digital images of an eye to improve digital visualization in ophthalmic surgery by using a retinal arcade alignment template. In step 600, a preoperative digital image of the fellow eye is captured, such as template digital image 592. In step 610, an intraoperative digital image of the surgical eye is captured, such as working digital image 591. In step 620, a mirror image of the digital image of the fellow eye is generated, such as digital mirror image 593. In step 630, the mirror image of the digital image of the fellow eye is registered to the digital image of the surgical eye using features of the retinal arcade to generate a retinal arcade alignment template, such as retinal arcade alignment template 594. For example, the digital images may be registered by registering the optic disc, the central retinal vein, the arteriovenous intersection, the branch veins, or any combination thereof. The digital images may be registered in 2D or 3D. In step 640, the retinal arcade alignment template is displayed as an overlay on the digital image of the surgical eye. In step 650, eye surgery is performed on the surgical eye using the retinal arcade alignment template. One example of an eye procedure that would benefit from improved digital visualization using the retinal arcade alignment template can be retinal reattachment following vitrectomy.

[0071] In another alternative example, a digital image optimization system 700 may improve digital images in ophthalmic surgery by providing super-resolution optimization of endoscopic digital images of the eye, as shown in Figure 7. This may improve digital visualization in ophthalmic surgery by increasing the resolution of low-resolution digital images, for example, endoscopic digital images of about 30,000 pixels, so that the low-resolution digital images can be displayed with improved image quality on a high-resolution digital display, for example, a digital display with over 1 million pixels.

[0072] 7, digital image optimization system 700 may include endoscope 715, optical fiber 740, fiber optic light source 741, camera 750, image processing system 770, and digital display 790. Image processing system 770 may further include processor 780 and memory medium 781. Digital image optimization system 700 may provide optimized endoscopic digital images 791 of eye 101 to improve digital visualization in ophthalmic surgery.

[0073] The endoscope 715 may be inserted into the eye 101. The endoscope 715 may be positioned such that a desired field of view 760 of the interior of the eye 101 is captured in the optimized endoscopic digital image 791. The optical fiber 740 may be positioned within the endoscope 715 and may extend to the tip of the endoscope 715. The optical fiber 740 may include approximately 30,000 image fibers. Alternatively, the optical fiber 740 may include any suitable number of image fibers to provide the desired optimized endoscopic digital image 791 of the eye 101 using the endoscope 715.

[0074] The optical fiber 740 may be coupled to a fiber optic light source 741. The fiber optic light source 741 may be a laser source, a narrowband laser source, a broadband laser source, a supercontinuum laser source, an incandescent lamp, a halogen lamp, a metal halide lamp, a xenon lamp, a mercury vapor lamp, a light emitting diode (LED), a laser engine, other suitable sources, or any combination thereof. Light reflected from the interior of the eye 101 may propagate through an image fiber in the optical fiber 740 and may be detected by a camera 750. The digital image optimization system 700 may also include an eyepiece of the endoscope 715 in addition to the camera 750 (not shown).

[0075] The camera 750 may include at least one camera sensor 752. The at least one camera sensor 752 may be a complementary metal-oxide semiconductor (CMOS) sensor or a charge-coupled device (CCD) sensor. The camera 750 may be a monochrome camera or a color camera, and the at least one camera sensor 752 may be a monochrome image sensor or a color image sensor. The at least one camera sensor 752 may capture digital images using light propagating through the optical fiber 740, which may be light reflected from the interior of the eye 101. The at least one camera sensor 752 may capture digital images of the eye 101, which may be endoscopic digital images of the eye 101.

[0076] The optical fiber 740, the optical fiber light source 741, the camera 750, and the at least one camera sensor 752 may be controlled by a control device 742. For example, the control device 742 may adjust the intensity of the optical fiber light source 741, the sensitivity of the at least one camera sensor 752, or any combination thereof. Although FIG. 7 shows one endoscope 715 in the digital image optimization system 700, the digital image optimization system 700 may include multiple endoscopes 715, optical fibers 740, cameras 750, and camera sensors 752. In this case, multiple endoscopes 715 may be inserted into multiple locations in the eye 101 to provide multiple endoscopic digital images of the eye.

[0077] The digital image optimization system 700 may include an image processing system 770. Digital images captured by the at least one camera sensor 752 may be processed by the image processing system 770. The image processing system 770 may include a processor 780. The camera 750 may use the at least one camera image sensor 752 to detect light reflected from the interior of the eye 101 and propagated by the optical fiber 740 and send a signal corresponding to the detected light to the processor 780. The processor 780 may execute instructions to generate an endoscopic digital image of the eye 101.

[0078] The endoscopic digital image of the eye 101 captured by the camera 750 may be processed by an image processing system 770. The image processing system 770 may also include a memory medium 781. The endoscopic digital image of the eye 101 may be stored in the memory medium 781. The processor 780 may execute instructions to apply an endoscopic digital image optimization algorithm to the endoscopic digital image of the eye 101 to generate an optimized endoscopic digital image of the eye 791. The optimized endoscopic digital image of the eye 791 may be a super-resolution digital image of the eye. The super-resolution digital image may be generated by upscaling a low-resolution endoscopic digital image of the eye to generate a high-resolution endoscopic digital image of the eye. As will be described, examples of methods that may be used to generate the optimized endoscopic digital image of the eye 791 include, but are not limited to, interpolation, multiple-exposure image noise reduction, machine learning, deep learning, or any combination thereof. For example, using interpolation methods may generate the optimized endoscopic digital image of the eye 791 more quickly than using machine learning methods. For example, machine learning methods can be used to generate optimized endoscopic digital images 791 of the eye that are improved over those that are obtained using interpolation methods.

[0079] The endoscopic digital image optimization algorithm may use interpolation to generate the optimized endoscopic digital image 791 of the eye. For example, the endoscopic digital image optimization algorithm may generate the optimized endoscopic digital image 791 of the eye by replacing low-quality target pixels in the endoscopic digital image of the eye 101 with neighboring pixels that have higher contrast, sharpness, clarity, or any combination thereof. The low-quality target pixels may be pixels that do not display an expected color, pixels that permanently display a particular color, permanently white pixels, permanently black pixels, pixels that permanently display as "hot pixels," dead pixels, or any combination thereof. The target pixels may be modified to optimize pixel image quality without any noticeable image distortion, provided that the area being modified is generally small. Examples of interpolation methods that may be included in the endoscopic digital image optimization algorithm may include, but are not limited to, nearest neighbor interpolation, linear interpolation, bilinear interpolation, bicubic interpolation, anisotropic filtering, or any combination thereof.

[0080] In another example, the endoscopic digital image optimization algorithm may use multiple exposure image noise reduction to generate an optimized endoscopic digital image 791 of the eye. The endoscope 715 may be dynamically scanned by the surgeon over the contrast region to combine multiple endoscopic digital images of the eye 101 to provide a single optimized endoscopic digital image 791 of the eye.

[0081] The endoscopic digital image optimization algorithm may be trained using a machine learning model. The machine learning model may be trained using a plurality of training images, as discussed, which may be a plurality of endoscopic digital images of an eye that have been successfully optimized using an image processing method, such as one of the image processing methods described above. For example, the optimization of an endoscopic digital image of an eye may be considered successful if a surgeon successfully completes surgery using the optimized endoscopic digital image of the eye.

[0082] In a further example, the endoscopic digital image optimization algorithm may use deep learning methods to generate optimized endoscopic digital images 791 of the eye. For example, the endoscopic digital image optimization algorithm may utilize single image super-resolution (SISR) using deep learning. SISR may be a process of generating a high-resolution digital image from a single low-resolution digital image. In one example, a very deep super-resolution (VDSR) neural network may be used for SISR. The VDSR neural network may learn a mapping between low-resolution digital images and high-resolution digital images. In general, low-resolution digital images and high-resolution digital images of the same object may have similar image content. The low-resolution digital image may differ from the high-resolution digital image in high-frequency details. The difference in pixel values ​​between a high-resolution image of the same object and a low-resolution image upscaled to match the size of the high-resolution image may be referred to as a residual image. The VDSR neural network may be trained to estimate the residual image from a set of training images of pairs of low-resolution and high-resolution endoscopic digital images of the eye. A high-resolution endoscopic digital image of the eye may be reconstructed from the low-resolution endoscopic digital image of the eye by adding the estimated residual image to the upscaled low-resolution endoscopic digital image of the eye. By using the VDSR neural network, SISR may restore a high-resolution endoscopic digital image of the eye 101 from the low-resolution endoscopic digital image of the eye 101. The high-resolution endoscopic digital image of the eye 101 may be an optimized endoscopic digital image of the eye 791. Alternatively, other suitable neural networks may be used by the endoscopic digital image optimization algorithm to perform SISR.

[0083] The endoscopic digital image optimization algorithm may, at least in part, reduce noise, increase resolution, improve digital quality, or any combination thereof, of the endoscopic digital image of the eye 101 captured by the camera 750. For example, the optimized endoscopic digital image of the eye 791 may have increased contrast, sharpness, clarity, dynamic range, or any combination thereof, compared to the endoscopic digital image of the eye 101 captured by the camera 750 to which the endoscopic digital optimization algorithm was not applied. Although the digital image optimization system 700 includes an endoscope 715, super-resolution digital images of the eye may equally be provided for digital images of the eye captured using a camera, such as camera 150 or camera 550.

[0084] The optimized endoscopic digital image 791 may be displayed on a digital display 790. The digital display 790 may include any type of screen or projector capable of displaying the optimized endoscopic digital image 791 at sufficient resolution for use in ophthalmic surgery. For example, it may include any type of screen or projector used in connection with ophthalmic surgery, including types of displays used in conventional vitreoretinal surgical systems that present digital images. The digital display 790 may simultaneously display the optimized endoscopic digital image 791 with another digital image of the eye 101, such as the optimized digital image 191 of the eye. The digital display 790 may display a digital image that is a combined digital image of the endoscopic digital image of the eye 101 and the optimized endoscopic digital image 791. The combined digital image may be generated by the processor 780. The proportion of the endoscopic digital image of the eye 101 and the optimized endoscopic digital image 791 of the eye displayed in the combined image may be controlled by the surgeon using, for example, a slider bar controlled by a controller 792. In this way, the surgeon can control the extent to which the optimized endoscopic digital image 791 is used for visualization during surgery.

[0085] The digital display 790 can display a single image or two images for stereoscopic viewing. The digital display 790 can be a digital display, a screen, a head-up display, a head-mounted display, or any combination thereof, and can include multiple displays. The digital display 790 can be a flat-panel display or an ultra-high-definition 3D flat-panel display. The digital display 790 can be a 3D organic light-emitting diode (OLED) surgical display. Images displayed on the digital display 790 can be viewed through passive circularly polarized glasses. The digital display 790 can be a component of a Digitally Assisted Vitreoretinal Surgery ("DAVS") system or the NGENUITY® 3D Visualization System (Novartis AG Corp., Switzerland).

[0086] Alternatively, the digital display 790 may be a picture-in-picture display and may display at least two digital images simultaneously. For example, the digital display 790 may simultaneously display an optimized endoscopic digital image 791 and another digital image of the eye 101, such as an optimized digital image of the eye 191. The optimized endoscopic digital image 791 may be displayed as a main image, and the other digital image of the eye 101 may be displayed in an inset position. Alternatively, the other digital image of the eye 101 may be displayed as a main image, and the optimized endoscopic digital image 791 may be displayed in an inset position.

[0087] Digital display 790 may display optimized endoscopic digital images 791 generated by processor 780 or another processor, as well as other information generated by processor 780 or another processor. Such information may include graphical or textual information such as surgical parameters, surgical modes, flow rates, intraocular pressure, endoscopic video, OCT images, warnings, digital images, color coding, augmented reality information, etc. Processor 780 may reformat video produced using camera 750 for display on digital display 790, viewable using circularly polarized glasses, digital eyepieces, or a head-mounted display.

[0088] 8 presents a flowchart of a method for optimizing an endoscopic digital image of an eye to improve digital visualization in ophthalmic surgery by using an endoscopic digital image optimization algorithm. In step 800, an endoscopic digital image of a patient's eye is captured using an endoscope, such as endoscope 715. In step 810, an endoscopic digital image optimization algorithm is applied to the endoscopic digital image of the eye to generate an optimized endoscopic digital image of the eye, such as optimized endoscopic digital image of the eye 791. The endoscopic digital image optimization algorithm may use interpolation, multi-exposure image noise reduction, machine learning, deep learning, or any combination thereof. In step 820, the optimized endoscopic digital image of the eye is displayed on a digital display, such as digital display 790.

[0089] FIG. 9 presents a flowchart of a method for generating an endoscopic digital image optimization algorithm using a machine learning model to improve digital visualization in ophthalmic surgery. In step 900, an endoscopic digital image of a patient's eye is captured, such as an endoscopic digital image of eye 101 captured by endoscope 715. In step 910, the endoscopic digital image of the eye is optimized using an image processing method, such as interpolation, multi-exposure image noise reduction, or a combination thereof, to generate an optimized endoscopic digital image of the eye, such as optimized endoscopic digital image of the eye 791. The optimization of the endoscopic digital image of the eye may be deemed successful if the surgeon successfully completes surgery using the optimized endoscopic digital image of the eye. In step 920, the endoscopic digital image of the eye and the optimized endoscopic digital image of the eye are used as input and output images, respectively, in a set of training images for training a machine learning model. In step 930, steps 900-920 are repeated for multiple input and output images of multiple endoscopic digital images of multiple patients. These images may be collected from a wide range of surgeons and eye clinics. The multiple input images and output images may be used to train a machine learning model to predict an output image, e.g., an optimized endoscopic digital image of an eye, when only an input image, e.g., an endoscopic digital image of an eye, is provided. In step 940, the trained machine learning model may be used as an endoscopic digital image optimization algorithm to improve the endoscopic digital image of the eye.

[0090] Digital image optimization system 100, digital image optimization system 500, and digital image optimization system 700 may be used in conjunction with optimization network 1000, as shown in FIG. 10 . Optimization network 1000 may further include a communications network 1010 and a digital image management system 1015. FIG. 10 shows an example of generating an optimized digital image of an eye 1191 from a digital image of eye 101 using digital image management system 1015. While FIG. 10 includes camera 150 in optimization network 1000, optimization network 1000 may alternatively include an endoscope, such as endoscope 715, or any other suitable ophthalmic visualization system. Digital image management system 1015 may communicate with a local image processing system, such as image processing system 170, via communications network 1010. Alternatively, digital image management system 1015 may be part of or include a local image processing system, such as image processing system 170.

[0091] The digital image management system 1015 may train the machine learning model using multiple training images of the eye 101. For example, the digital image management system 1015 may train the machine learning model offline using multiple digital images of the eye captured at the beginning of surgery and multiple paired digital images of the same eye captured at the end of surgery, e.g., digital image of the eye at the beginning of surgery 201 and digital image of the eye at the end of surgery 202. Alternatively, the digital image management system 1015 may train the machine learning model offline using multiple endoscopic digital images of the eye that have been successfully optimized using image processing methods. The training images used to train the machine learning model may be uploaded to the digital image management system 1015 via the communication network 1010. The images used to train the machine learning model may be captured using a digitally assisted vitreoretinal surgery system, e.g., the NGENUITY® 3D visualization system (Novartis AG Corp., Switzerland). The images used to train the machine learning model may be captured by multiple surgeons. The images used to train the machine learning model may be images of multiple patients at multiple locations. The images used to train the machine learning model may be images of a similar resolution, images captured with a similar camera, images captured during a similar medical procedure, or any combination thereof.

[0092] During surgery, the digital image management system 1015 may process the digital image of the eye 101 using the trained machine learning model to generate an optimized digital image of the eye 1191. The optimized digital image of the eye 1191 may have lower instrument glare, lower optical aberrations, lower vitreous opacity, or any combination thereof, than a digital image of the eye captured by the camera 150 that has not been processed by the digital image management system 1015. The optimized digital image of the eye 1191 may have higher contrast, sharpness, clarity, dynamic range, or any combination thereof, than a digital image of the eye 101 captured by the camera 150 that has not been processed by the digital image management system 1015.

[0093] FIG. 11 illustrates an example of a solution for identifying an optimized digital image of an eye, which may be optimized digital image of an eye 1191. The digital image management system 1015 may include an image processing system 1170, which may further include a processor 1180, a memory medium 1181, and a machine learning model 1025. The image processing system 1170 may identify an optimized digital image of the eye from the digital image of the eye 101 by calculating a difference based on information in the digital image of the eye 101 and combining the difference with the digital image of the eye 101. The difference may be calculated based on raw pixel data of the digital image of the eye 101. For example, the digital image of the eye 101 may be captured by the digital image optimization system 100 at the start of surgery. The digital image of the eye 101 may include instrument glare, optical aberrations, vitreous opacity, or any combination thereof. The image processing system 1170 may calculate the difference based on raw pixel data of the digital image of the eye 101. The difference may be calculated using the machine learning model 1025. The image processing system 1170 may further identify an optimized digital image of the eye by combining the digital image of the eye 101 with the difference.

[0094] 11 , digital image optimization system 100 may capture a digital image of eye 101 and transmit it to image processing system 1170. Digital image optimization system 100 may transmit the digital image of eye 101 to image processing system 1170 using a communications network, such as communications network 1010. Image processing system 1170 may be included in digital image management system 1015. Thus, image processing system 1170 may process digital images of the eye captured from a remote ophthalmic visualization system to generate optimized digital images. Alternatively, once trained, image processing system 1170 may be stored locally to the ophthalmic visualization system, for example, in image processing system 170 within digital image optimization system 100. In this example, image processing system 1170 may locally optimize the digital image of eye 101.

[0095] The image processing system 1170 may use the machine learning model 1025 to calculate the difference between the digital images of the eye 101. The machine learning model 1025 may be trained using multiple training images 1030, as shown in FIG. 11 . Multiple training images 1030 of the eye 1101 may be acquired. Each set of training images 1030 includes an input image, which may be, for example, a digital image 201 of the eye at the start of surgery, and an output image, which may be, for example, a digital image 202 of the eye at the end of surgery. The image processing system 1170 may access the training images 1030 to train the machine learning model 1025. The machine learning model 1025 may be trained based on (1) extracting features from the input image and (2) learning a relationship function between the extracted features and the difference between the input image and the output image. Thus, if only an input image is provided, the machine learning model 1025 may be used to predict an output image. Alternatively, any suitable training images 1030 may be acquired in any suitable manner to train the machine learning model 1025 to optimize the digital images of the eye.

[0096] FIG. 12 illustrates an example integrated architecture 1200 of a machine learning model 1025 for calculating differences based on information in digital images of an eye, for example, calculating differences between a digital image of an eye 201 at the start of surgery and a digital image of an eye 202 at the end of surgery. The integrated architecture 1200 may include a convolutional neural network (CNN) 1210 and at least one long short-term memory unit (LSTM) 1220. The convolutional neural network 1210 may include a type of deep feedforward artificial neural network that may be effective for analyzing images. The convolutional neural network 1210 may use a variety of multilayer perceptrons, which require minimal preprocessing. Multilayer perceptrons are also known as shift-invariant or space-invariant artificial neural networks due to their weight-sharing architecture and translation-invariant properties. The convolutional neural network 1210 may include multiple successive layers. The at least one long-short-term memory unit 1220 may include a recurrent neural network, which may ultimately be used as a building block or hidden layer of a larger recurrent neural network. The long-short-term memory unit 1220 may itself be a recurrent network, as it may include recurrent connections similar to those in traditional recurrent neural networks. The convolutional neural network 1210 and the at least one long-short-term memory unit 1220 may be integrated. The integration may be achieved by inserting the at least one long-short-term memory unit 1220 after the last of multiple consecutive layers of the convolutional neural network 1210. The convolutional neural network 1210 may include a last consecutive layer that is a fully connected layer 1211. The at least one long-short-term memory unit 1220 may be inserted after the fully connected layer 1211. The at least one long-short-term memory unit 1220 may correspond to raw pixel data of a digital image of the eye, such as image width, image length, color channel data, or any combination thereof. The output of the at least one long-term short-term memory unit 1220 may be an output vector 1230 .The image processing system 1170 may use the output vector 1230 to calculate the difference based on information in the digital image of the eye. Alternatively, the image processing system 1170 may use any suitable architecture for training the machine learning model 1025.

[0097] The image processing system 1170 may generate feature representations of the plurality of training images 1030 by processing the plurality of training images 1030 using a convolutional neural network 1210. The convolutional neural network 1210 may be based on a pre-trained classification network. For example, the pre-trained classification network may include a residual network. The image processing system 1170 may implement the convolutional neural network 1210 by modifying the last fully connected layer 1211 of the pre-trained classification network. The image processing system 1170 may use the output of the convolutional neural network 1210 as the generated feature representations of the plurality of training images 1030. An example of a feature representation may be a numerical vector. The image processing system 1170 may also use the output of the convolutional neural network 1210 as input to at least one long-term short-term memory unit 1220. Alternatively, the feature representations may be generated using any suitable convolutional neural network in any suitable manner.

[0098] The image processing system 1170 may learn a relationship function between the feature representations of the plurality of training images 1030 and the differences between pairs of input and system images of the training images 1030. Learning the relationship function may be based on the machine learning integrated architecture 1200 and a particular loss function. The relationship function may include a mapping function. The input of the mapping function may include the feature representations of the input images and the differences between the input images and the corresponding output images. The output of the mapping function may include the differences between the predicted hypothetical optimized digital images of the eye and the corresponding output images. The differences may include values ​​corresponding to raw pixel data of the digital images, such as image width, image length, color channel data, or any combination thereof. The mapping function may further be based on a particular loss function. For example, the loss function may be a smoothed L1 loss function. For any variable x, the smoothed L1 loss function is

number

[0099] FIG. 13 presents a flowchart of a method for training a machine learning model, such as machine learning model 1025. In step 1300, an image processing system, such as image processing system 1170, may input a plurality of training images, such as training images 1030, to an integrated architecture, such as integrated architecture 1200. The plurality of training images may include a plurality of input images and a plurality of corresponding output images. The input image may be, for example, digital image 201 of the eye at the start of surgery, and the output image may be, for example, digital image 202 of the eye at the end of surgery. In step 1310, the image processing system may use a convolutional neural network, such as convolutional neural network 1210, to generate feature representations for a plurality of input images of the plurality of training images. In step 1320, the image processing system may use the integrated architecture to calculate a plurality of hypothetical differences between a plurality of predicted optimized digital images of the eye and corresponding output images. The plurality of hypothetical differences may be calculated using the feature representations of the plurality of corresponding input images. In step 1330, the image processing system may calculate a loss between the plurality of hypothetical differences and a plurality of differences calculated between the plurality of input images and the plurality of corresponding output images. The loss may be calculated using a loss function, which may include a smooth L1 loss function. In step 1340, the image processing system may evaluate whether the overall loss is less than a predefined threshold. If the overall loss is not less than the predefined threshold, the image processing system may proceed to step 1350. In step 1350, the image processing system may update the parameters of the integrated architecture. The cycle of steps 1310-1340 may then be repeated. If the overall loss is less than the predefined threshold, the image processing system may proceed to step 1360. In step 1360, the image processing system may store the parameters of the integrated architecture and terminate training. This may be because the machine learning model has converged. Alternatively, any suitable method of training a machine learning model to optimize the digital image of the eye may be used.

[0100] Image processing system 1170 may be trained in a system associated with digital image management system 1015. Once trained, image processing system 1170 may run on a server and be used for cloud-based digital image processing services. Alternatively, once trained, image processing system 1170 may be distributed to a local ophthalmic visualization system, which may include a local digital image optimization system such as digital image optimization system 100, digital image optimization system 500, digital image optimization system 700, or any combination thereof.

[0101] For example, the trained image processing system 1170 may receive a digital image of the eye. The digital image of the eye may be captured by a camera, such as camera 150, or may be captured by an endoscope, such as endoscope 715. The image processing system 1170 may compute an optimized digital image of the eye by processing the digital image of the eye using the trained machine learning model 1025. The image processing system 1170 may first generate a feature representation of the digital image of the eye by processing the digital image of the eye using the trained machine learning model 1025, for example, using a convolutional neural network 1210. The image processing system 1170 may then process the generated feature representation and the digital image of the eye using the trained machine learning model, for example, using at least one long-term short-term memory unit 1220. The image processing system 1170 may then output a difference between the digital image of the eye and a predicted optimized digital image of the eye. The image processing system 1170 may use this difference to compute an optimized digital image of the eye from the digital image of the eye. Alternatively, the image processing system 1170 may calculate an optimized digital image of any digital image of the eye in any suitable manner.

[0102] Digital image optimization system 100, digital image optimization system 500, digital image optimization system 700, and optimization network 1000 may be used in conjunction with digital time-out system 1400, as shown in Figure 14. Digital time-out system 1400 may include identification marker 1405, camera 1450, image processing system 1470, digital display 1490, communication network 1010, and digital image management system 1015. Image processing system 1470 may further include processor 1480 and memory medium 1481.

[0103] Camera 1450 may be a camera such as camera 150. Camera 1450 may be a component of a Digitally Assisted Vitreoretinal Surgery ("DAVS") system or may be a component of the NGENUITY® 3D Visualization System (Novartis AG Corp., Switzerland). Digital display 1490 may be a display such as digital display 190. Digital display 1490 may be a component of a Digitally Assisted Vitreoretinal Surgery ("DAVS") system or may be a component of the NGENUITY® 3D Visualization System (Novartis AG Corp., Switzerland).

[0104] The identification marker 1405 may be affixed to the patient 1410. The identification marker 1405 may be affixed to the eye 101 of the patient 1410. The identification marker 1405 may be affixed to the eye 101 of the patient 1410 that will be operated on. Alternatively, the identification marker 1405 may be affixed to the patient 1410 in any location that facilitates surgery. The identification marker 1405 may be a removable transfer tattoo, a patch, a sticker, tape, or any combination thereof. The identification marker 1405 may also be any other suitable marker affixed to the patient 1410, such as a marker that cannot come off, a marker that cannot smudge, a marker that cannot be accidentally applied, or a combination thereof.

[0105] The identification marker 1405 may have a white background or a black background. Alternatively, the identification marker 1405 may be black against a white background. The identification marker 1405 may include a locator 1430, such as a crosshair. The camera 1450 may identify the locator 1430 and position itself at a specific location relative to the locator 1430. The camera 1450 may be a robotically controlled camera. The camera 1450 may be capable of moving with six degrees of freedom. The locator 1430 may enable the camera 1450 to automatically position itself over the eye 101 of the patient 1410.

[0106] The identification marker 1405 may include machine-readable information 1420 associated with the patient 1410. For example, the machine-readable information 1420 may include machine-readable marks such as a barcode. The barcode may be a one-dimensional barcode. The barcode may be a multidimensional barcode. The barcode may be a quick response (QR) code 1425. In another example, the machine-readable information 1420 may be or include symbols. The symbols may be or include characters in a handwritten language (e.g., English, German, French, Chinese, Russian, etc.) that may be processed via an optical character recognition (OCR) process and / or system.

[0107] The machine-readable information 1420 may be used to store information associated with the patient 1410. The information associated with the patient 1410 may include identification information associated with the patient 1410. The identification information may include one or more of a name, a date of birth, and a government identification number, among others. The information associated with the patient 1410 may also include medical information associated with the patient 1410. The medical information associated with the patient 1410 may include one or more of a medical procedure, a medication list, the patient's 1410's physician, an area of ​​the patient 1410 that is the subject of the medical procedure, one or more drug allergies, a diagnosis, and the patient's 1410 orientation during the medical procedure, among others. The medical information associated with the patient 1410 may include the eye operated on, e.g., the right eye or the left eye. In another example, the information associated with the patient 1410 may include indexing information. The indexing information may be used to index into a database and / or storage device that stores information associated with the patient 1410. The indexing information may be used as, or may be used to create, a key that may be used to retrieve information associated with the patient 1410 from databases and / or storage devices that store information associated with the patient 1410 .

[0108] The machine-readable information 1420 may be printed on the identification marker 1405 in any type of support medium format, for example, paper, cloth, plastic card, sticker, or any combination thereof. The machine-readable information 1420 may include a bar code and / or a custom optical code and / or mark. The camera 1450 may capture a digital image of the identification marker 1405 and send a signal to the processor 1480. The camera 1450 may also capture a digital image of the operated eye 101 of the patient 1410 and send a signal to the processor 1480. The camera 1450 may acquire the machine-readable information 1420. The machine-readable information 1420 may be processed by the image processing system 1470 using the processor 1480. The digital image of the eye 101 of the patient 1410 may also be processed by the image processing system 1470 using the processor 1480. The camera 1450 may automatically acquire the machine-readable information 1420 by recognizing the identification marker 1405 in the operating room. The machine-readable information 1420 may initiate an automatic white balance color calibration of the camera 1450 when the camera 1450 acquires the machine-readable information 1420.

[0109] Processing the machine-readable information 1420 may include processing the one or more images to identify information stored via the machine-readable information 1420. For example, the one or more images may include one or more barcodes that include information related to the medical procedure. The one or more barcodes may include information described herein, and processing the machine-readable information 1420 may include retrieving the information related to the medical procedure from the machine-readable information 1420.

[0110] The machine-readable information 1420 may be processed using the processor 1480 to identify patient data. In one example, the machine-readable information 1420 may include patient data. In another example, processing the machine-readable information 1420 to identify patient data may include retrieving one or more body parts of the patient from a storage device. The patient data retrieved from the storage device may include information about a planned medical procedure, image data from a previous eye surgery or examination, diagnostic data from a previous eye surgery or examination, information about a previous eye condition, or any combination thereof. The retrieved patient data may be displayed on the digital display 1490. The retrieved patient data may be displayed as a picture-in-picture display on the digital display 1490. The machine-readable information 1420 may include encrypted information. For example, processing the machine-readable information 1420 to identify patient data may include decrypting information stored via the machine-readable information 1420 to identify the patient data.

[0111] The machine-readable information 1420 may be read at the start of surgery. The machine-readable information 1420 may be read at the end of surgery. Alternatively, the machine-readable information 1420 may be read at any time necessary during surgery. Reading the machine-readable information 1420 may initiate a digital timeout. The digital timeout may provide a computational check that information associated with the patient 1410 matches information provided to the surgeon, operating room staff, or a combination thereof. The digital timeout may be used to detect any errors in the information provided to the surgeon, operating room staff, or a combination thereof compared to the information associated with the patient 1410. For example, the digital timeout may provide the surgical equivalent of a parity check of the patient's identity, the planned medical procedure, the operating eye, or any combination thereof.

[0112] The digital time-out may confirm the identity of the patient 1410. During the digital time-out, the image processing system 1470 may compare the identification information associated with the patient 1410 stored in the machine-readable information 1420 with information provided to the surgeon, operating room staff, or a combination thereof. During the digital time-out, the image processing system 1470 may use the machine-readable information 1420 to retrieve an image of the patient's eye and compare this image to the digital image of the eye 101 of the patient 1410 captured using the camera 1450. This may confirm the identity of the patient 1410.

[0113] The digital timeout may confirm the medical procedure planned for the patient 1410. During the digital timeout, the image processing system 1470 may compare the identity of the operative eye provided in the medical information associated with the patient 1410 with the identity of the operative eye in the information provided to the surgeon, operating room staff, or a combination thereof. During the digital timeout, the image processing system 1470 may compare the medical procedure being performed with the planned medical procedure recorded in the medical information associated with the patient 1410.

[0114] Image processing system 1470 may identify discrepancies between information associated with patient 1410 provided by machine-readable information 1420 and information provided to a surgeon, operating room staff, or combination thereof. Image processing system 1470 may identify matches between information associated with patient 1410 and information provided to a surgeon, operating room staff, or combination thereof. Image processing system 1470 may identify discrepancies, matches, or combinations thereof between information associated with patient 1410 and information provided to a surgeon, operating room staff, or combinations thereof by displaying the discrepancies, matches, or combinations thereof between information associated with patient 1410 and information provided to a surgeon, operating room staff, or combinations thereof on digital display 1490. Alternatively, image processing system 1470 may report discrepancies, matches, or combinations thereof between information associated with patient 1410 and information provided to a surgeon, operating room staff, or combinations thereof using any suitable means. If there is a discrepancy between the information associated with the patient 1410 and the information provided to the surgeon, the operating room staff, or a combination thereof, the digital timeout system 1400 may prevent the surgeon from using medical equipment, such as the camera 1450.

[0115] The machine-readable information 1420 may also be utilized in an optimization network, such as optimization network 1000. The machine-readable information 1420 may include indexing information that may be associated with or digitally stored with a corresponding particular digital image of the eye 101. For example, the indexing information provided by the machine-readable information 1420 may be digitally stored in the digital image management system 1015 along with the corresponding digital image of the eye 101 of the patient 1410. The digital image management system 1015 may generate an optimized digital image of the eye 1491 from the digital image of the eye 101. The digital image management system 1015 may communicate with a local image processing system, such as image processing system 1470, via the communications network 1010.

[0116] The digital image management system 1015 may include an image processing system, such as image processing system 1170, that may use the patient's digital images of the eye 101 in a set of training images, such as training images 1030, to train a machine learning model, such as machine learning model 1025. The information associated with the patient 1410 provided by the machine-readable information 1420 may provide additional information about the patient's digital images of the eye 101, which may be useful in record-keeping for storing the digital images of the eye in the optimization network. The information associated with the patient 1410 provided by the machine-readable information 1420 may provide additional information about the patient's digital images of the eye 101, which may also be useful in selecting training images for training the machine learning model. For example, information such as gender, age, ocular condition, or any combination thereof may be used to filter the digital images of the eye that are selected as training images to provide the image processing system trained to calculate optimized images of the eye.

[0117] FIG. 15 presents a flowchart of a method for implementing a digital timeout. In step 1500, machine-readable information, such as machine-readable information 1420 printed on identification marker 1405, may be acquired by a camera, such as camera 1450. In step 1510, the machine-readable information is processed, for example, by image processing system 1470. In step 1520, the accuracy of the patient's identification information compared to information provided to an in-room surgeon, staff, or combination thereof is determined based on the machine-readable information. The accuracy of the patient's identification information may be determined by comparing identification information associated with the patient, for example, patient 1410, to information provided to an in-room surgeon, staff, or combination thereof, which may be, for example, a list of patients scheduled for medical procedures that day. In step 1530, if the patient's identification information is incorrect, the planned medical procedure is stopped. In step 1540, if the patient's identification information is accurate, the accuracy of the planned medical procedure compared to information provided to an in-room surgeon, staff, or combination thereof is determined based on the machine-readable information. The accuracy of the planned medical procedure may be determined by comparing medical information associated with the patient, e.g., patient 1410, with information provided to the surgeon, staff, or a combination thereof in the operating room, which may be, for example, a list of medical procedures scheduled for that day. If the planned medical procedure is incorrect, in step 1550, the planned medical procedure is stopped. If the planned medical procedure is accurate, in step 1560, the planned medical procedure is performed.

[0118] FIG. 16 presents a flowchart of a method for optimizing a digital image of an eye in ophthalmic surgery. In step 1600, a digital image of an eye, such as a digital image of eye 101, is captured by a digital image optimization system. The digital image optimization system may be, for example, digital image optimization system 100, and the digital image of the eye may be captured by, for example, camera 150. In another example, the digital image optimization system may be digital image optimization system 500, and the digital image of the eye may be captured by camera 550. Alternatively, the digital image optimization system may be digital image optimization system 700, and the digital image of the eye may be captured by an endoscope, such as endoscope 715. In step 1610, the digital image of the eye is associated with processed machine-readable information, for example, machine-readable information 1420. The machine-readable information may be processable when the digital image of the eye is captured during a digital timeout, for example, a digital timeout initiated using digital timeout system 1400. The machine-readable information may include information associated with the patient whose eye is in the digital image of the eye. In step 1620, the digital image of the eye is used to train a machine learning model, such as machine learning model 1025, in optimization network 1000. For example, the digital image of the eye may be included as an input in a plurality of training images, such as training images 1030. The digital image of the eye may have been selected for inclusion in training images 1030 based on the processed machine-readable information. In step 1630, the digital image of the eye is processed by a trained machine learning model, such as trained machine learning model 1025. In an alternative example, step 1630 may follow directly from step 1600 or from step 1610 if the machine learning model has already been trained. In step 1640, an optimized digital image of the eye is generated by an image processing system, such as image processing system 1170. In step 1650, the optimized digital image of the eye is displayed on a digital display.For example, the optimized digital image may be displayed on digital display 190, digital display 590, digital display 790, or digital display 1490.

[0119] Digital image optimization system 100, digital image optimization system 500, digital image optimization system 700, optimization network 1000, or digital timeout system 1400 may be used in combination with computer system 1700, as shown in diagram 1700. Computer system 1700 may include a processor 1710, a volatile memory medium 1720, a non-volatile memory medium 1730, and input / output (I / O) devices 1740. Volatile memory medium 1720, non-volatile memory medium 1730, and I / O devices 1740 may be communicatively coupled to processor 1710.

[0120] The term “memory medium” may refer to “memory,” “storage device,” “memory device,” “computer-readable medium,” and / or “tangible computer-readable storage medium.” For example, memory medium may include, without limitation, storage media such as direct access storage devices including hard disk drives, sequential access storage devices such as tape disk drives, compact discs (CDs), random access memory (RAM), read-only memory (ROM), CD-ROMs, digital versatile discs (DVDs), electrically erasable programmable read-only memory (EEPROM), flash memory, non-transitory media, or any combination thereof. As shown in FIG. 17 , non-volatile memory medium 1730 may include processor instructions 1732. Processor instructions 1732 may be executed by processor 1710. In one example, one or more portions of processor instructions 1732 may be executed via non-volatile memory medium 1730. In another example, one or more portions of processor instructions 1732 may be executed via volatile memory medium 1720. One or more portions of the processor instructions 1732 may be transferred to the volatile memory medium 1720 .

[0121] The processor 1710 may execute the processor instructions 1732 in performing at least a portion of one or more systems, one or more flowcharts, one or more processes, and / or one or more methods described herein. For example, the processor instructions 1732 may comprise, code, and / or encode a plurality of instructions in accordance with at least a portion of one or more systems, one or more flowcharts, one or more methods, and / or one or more processes described herein. While the processor 1710 is shown as a single processor, the processor 1710 may be or include multiple processors. One or more of the storage medium and memory medium may be a software product, a program product, and / or an article of manufacture. For example, the software product, the program product, and / or the article of manufacture may be comprised, coded, and / or encoded with processor-executable instructions in accordance with at least a portion of one or more systems, one or more flowcharts, one or more methods, and / or one or more processes described herein.

[0122] Processor 1710 may include any suitable system, device, or apparatus operable to interpret and execute program instructions, process data, or both stored on a memory medium and / or received over a network. Processor 1710 may also include one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), or other circuitry configured to interpret and execute program instructions, process data, or both.

[0123] I / O devices 1740 may include any one or more devices that permit, authorize, and / or enable a user to interact with computer system 1700 and related elements by enabling input from and output to a user. Facilitating input from a user allows the user to operate and / or control computer system 1700, and facilitating output to a user allows computer system 1700 to display the effects of the user's operation and / or control. For example, I / O devices 1740 enable a user to input data, instructions, or both into computer system 1700 and otherwise operate and / or control computer system 1700 and its associated components. I / O devices may include user interface devices such as a keyboard, mouse, touchscreen, joystick, handheld lens, tool tracking device, coordinate input device, or any other I / O device suitable for use with the system.

[0124] The I / O devices 1740 may include, among other things, one or more buses, one or more serial devices, and / or one or more network interfaces that may facilitate and / or allow the processor 1710 to implement at least a portion of one or more systems, processes, and / or methods described herein. In one example, the I / O devices 1740 may include a storage interface that may facilitate and / or allow the processor 1710 to communicate with external storage devices. The storage interface may include, among other things, one or more of a Universal Serial Bus (USB) interface, a Serial ATA (SATA) interface, a Parallel ATA (PATA) interface, and a Small Computer System Interface (SCSI). In a second example, the I / O devices 1740 may include a network interface that may facilitate and / or allow the processor 1710 to communicate with a network. The I / O devices 1740 may include one or more of a wireless network interface and a wired network interface. In a third example, the I / O device(s) 1740 may include one or more of a Peripheral Component Interconnect (PCI) interface, a PCI Express (PCIe) interface, a Serial Peripheral Interconnect (SPI) interface, and an Inter-Integrated Circuit (I2C) interface, among others. In a fourth example, the I / O device(s) 1740 may include circuitry that may enable the processor 1710 to communicate data with one or more sensors. In a fifth example, the I / O device(s) 1740 may facilitate and / or enable the processor 1710 to communicate data with one or more of the display 1750 and the digital image optimization system 100, among others. As shown in FIG. 17 , the I / O device(s) 1740 may be coupled to a network 1770. For example, the I / O device(s) 1740 may include a network interface. In another example, the network 1770 may include the network 1010.

[0125] Network 1770 may include a wired network, a wireless network, an optical network, or any combination thereof. Network 1770 may include and / or be coupled to various types of communication networks. For example, network 1770 may include and / or be coupled to a local area network (LAN), a wide area network (WAN), the Internet, a public switched telephone network (PSTN), a cellular telephone network, a satellite telephone network, or any combination thereof. A WAN may include a private WAN, a corporate WAN, a public WAN, or any combination thereof.

[0126] 17 shows computer system 1700 as being external to digital image optimization system 100, digital image optimization system 100 may include computer system 1700. For example, processor 1710 may be or include processor 180.

[0127] 18A-18C illustrate an example of a medical system 1800. As shown in FIG. 18A, medical system 1800 may include digital image optimization system 100. Alternatively, medical system 1800 may include digital image optimization system 500, digital image optimization system 700, optimization network 1000, digital timeout system 1400, or any combination thereof. As shown in FIG. 18B, medical system 1800 may include digital image optimization system 100 and computer system 1700. Digital image optimization system 100 may be communicatively coupled to computer system 1700. As shown in FIG. 18C, medical system 1800 may include digital image optimization system 100, which may include computer system 1700.

[0128] Digital image optimization system 100, digital image optimization system 500, digital image optimization system 700, optimization network 1000, digital timeout system 1400, computer system 1700, medical system 1800, and their components may be combined with other elements of the visualization tools and systems described herein unless clearly mutually exclusive. For example, digital image optimization system 500 may be combined with digital timeout system 1400 and may be used in conjunction with other optimization systems, visualization systems, computer systems, and medical systems described herein.

[0129] The above disclosed subject matter should be considered illustrative and not limiting, and the appended claims are intended to cover all such modifications, improvements, and other embodiments that fall within the true spirit and scope of the present disclosure. For example, while digital image optimization systems are most commonly required to improve digital visualization in ophthalmic surgery, the systems and methods described herein can be employed where useful in other procedures, such as purely diagnostic procedures, that are not normally considered surgical procedures. The present invention includes the following aspects. [Aspect 1] 1. A digital image optimization system comprising: a camera comprising at least one sensor operable to detect light reflected from the eye and to transmit a signal corresponding to the detected light to a processor; An image processing system including the processor, Executing instructions to generate a digital image of the eye; and an image processing system operable to execute instructions to apply a digital image optimization algorithm including a trained machine learning model to the digital image of the eye to generate an optimized digital image of the eye; a digital display operable to display the optimized digital image of the eye. [Aspect 2] 2. The digital image optimization system of claim 1, wherein the digital image optimization algorithm at least partially reduces instrument glare, optical aberrations, vitreous opacities, or any combination thereof, in the digital image of the eye to generate the optimized digital image of the eye. [Aspect 3] 2. The digital image optimization system of claim 1, wherein the optimized digital image of the eye has greater contrast, sharpness, clarity, dynamic range, or any combination thereof than the digital image of the eye. [Aspect 4] 2. The digital image optimization system of claim 1, wherein the optimized digital image of the eye has less noise, distortion, vignetting, or any combination thereof than the digital image of the eye. [Aspect 5] 2. The digital image optimization system of claim 1, wherein the digital image optimization algorithm utilizes interpolation, multi-exposure image noise reduction, or a combination thereof to generate the optimized digital image of the eye. [Aspect 6] 2. The digital image optimization system of claim 1, wherein the trained machine learning model is trained using a plurality of training images from a red-boosting library. [Aspect 7] 2. The digital image optimization system of claim 1, wherein the trained machine learning model was trained using a plurality of digital images of the eye captured at the beginning of surgery and a plurality of corresponding digital images of the eye captured at the end of the surgery. [Aspect 8] 1. A digital image optimization system comprising: A camera comprising at least one sensor, the at least one sensor comprising: detecting light reflected from the fellow eye and transmitting a signal corresponding to the detected light to a processor; and a camera operable to detect light reflected from the patient's eye and to transmit a signal corresponding to the detected light to a processor; An image processing system including the processor, Executing instructions to generate a template digital image of the fellow eye; Executing instructions to generate a digital mirror image of the template digital image; Executing instructions to generate a working digital image of the surgical eye; and an image processing system operable to execute instructions to align the digital mirror image with the working digital image to generate a retinal arcade alignment template; a digital display operable to display the working digital image and the retinal arcade arrangement template. [Aspect 9] 9. The digital image optimization system of claim 8, wherein the retinal arcade alignment template is displayed as an overlay on the working digital image. [Aspect 10] A digital image optimization system as described in aspect 8, wherein the template digital image includes an ocular structure that is the retina of the fellow eye, the central retinal vein of the fellow eye, the retinal arcade of the fellow eye, the optic disc of the fellow eye, or any combination thereof. [Aspect 11] A digital image optimization system as described in aspect 8, wherein the working digital image includes an ocular structure that is the retina of the surgical eye, the central retinal vein of the surgical eye, the retinal arcade of the surgical eye, the optic disc of the surgical eye, or any combination thereof. [Aspect 12] A digital image optimization system as described in aspect 8, wherein the working digital image includes a field of view of the surgical eye equivalent to a field of view of the template digital image of the fellow eye. [Aspect 13] The digital image optimization system of aspect 8, wherein the alignment of the digital image to the working digital image includes alignment of ocular structures, such as the optic nerve disc of the fellow eye and the optic nerve disc of the surgical eye, the central retinal vein of the fellow eye and the central retinal vein of the surgical eye, the arterial vein intersection of the fellow eye and the arterial vein intersection of the surgical eye, the branch vein of the fellow eye and the branch vein of the surgical eye, or any combination thereof. [Aspect 14] 1. A digital image optimization system comprising: an endoscope including an optical fiber; a camera comprising at least one sensor operable to detect light reflected from within the eye and propagated through the optical fiber and to transmit a signal corresponding to the detected light to a processor; An image processing system including the processor, Executing instructions to generate an endoscopic digital image of the eye; and an image processing system operable to execute instructions to apply an endoscopic digital image optimization algorithm including a trained machine learning model to the endoscopic digital image of the eye to generate an optimized endoscopic digital image of the eye; a digital display operable to display the optimized digital image of the eye.

Claims

1. 1. A digital image optimization system comprising: a camera comprising at least one sensor operable to detect light reflected from the eye and to transmit a signal corresponding to the detected light to a processor; An image processing system including the processor, Executing instructions to generate a digital image of the eye; and an image processing system operable to execute instructions to apply a digital image optimization algorithm including a trained machine learning model to the digital image of the eye to generate an optimized digital image of the eye; a digital display operable to display the optimized digital image of the eye; The trained machine learning model was trained using a plurality of digital images of the eye captured at the beginning of surgery and a plurality of corresponding digital images of the eye captured at the end of surgery. Digital image optimization system.

2. 10. The digital image optimization system of claim 1, wherein the digital image optimization algorithm at least partially reduces instrument glare, optical aberrations, vitreous opacities, or any combination thereof in the digital image of the eye to generate the optimized digital image of the eye.

3. 10. The digital image optimization system of claim 1, wherein the optimized digital image of the eye has equal or greater contrast, sharpness, clarity, dynamic range, or any combination thereof than the digital image of the eye.

4. 10. The digital image optimization system of claim 1, wherein the optimized digital image of the eye has less noise, distortion, vignetting, or any combination thereof than the digital image of the eye.

5. 10. The digital image optimization system of claim 1, wherein the digital image optimization algorithm utilizes interpolation, multi-exposure image noise reduction, or a combination thereof to generate the optimized digital image of the eye.

6. 2. The digital image optimization system of claim 1, wherein the trained machine learning model is trained using a plurality of training images from a red boost library, the red boost library including images of a poor red reflex of the eye and images of a bright red reflex of the eye.

Images